Jellyfish are about 95% water, making them some of the most diaphanous, delicate animals on the planet. But the remaining 5% of them have yielded important scientific discoveries, like green fluorescent protein (GFP) that is now used extensively by scientists to study gene expression, and life-cycle reversal that could hold the keys to combating aging. Jellyfish may very well harbor other, potentially life-changing secrets, but the difficulty of collecting them has severely limited the study of such “forgotten fauna.” The sampling tools available to marine biologists on remotely operated vehicles (ROVs) were largely developed for the marine oil and gas industries, and are much better-suited to grasping and manipulating rocks and heavy equipment than jellies, often shredding them to pieces in attempts to capture them.
Now, a new technology developed by researchers at Harvard’s Wyss Institute for Biologically Inspired Engineering, John A. Paulson School of Engineering and Applied Sciences (SEAS), and Baruch College at CUNY offers a novel solution to that problem in the form of an ultra-soft, underwater gripper that uses hydraulic pressure to gently but firmly wrap its fettuccini-like fingers around a single jellyfish, then release it without causing harm. The gripper is described in a new paper published in Science Robotics.
“Our ultra-gentle gripper is a clear improvement over existing deep-sea sampling devices for jellies and other soft-bodied creatures that are otherwise nearly impossible to collect intact,” said first author Nina Sinatra, Ph.D., a former graduate student in the lab of Robert Wood at the Wyss Institute. “This technology can also be extended to improve underwater analysis techniques and allow extensive study of the ecological and genetic features of marine organisms without taking them out of the water.”
The gripper’s six “fingers” are composed of thin, flat strips of silicone with a hollow channel inside bonded to a layer of flexible but stiffer polymer nanofibers. The fingers are attached to a rectangular, 3D-printed plastic “palm” and, when their channels are filled with water, curl in the direction of the nanofiber-coated side. The fingers each exert an extremely low amount of pressure – about 0.0455 kPA, or less than one-tenth of the pressure of a human’s eyelid on their eye. By contrast, current state-of-the-art soft marine grippers, which are used to capture delicate but more robust animals than jellyfish, exert about 1 kPA.
The researchers fitted their ultra-gentle gripper to a specially created hand-held device and tested its ability to grasp an artificial silicone jellyfish in a tank of water to determine the positioning and precision required to collect a sample successfully, as well as the optimum angle and speed at which to capture a jellyfish. They then moved on to the real thing at the New England Aquarium, where they used the grippers to grab swimming moon jellies, jelly blubbers, and spotted jellies, all about the size of a golf ball.
The gripper was successfully able to trap each jellyfish against the palm of the device, and the jellyfish were unable to break free from the fingers’ grasp until the gripper was depressurized. The jellyfish showed no signs of stress or other adverse effects after being released, and the fingers were able to open and close roughly 100 times before showing signs of wear and tear.
“Marine biologists have been waiting a long time for a tool that replicates the gentleness of human hands in interacting with delicate animals like jellyfish from inaccessible environments,” said co-author David Gruber, Ph.D., who is a Professor of Biology and Environmental Science at Baruch College, CUNY and a National Geographic Explorer. “This gripper is part of an ever-growing soft robotic toolbox that promises to make underwater species collection easier and safer, which would greatly improve the pace and quality of research on animals that have been under-studied for hundreds of years, giving us a more complete picture of the complex ecosystems that make up our oceans.”
The ultra-soft gripper is the latest innovation in the use of soft robotics for underwater sampling, an ongoing collaboration between Gruber and Wyss Founding Core Faculty member Robert Wood, Ph.D. that has produced the origami-inspired RAD sampler and multi-functional “squishy fingers” to collect a diverse array of hard-to-capture organisms, including squids, octopuses, sponges, sea whips, corals, and more.
“Soft robotics is an ideal solution to long-standing problems like this one across a wide variety of fields, because it combines the programmability and robustness of traditional robots with unprecedented gentleness thanks to the flexible materials used,” said Wood, who is the co-lead of the Wyss Institute’s Bioinspired Soft Robotics Platform, the Charles River Professor of Engineering and Applied Sciences at SEAS, and a National Geographic Explorer.
“At the Wyss Institute we are always asking, ‘How can we make this better?’ I am extremely impressed by the ingenuity and out-of-the-box thinking that Rob Wood and his team have applied to solve a real-world problem that exists in the open ocean, rather than in the laboratory. This could help to greatly advance ocean science,” said Wyss Institute Founding Director Donald Ingber, M.D., Ph.D., who is also the Judah Folkman Professor of Vascular Biology at Harvard Medical School, the Vascular Biology Program at Boston Children’s Hospital, and Professor of Bioengineering at SEAS.
The team is continuing to refine the ultra-soft gripper’s design, and aims to conduct studies that evaluate the jellyfishes’ physiological response to being held by the gripper, to more definitively prove that they do not cause the animals stress. Wood and Gruber are also co-Principal Investigators of the Schmidt Ocean Institute’s “Designing the Future” project, and will be further testing their various underwater robots on an upcoming expedition aboard the research ship Falkor in 2020.
Additional authors of the paper are Clark Teeple, Daniel Vogt, M.S., and Kevin Kit Parker, Ph.D. from the Wyss Institute and Harvard SEAS. Parker is a Founding Core Faculty member of the Wyss Institute and the Tarr Family Professor of Bioengineering and Applied Physics at SEAS. The research was supported by the National Science Foundation, The Harvard University Materials Research Science and Engineering Center, The National Academies Keck Futures Initiative, and the National Geographic Society.
InSystems and Model Engineering Solutions jointly developed a Simulink model of an adaptive fleet of InSystems’ proANT collaborating transport robots. The goal was to capture the desired adaptive system behavior to more effectively deal with the typical goals and challenges of collaborative embedded system groups (CSGs). A fleet of robots has to react to dynamic changes in the policy of the manufacturing execution system or the number and nature of its members to safeguard its functionality. The consistent application of a model-based development process for automation systems offers a variety of benefits to deal with these challenges. First and foremost, the specification of the CSG in the form of executable models allows for a fully virtual simulated representation of the robot fleet. This provides a sound foundation to efficiently develop and maintain the actual system. To exploit the full potential, a model-based approach relies on the reusability of models and test beds throughout the different development phases. Secondly, the model-based development process profits from a fully integrated tool chain that highly automatizes associated development activities. These include requirements management, modelling, and simulation as well as integrated quality assurance tasks, most notably, model-based static analysis and requirements-based testing. Tools such as the MES Model Examiner® and the MES Test Manager® are beneficial in streamlining the process.
The post Modelling of a Transport Robot Fleet in Simulink appeared first on Roboticmagazine.
D400 Depth Cameras. Credit: Intel CorporationThe Intel RealSense cameras have been gaining in popularity for the past few years for use as a 3D camera and for visual odometry. I had the chance to hear a presentation from Daniel Piro about using the Intel RealSense cameras generally and for SLAM (Simultaneous Localization and Mapping). The following post is based on his talk.
Depth information is important since that gives us the information needed to understand shapes, sizes, and distance. This lets us (or a robot) know how far it is from items to avoid running into things and to plan path around obstacles in the image field of view. Traditionally this information has come from RADAR or LIDAR, however in some applications we can also get that from cameras. In cameras we often get depth from using 2 cameras for stereo vision.
The Intel RealSense Depth camera (D400 series) uses stereoscopic depth sensing to determine the range to an item. So essentially it has two cameras and can do triangulation from them for stereo. This sensor uses two infrared cameras for the stereo and then also has an RGB camera onboard. So you can get 4 data products from the sensor; RGB image, depth image, left infrared image, and right infrared image. Think of each image frame as a 3D snapshot of the environment, where each color (RGB) pixel also has a range value (depth) to the item that is in the image. The farther the items are from the camera the greater the range/depth error will be.
The D400 cameras have an infrared projector for getting better features on surfaces with minimal texture in the infrared camera for computing the stereo reconstruction. This projector can be turned on and off if you want. Disabling the projector is often useful for tracking applications (since the projected dots don’t move with the items being tracked).
One thing to be aware is that the infrared images are rectified (to make the images look the same in a common plane) in the camera, however the RGB camera image is not rectified. This means that if you want the depth and RGB images to line up well, you need to manually rectify the RGB image.
Pro Tip 1: The driver has a UV map to help map from the depth pixel to the RGB image to help account for the difference in image sizes. This lets you match the depth to RGB image data points better.
Pro Tip 2: If using the D435i (the IMU version), use the timestamps from the images and IMU to synchronize the two sensors. If you use system time (from your computer) there will be more error (partially due to weird USB timing).
Pro Tip 3: The cameras have Depth Presets. These are profiles that let you optimize various settings for various conditions. Such as high density, high accuracy, etc..
Pro Tip 4: Make sure your exposure is set correctly. If you are using auto exposure try changing the Mean Intensity Set Point (the setting is not where exposure is, it is under AE control, not the most obvious).
If you want to use manual exposure you can play with are Exposure setpoint and Gain constant. Start with the exposure setpoint, then adjust the gain.
You might also want to see this whitepaper for more methods of tuning the cameras for better performance.
Visual odometry is the generic term for figuring out how far you have moved using a camera. This is as opposed to “standard” odometry using things such as wheel encoders, or inertial odometry with a IMU.
RealSense T265 is a tracking camera that is designed to be more optimal for Visual Odometry and SLAM (wider field of view and not using infrared light). It can do SLAM onboard as well as loop closure. However, this camera is not able to return RGB images (since it does not have a RGB camera onboard) and the depth returned is not as good as the D400 series (and can be a little trickier to get).
Using both a RealSense D435i sensor and a RealSense T265 sensor can provide both the maps and the better quality visual odometry for developing a full SLAM system. The D435i used for the mapping, and the T265 for the tracking.
Intel provides the RealSense SDK2.0 library for using the RealSense cameras. It is Open Source and work on Mac, Windows, Linux, and Android. There are also ROS and OpenCV wrappers.
Click here for the developers page with the SDK.
Within the SDK (software development kit) it includes a viewer to let you view images, record images, change settings, or update the firmware.
Pro Tip 5: Spend some time with the viewer looking at the camera and depth images when designing your system so you can compare various mounting angles, heights, etc.. for the cameras.
The RealSense SDK (software development kit) has a few filters that can run on your computer to try and improve the returned depth map. You can play with turning these on and off in the viewer. Some of these include:
Within ROS there is a realsense-ros package that provides a wrapper for working with the cameras in ROS and lets you view images and other data in RVIZ.
ROS RealSense Occupancy Map package is available as an experimental feature in a separate branch of the RealSense git repo. This uses both the D400 and T265 cameras for creating the map.
For SLAM with just the D435i sensor, see here.
Pro Tip 6: You can use multiple T265 sensors for better accuracy. For example, if one sensor is pointed forward and another backwards; you can use the confidence values from each sensor to feed into a filter.
I know this is starting to sound like a sales pitch
Pro Tip 7: If you have multiple cameras you can connect and query for a serial number to know which cameras is which.
Also if you remove the little connector at the top of the camera you can wire and chain multiple cameras together to synchronize them. (This should work in the SDK 2.0)
See this whitepaper for working with multiple camera configurations. https://simplecore.intel.com/realsensehub/wp-content/uploads/sites/63/Multiple_Camera_WhitePaper04.pdf
Pro Tip 8: Infinite depth points (such as points to close or to far from the sensor) have a depth value of 0. This is good to know for filtering.
Here are two code snippet’s for using the cameras provided by Daniel, to share with you. The first one is for basic displaying of images using python. The second code snippet is using OpenCV to also detect blobs, also using python.
## License: Apache 2.0. See LICENSE file in root directory.
## Copyright(c) 2015-2017 Intel Corporation. All Rights Reserved.
###############################################
## Open CV and Numpy integration ##
###############################################
import pyrealsense2 as rs
import numpy as np
import cv2
cv2.namedWindow('RealSense', cv2.WINDOW_AUTOSIZE)
# Configure depth and color streams
pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)
# Start streaming
pipeline.start(config)
try:
while True:
# Wait for a coherent pair of frames: depth and color
frames = pipeline.wait_for_frames()
depth_frame = frames.get_depth_frame()
color_frame = frames.get_color_frame()
if not depth_frame or not color_frame:
continue
# Convert images to numpy arrays
depth_image = np.asanyarray(depth_frame.get_data())
color_image = np.asanyarray(color_frame.get_data())
# Apply colormap on depth image (image must be converted to 8-bit per pixel first)
scaled_depth=cv2.convertScaleAbs(depth_image, alpha=0.08)
depth_colormap = cv2.applyColorMap(scaled_depth, cv2.COLORMAP_JET)
# Stack both images horizontally
images = np.hstack((color_image, depth_colormap))
# Show images
cv2.imshow('RealSense', images)
k = cv2.waitKey(1) & 0xFF
if k == 27:
break
finally:
# Stop streaming
pipeline.stop()## License: Apache 2.0. See LICENSE file in root directory.
## Copyright(c) 2015-2017 Intel Corporation. All Rights Reserved.
###############################################
## Open CV and Numpy integration ##
###############################################
import pyrealsense2 as rs
import numpy as np
import cv2
def nothing(args):
pass
def detectBlobs(mask):
# Set up the SimpleBlobdetector with default parameters.
params = cv2.SimpleBlobDetector_Params()
# Change thresholds
params.minThreshold = 1;
params.maxThreshold = 255;
# Filter by Area.
params.filterByArea = True
params.maxArea = 4000
params.minArea = 300
# Filter by Circularity
params.filterByCircularity = True
params.minCircularity = 0.1
# Filter by Convexity
params.filterByConvexity = True
params.minConvexity = 0.5
# Filter by Inertia
params.filterByInertia = True
params.minInertiaRatio = 0.1
detector = cv2.SimpleBlobDetector_create(params)
# Detect blobs.
reversemask= mask
keypoints = detector.detect(reversemask)
im_with_keypoints = cv2.drawKeypoints(mask, keypoints, np.array([]),
(0,0,255), cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)
return im_with_keypoints
def thresholdDepth(depth):
depth[depth==0] = 255 #set all invalid depth pixels to 255
threshold_value = cv2.getTrackbarPos('Threshold','Truncated Depth')
# Zero if dist>TH
ret,truncated_depth=cv2.threshold(scaled_depth,threshold_value,255,cv2.THRESH_BINARY_INV)
return truncated_depth
cv2.namedWindow('RealSense', cv2.WINDOW_AUTOSIZE)
cv2.namedWindow('Truncated Depth', cv2.WINDOW_AUTOSIZE)
cv2.createTrackbar('Threshold','Truncated Depth',30,255,nothing)
# Configure depth and color streams
pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)
# Start streaming
pipeline.start(config)
try:
while True:
# Wait for a coherent pair of frames: depth and color
frames = pipeline.wait_for_frames()
depth_frame = frames.get_depth_frame()
color_frame = frames.get_color_frame()
if not depth_frame or not color_frame:
continue
# Convert images to numpy arrays
depth_image = np.asanyarray(depth_frame.get_data())
color_image = np.asanyarray(color_frame.get_data())
# Apply colormap on depth image (image must be converted to 8-bit per pixel first)
scaled_depth=cv2.convertScaleAbs(depth_image, alpha=0.08)
depth_colormap = cv2.applyColorMap(scaled_depth, cv2.COLORMAP_JET)
# Stack both images horizontally
images = np.hstack((color_image, depth_colormap))
# Show images
cv2.imshow('RealSense', images)
truncated_depth=thresholdDepth(scaled_depth)
truncated_depth=detectBlobs(truncated_depth)
cv2.imshow('Truncated Depth', truncated_depth)
k = cv2.waitKey(1) & 0xFF
if k == 27:
break
finally:
# Stop streaming
pipeline.stop()I hope you found this informative and can make use of the Pro Tips.
Thank you to Daniel for presenting this information and allowing me to share it. This content is based on his talk. Daniel has also provided the full slide set that can be accessed by clicking here.
Disclaimer: I have not received any funding or free items from Intel.
Liked this article? Take a second to support me on Patreon! This post appeared first on Robots For Roboticists.
A recent report by the OECD showed that Europe is only capitalising on a fraction (12%) of its digital potential. Most companies limit their digital use to email and internet. Small and Medium Sized Enterprises (SMEs) are particularly slow to integrate digital technologies. The recent Digital Economy and Society Index Report found that “Less than a fifth of companies in the EU-28 are highly digitised“ and that “Use of robots is low on an EU level, with 6.7% of all enterprises using industrial or service robots. The share of large enterprises that use robots is four times higher than the share of SMEs.”
Yet advances in robotics, AI, cloud computing, and big data have the potential to boost productivity and change the industrial landscape.
Digital Innovation Hubs
In an effort to increase uptake of digital technologies in companies, the European Commission promised 500M € in 2016, over 5 years, to launch Digital Innovation Hubs (DIHs). DIHs are regional one-stop-shops that help companies access the latest knowledge, expertise, and technology to prototype and test their own digital innovations, network, receive financing advice and market intelligence.
Key to the success of DIHs is to build on core technology infrastructures, so-called Competence Centres (CC), and other existing structures, and link them on a regional level. The idea is that every company should be able to access at least one DIH within their region. This is especially important since most SMEs confess not knowing where to start when it comes to integrating digital technologies into their companies.
“The DIHs funded by the EU Commission can boost digitalisation in Europe in an effective way by supporting both the public sector and SMEs to become more competitive in a global market” says Francesco Ferro, CEO of the Spanish SME PAL Robotics.
The catalogue of Digital Innovation Hubs, hosted by the Joint Research Centre in Sevilla, currently includes over 290 operational hubs across Europe, with 200 more in preparation. The hubs span the full digital landscape, including robotics, artificial intelligence, big data, high-performance computing and cybersecurity, as well as its applications in healthcare, manufacturing, agriculture and food production, maintenance and inspection.
Focus on Robotics
Robotics is central to Europe’s Digital Strategy and 196 of the registered DIHs have identified robotics as an area of expertise. In addition, the European Commission awarded 80M € since 2018 to 5 DIH networks with a robotics focus. Each of these brings together one or two dozen DIHs, with the aim of expanding to even more during the project.
The Horizon 2020 projects Trinity and DIH² focus on Agile production, RIMA promotes the uptake of robotics for inspection and maintenance applications, DIH-HERO facilitates the healthcare robotics sector, and agROBOfood enables the adoption of robotics technologies in the agri-food sector. To coordinate these projects, the Coordination and Support Action (CSA) RODIN started last November with funding of 2M € over five years to cover the whole life-cycle of the network projects. The aim is to present a harmonised interface to the overall landscape of robotics DIHs, coordinate events and calls, share best practice, and connect with similar networks in other areas, such as DIHNET.EU (Next Generation European DIH Network).
Open calls for funding from robotics DIHs
Networks of Networks
DIHNET.EU, the 1M € three-year project launched in November, coordinates the whole network of DIHs in different domains such as robotics, cybersecurity, photonics, advanced manufacturing, artificial intelligence, high-performance-computing, software, big data, agriculture, and more. Think of it as a pan-European network of networks. DIHNET.EU promotes collaboration between DIHs through workshops and events, provides tools and an on-line community platform for DIHs, and is the link to the JRC that maintains the DIH catalogue. The project provides an overview of DIHs to the outside world and helps the European Commission shape DIH-related workshops. To promote the sharing of best practice, DIHNET.EU just launched the Champions Challenge to identify, show and support success stories from mature DIHs that can inspire and guide other DIHs in their developments.
“The DIHNET.EU project is meant to link all relevant initiatives around DIHs together and to create sustainability. Sharing of best practice, creation of awareness and synergies across application sectors, technologies, and regions will help shape a pan-European business ecosystem” says Reinhard Lafrenz, Secretary General of euRobotics.