By Jessica Montgomery, Senior Policy Adviser
In early December, 8000 machine learning researchers gathered in Long Beach for 2017’s Neural Information Processing Systems conference. In the margins of the conference, the Royal Society and Foreign and Commonwealth Office Science and Innovation Network brought together some of the leading figures in this community to explore how the advances in machine learning and AI that were being showcased at the conference could be harnessed in a way that supports broad societal benefits. This highlighted some emerging themes, at both the meeting and the wider conference, on the use of AI for social good.
Behind (or beyond) the headlines proclaiming that AI will save the world or destroy our jobs, there lie significant questions about how, where, and why society will make use of AI technologies. These questions are not about whether the technology itself is inherently productive or destructive, but about how society will choose to use it, and how the benefits of its use can be shared across society.
In healthcare, machine learning offers the prospect of improved diagnostic tools, new approaches to healthcare delivery, and new treatments based on personalised medicine. In transport, machine learning can support the development of autonomous driving systems, as well as enabling intelligent traffic management, and improving safety on the roads. And socially-assistive robotics technologies are being developed to provide assistance that can improve quality of life for their users. Teams in the AI Xprize competition are developing applications across these areas, and more, including education, drug-discovery, and scientific research.
Alongside these new applications and opportunities come questions about how individuals, communities, and societies will interact with AI technologies. How can we support research into areas of interest to society? Can we create inclusive systems that are able to navigate questions about societal biases? And how can the research community develop machine learning in an inclusive way?
Applying AI to public policy challenges often requires access to complex, multi-modal data about people and public services. While many national or local government administrations, or non-governmental actors, hold significant amounts of data that could be of value in applications of AI for social good, this data can be difficult to put to use. Institutional, cultural, administrative, or financial barriers can make accessing the data difficult in the first instance. If accessible in principle, this type of data is also often difficult to use in practice: it might be held in outdated systems, be organised to different standards, suffer from compatibility issues with other datasets, or be subject to differing levels of protection. Enabling access to data through new frameworks and supporting data management based on open standards could help ease these issues, and these areas were key recommendations in the Society’s report on machine learning, while our report on data governance sets out high-level principles to support public confidence in data management and use.
In addition to requiring access to data, successful research in areas of social good often require interdisciplinary teams that combine machine learning expertise with domain expertise. Creating these teams can be challenging, particularly in an environment where funding structures or a pressure to publish certain types of research may contribute to an incentives structure that favours problems with ‘clean’ solutions.
Supporting the application of AI for social good therefore requires a policy environment that enables access to appropriate data, supports skills development in both the machine learning community and in areas of potential application, and that recognises the role of interdisciplinary research in addressing areas of societal importance.
The Royal Society’s machine learning report comments on the steps needed to create an environment of careful stewardship of machine learning, which supports the application of machine learning, while helping share its benefits across society. The key areas for action identified in the report – in creating an amenable data environment, building skills at all levels, supporting businesses, enabling public engagement, and advancing research – aim to create conditions that support the application of AI for social good.
In addition to these application-focused issues, there are broader challenges for machine learning research to address some of the ethical questions raised around the use of machine learning.
Many of these areas were explored by workshops and talks at the conference. For example, a tutorial on fairness explored the tools available for researchers to examine the ways in which questions about inequality might affect their work. A symposium on interpretability explored the different ways in which research can give insights into the sometimes complex operation of machine learning systems. Meanwhile, a talk on ‘the trouble with bias’ considered new strategies to address bias.
The Royal Society has set out how a new wave of research in key areas – including privacy, fairness, interpretability, and human-machine interaction – could support the development of machine learning in a way that addresses areas of societal interest. As research and policy discussions around machine learning and AI progress, the Society will be continuing to play an active role in catalysing discussions about these challenges.
For more information about the Society’s work on machine learning and AI, please visit our website at: royalsociety.org/machine-learning
By Lindsay Brownell
DNA has often been compared to an instruction book that contains the information needed for a living organism to function, its genes made up of distinct sequences of the nucleotides A, G, C, and T echoing the way that words are composed of different arrangements of the letters of the alphabet. DNA, however, has several advantages over books as an information-carrying medium, one of which is especially profound: based on its nucleotide sequence alone, single-stranded DNA can self-assemble, or bind to complementary nucleotides to form a complete double-stranded helix, without human intervention. That would be like printing the instructions for making a book onto loose pieces of paper, putting them into a box with glue and cardboard, and watching them spontaneously come together to create a book with all the pages in the right order.
But just as paper can also be used to make origami animals, cups, and even the walls of houses, DNA is not limited to its traditional purpose as a passive repository of genetic blueprints from which proteins are made – it can be formed into different shapes that serve different functions, simply by controlling the order of As, Gs, Cs, and Ts along its length. A group of scientists at the Wyss Institute for Biologically Inspired Engineering at Harvard University is investigating this exciting property of DNA molecules, asking, “What types of systems and structures can we build with them?”
They’ve decided to build robots.
At first glance, there might not seem to be much similarity between a strand of DNA and, say, a Roomba
or Rosie the Robot from The Jetsons. “Looking at DNA versus a modern-day robot is like comparing a piece of string to a tractor trailer,” says Wyss Faculty member Wesley Wong, Ph.D., Assistant Professor of Biological Chemistry and Molecular Pharmacology (BCMP) at Harvard Medical School (HMS) and Investigator at Boston Children’s Hospital. Despite the vast difference in their physical form, however, robots and DNA share the ability to be programmed to complete a specific function – robots with binary computer code, DNA molecules with their nucleotide sequences.
Recognizing that commonality, the Wyss Institute created the cross-disciplinary Molecular Robotics Initiative in 2016, which brings together researchers with experience in the disparate disciplines of robotics, molecular biology, and nanotechnology to collaborate and help inform each other’s work to solve the fields’ similar challenges. Wong is a founding member of the Initiative, along with Wyss Faculty members William Shih, Ph.D., Professor of BCMP at HMS and Dana-Farber Cancer Institute; Peng Yin, Ph.D., Professor of Systems Biology at HMS; and Radhika Nagpal, Ph.D., Fred Kavli Professor of Computer Science at Harvard’s John A. Paulson School of Engineering and Applied Sciences (SEAS); as well as other Wyss scientists and support staff.
“We’re not used to thinking about molecules inside cells doing the same things that computers do. But they’re taking input from their environment and performing actions in response – a gene is either turned on or off, a protein channel is either open or closed, etc. – in ways that can resemble what computer-controlled systems do,” says Shih. “Molecules can do a lot of things on their own that robots usually have trouble with (move autonomously, self-assemble, react to the environment, etc.), and they do it all without needing motors or an external power supply,” adds Wyss Founding Director Don Ingber, M.D., Ph.D., who is also the Judah Folkman Professor of Vascular Biology at HMS and the Vascular Biology Program at Boston Children’s Hospital, as well as a Professor of Bioengineering at SEAS. “Programmable biological molecules like DNA have almost limitless potential for creating transformative nanoscale devices and systems.”
Molecular Robotics capitalizes on the recent explosion of technologies that read, edit, and write DNA (like next-generation sequencing and CRISPR) to investigate the physical properties of DNA and its single-stranded cousin RNA. “We essentially treat DNA not only as a genetic material, but as an incredible building block for creating molecular sensors, structures, computers, and actuators that can interact with biology or operate completely separately,” says Tom Schaus, M.D., Ph.D., a Staff Scientist at the Wyss Institute and Molecular Robotics team member.
Many of the early projects taking advantage of DNA-based self-assembly were static structures. These include DNA “clamshell” containers that can be programmed to snap open and release their contents in response to specific triggers, and DNA “bricks” whose nucleotide sequences allow their spontaneous assembly into three-dimensional shapes, like tiny Lego bricks that put themselves together to create sculptures automatically. Many of these structures are three-dimensional, and some incorporate as many as 10,000 unique DNA strands in a single complete structure.
The reliable specificity of DNA and RNA (where A always binds with T or U, C always with G) allows for not only the construction of static structures, but also the programming of dynamic systems that sense and respond to environmental cues, as seen in traditional robotics. For example, Molecular Robotics scientists have created a novel, highly controllable mechanism that automatically builds new DNA sequences from a mixture of short fragments in vitro. It utilizes a set of hairpin-shaped, covalently-modified DNA strands with a single-stranded “overhang” sequence dangling off one end of the hairpin. The overhang sequence can bind to a complementary free-floating fragment of DNA (a “primer”) and act as a template for its extension into a double-stranded DNA sequence. The hairpin ejects the new double strand and can then be re-used in subsequent reactions to produce multiple copies of the new strand.
Such extension reactions can be programmed to occur only in the presence of signal molecules, such as specific RNA sequences, and can be linked together to create longer DNA product strands through “Primer Exchange Reactions” (PER). PER can in turn be programmed to enzymatically cut and destroy particular RNA sequences, record the order in which certain biochemical events happen, or generate components for DNA structure assembly.
PER reactions can also be combined into a mechanism called “Autocycling Proximity Recording” (APR), which records the geometry of nano-scale structures in the language of DNA. In this instance, unique DNA hairpins are attached to different target molecules in close proximity and, if any two targets are close enough together, produce new pieces of DNA containing the molecular identities (“names”) of those two targets, allowing the shape of the underlying structure to be determined by sequencing that novel DNA.
Another tool, called “toehold switches,” can be used to exert complex and precise control over the machinery inside living cells. Here, a different, RNA-based hairpin is designed to “open” when it binds to a specific RNA molecule, exposing a gene sequence in its interior that can be translated into a protein that then performs some function within the cell. These synthetic circuits can even be built with logic-based sequences that mimic the “AND,” “OR,” and “NOT” system upon which computer languages are based, which prevents the hairpin from opening and its gene from being translated except under very specific conditions.
Such an approach could induce cells that are deficient in a given protein to produce more of it, or serve as a synthetic immune system that, when it detects a given problem in the cell, produces a toxin that kills it to prevent it from spreading an infection or becoming cancerous. [toeholds] “Because we have a thorough understanding of DNA and RNA’s properties and how their bases pair together, we can use that simple machinery to design complex circuits that allow us to precisely interact with the molecular world,” says Yin. “It’s an ability that has been dreamed about for a long time, and now, we’re actually making it a reality.”
The potential applications of that ability are seemingly endless. In addition to the previously mentioned tools, Molecular Robotics researchers have created loops of DNA attached to microscopic beads to create “calipers” that can both measure the size, structure, and stiffness of other molecules, and form the basis of inexpensive protein recognition tests. Another advance is folding single-stranded DNA into molecular origami to create molecular structures, rather than traditional double-stranded DNA. Some academic projects are already moving into the commercial sector. These include a low-cost alternative to super-resolution microscopy that can image up to 100 different molecular targets in a single sample (DNApaint), as well as a multiplexed imaging technique that integrates fluorescent probes into self-folding DNA structures and enables simultaneous visualization of ultra-rare DNA and/or RNA molecules.
We’re trying to push the limits of these really dumb little molecules to get them to behave in sophisticated, collective ways – it’s a new frontier for DNA nanotechnology.
One of the major benefits of engineering molecular machines is that they’re tiny, so it’s relatively easy to create a large amount of them to complete any one task (for example, circulating through the body to detect any rogue cancer DNA). Getting simple, individual molecules to interact with each other to achieve a more complex, collective task (like relaying the information that cancer has been found), however, is a significant challenge, and one that the roboticists in Molecular Robotics are tackling at the macroscopic scale with inch-long “Kilobots.”
Taking cues from colonies of insects like ants and bees, Wyss researchers are developing swarms of robots that are themselves limited in function but can form complex shapes and complete tasks by communicating with each other via reflected infrared light. The insights gained from studies with the Kilobots are likely to be similar to those needed to solve similar problems when trying to coordinate molecular robots made of DNA.
“In swarm robotics, you have multiple robots that explore their environment on their own, talk to each other about what they find, and then come to a collective conclusion. We’re trying to replicate that with DNA but it’s challenging because, as simple as Kilobots are, they’re brilliant compared to DNA in terms of computational power,” says Justin Werfel, Ph.D., a Senior Research Scientist at the Wyss Institute and director of the Designing Emergence Laboratory at Harvard. “We’re trying to push the limits of these really dumb little molecules to get them to behave in sophisticated, collective ways – it’s a new frontier for DNA nanotechnology.”
Given the magnitude of the challenge and the short time the Molecular Robotics Initiative has existed, it is already making significant progress, with more than two dozen papers published and two companies (Ultivue and NuProbe) founded around its insights and discoveries. It may take years of creative thinking, risk taking, and swapping ideas across the members’ different expertise areas before a molecule of DNA is able to achieve the same task on the nanoscale that a robot can do on the human scale, but the team is determined to see it happen.
“Our vision with Molecular Robotics is to solve hard problems humanity currently faces using smaller, simpler tools, like a single loop of DNA or a single Kilobot that can act cooperatively en masse, instead of bigger, more complex ones that are harder to develop and become useless should any one part fail,” says Wong. “It’s an idea that definitely goes against the current status quo, and we’re lucky enough to be pursuing it here at the Wyss Institute, which brings together people with common goals and interests to create new things that wouldn’t exist otherwise.”
Click on the links below to explore research from the Molecular Robotics Initiative.
In 2016, the European Union co-funded 17 new robotics projects from the Horizon 2020 Framework Programme for research and innovation. 16 of these resulted from the robotics work programme, and 1 project resulted from the Societal Challenges part of Horizon 2020. The robotics work programme implements the robotics strategy developed by SPARC, the Public-Private Partnership for Robotics in Europe (see the Strategic Research Agenda).
EuRobotics regularly publishes video interviews with projects, so that you can find out more about their activities. This week features REELER: Responsible Ethical Learning with Robotics.
The project aims at aligning roboticists’ visions of a future with robots with empirically-based knowledge of human needs and societal concerns through a new proximity-based human-machine ethics that take into account how individuals and community connect with robot technologies.
The main outcome of REELER is a research-based roadmap presenting:
At the core of these guidelines is the concept of collaborative learning, which permeates all aspects of REELER and will guide future SSH-ICT research.
Integrating the recommendations of the REELER Roadmap for responsible and ethical learning in robotics in future robot design processes will enable the European robotics community to addresses human needs and societal concerns. Moreover, the project will give powerful instruments able to foster networking and exploit potentialities of future robotics projects.
AARHUS UNIVERSITY, DPU
AB.ACUS SRL
DE MONTFORT UNIVERSITY, CCSR
HOHENHEIM UNIVERSITY
Stine Trentemøller
If you enjoyed reading this article, you may also want to read:
See all the latest robotics news on Robohub, or sign up for our weekly newsletter.
Well, this year’s videos are getting creative!
Have a holiday robot video of your own that you’d like to share? Send your submissions to editors@robohub.org.
“Cozmo stars in Christmas Wrap” by Life with Cozmo
“Don’t be late for Christmas!” by FZI Living Lab
“LTU Robotics Team Christmas Video 2017” by the Control Engineering Group of Luleå University of Technology, Sweden.
Warning: This video is insane . . .
“Misletoe: A robot love story” by the Robot Drive-In Movies.
For more holiday videos, check last week’s post. Email us your holiday robot videos at editors@robohub.org!
When you think of a treadmill, what comes to your mind?
Perhaps the images of a person burning calories, or maybe the treadmill fail videos online. But almost certainly not a miniature treadmill for insects, and particularly not as a tool for understanding fundamental biology and its applications to technology.
Researchers have been studying insects walking on a treadmill.
But why!?
Traditional methods for investigating an insect’s biology include observing them in their natural habitat or in the lab, and manipulating the animal or its surroundings. While this is sufficient for some research questions, it has its limitations. It is challenging to study certain behaviours like flight and navigation as it is difficult to manipulate insects in motion. Scientists have been using the simple concept of a treadmill to address this. (1, 2). When insects fly or navigate, they typically use visual cues from their surroundings. So a screen with images/videos projected on it can be used to study how the insects behave with such cues. Alternatively, a virtuality reality setup added to the treadmill can help in manipulating the cues in real-time.
How do you make a treadmill for insects?
A miniature insect treadmill is a light-weight hollow Styrofoam ball suspended on an airflow. An ant, bee or a fly is tethered using a dental floss or a metal wire and placed on the top of the ball. Motion of the ball as the insect walks on it is recorded by two optical sensors similar to the one you find in a desktop mouse. This setup can be used as is outdoors, or with stationary images projected on a screen, or with a virtual reality screen instead. For virtual reality, as the ant walks on the ball, the sensors record the movement of the ball to extract the fictive movement of the insect in two dimensional space. This information is then transmitted to a computer which creates corresponding movement in the images/video on the virtual reality screen. For ants, this is almost as if they are walking and experiencing the change in the surroundings in real-time.
What can you learn from this about the insects?
Scientists have been able to learn about how visual cues influence flight and navigation in bees and ants by projecting them on a screen while tethered insects walk on a treadmill. Neural responses in different parts of their brain can also be recorded while the tethered insects are performing different behaviour. Such experiments can inform us about how they learn and remember different visual cues.
Do they show naturalistic behaviour on the treadmill?
At least in some ants like Cataglyphis fortis, the behaviours on the treadmill are similar to natural behaviour. However, the treadmill setup is still not free of shortcomings.
For example, restricting the movement of a flying insect like bees or flies tethered over the treadmill can affect their sensorymotor experience. Insect brains are evolved such that certain sensory feedback is required to elicit motor actions (behaviour). Flying on the treadmill might not feel the same for the insects. But recent technology has made it possible to use the virtual reality in real time for freely moving insects (and also mice and fish). High speed cameras can now record the 3D position of a freely flying insect, and transmit that to a computer which updates the visuals on the screen accordingly. The whole setup looks as if the insects are in a computer game.
The experimenters control the fly’s position (red circles) and its flight direction by providing strong visual motion stimuli. Left: live camera footage, Right: plot of flight positions. Credit: https://strawlab.org/freemovr
On the other hand, this setup cannot be used to study depth perception or 3D vision (stereopsis) in insects like praying mantises as the projections on the screen are two dimensional. Luckily, researchers at Newcastle University (link) have found another ingenious way — 3D movie glasses! They cut out mantis-eye-sized glasses out of an ordinary human 3D glasses and attach them to the mantis eyes using beeswax. The visuals on the screen can now be similar to any 3D movie. This technique can potentially help to build simpler 3D vision systems for robots.
Another challenge with the treadmill setup include not being able to re-create different kinds of sensory information that they experience in nature. This may also be achieved in future.
What are the applications of this fundamental research?
The treadmill with virtual reality setup is an example of how technology can advance science, and how fundamental biological research in turn can inspire technology. Since insects have simpler nervous and sensory systems than humans, they are easier to mimic. While the latest technology has helped uncover biological secrets of insects, that in turn can be an inspiration for bio-robots.
Take for example, the Moth robots. Moths use chemicals (pheromones) to communicate. So moths on a treadmill can navigate towards the smell. The motion of the treadmill as the tethered moth walks towards the smell can drive a small robot. Using the insect pilot in the cockpit of a robot, one could locate necessary odour signals in areas humans cannot reach.
Ants navigating on the treadmill can also inspire visually navigating robots and driverless cars (link). This can have applications ranging from disaster management to extra-terrestrial navigation. Perhaps in the future, ants-sized robots could visually navigate and search for the victims stuck under rubble after a devastating earthquake.
So the simple concept of a treadmill and the latest virtual reality can help biological research and inspire technology in different ways. What might be next, an insect gym?
By Paul Salmon, University of the Sunshine Coast
Millions of fans all over the world eagerly anticipated this week’s release of Star Wars: The Last Jedi, the eighth in the series. At last we will get some answers to questions that have been vexing us since 2015’s The Force Awakens.
Throughout the franchise, the core characters have been accompanied by a number of much-loved robots, including C-3PO, R2-D2 and more recently, BB-8 and K2-SO. While often fulfilling the role of wise-cracking sidekicks, these and other robots also play an integral role in events.
Interestingly, they can also tell us useful things about automation, such as whether it poses dangers to us and whether robots will ever replace human workers entirely. In these films, we see the good, bad and ugly of robots – and can thus glean clues about what our technological future might look like.
One major fear is that robots and automation will replace us, despite work design principles that tell us technology should be used as a tool to assist, rather than replace, humans. In the world of Star Wars, robots (or droids as they are known) mostly assist organic lifeforms, rather than completely replace them.
So for instance, C-3PO is a protocol droid who was designed to assist in translation, customs and etiquette. R2-D2 and the franchise’s new darling, BB-8, are both “astromech droids” designed to assist in starship maintenance.
In the most recent movie, Rogue One, an offshoot of the main franchise, we were introduced to K2-SO, a wisecracking advanced autonomous military robot who was caught and reprogrammed to switch allegiance to the rebels. K2-SO mainly acts as a co-pilot, for example when flying a U-Wing with the pilot Cassian Andor to the planet of Eadu.
In most cases then, the Star Wars droids provide assistance – co-piloting ships, helping to fix things, and even serving drinks. In the world of these films, organic lifeforms are still relied upon for most skilled work.
When organic lifeforms are completely replaced, it is generally when the work is highly dangerous. For instance, during the duel between Annakin and Obi Wan on the planet Mustafar in Revenge of the Sith, DLC-13 mining droids can be seen going about their work in the planet’s hostile lava rivers.
Further, droid armies act as the frontline in various battles throughout the films. Perhaps, in the future, we will be OK with losing our jobs if the work in question poses a significant risk to our health.
However, there are some exceptions to this trend in the Star Wars universe. In the realm of healthcare, for instance, droids have fully replaced organic lifeforms. In The Empire Strikes Back a medical droid treats Luke Skywalker after his encounter with a Wampa, a yeti-like snow beast on the planet Hoth. The droid also replaces his hand following his battle with Darth Vadar on the planet Bespin.
Likewise, in Revenge of the Sith, a midwife droid is seen delivering the siblings Luke and Leia on Polis Massa.
Perhaps this is one area in which Star Wars has it wrong: here on earth, full automation is a long way off in healthcare. Assistance from robots in healthcare is the more realistic prospect and is in fact, already here. Indeed, robots have been assisting surgeons in operating theatres for some time now.
Driverless vehicles are currently flavour of the month – but will we actually use them? In Star Wars, despite the capacity for spacecraft and star ships to be fully automated, organic lifeforms still take the controls. The spaceship Millenium Falcon, for example, is mostly flown by the smuggler Han Solo and his companion Chewbacca.
Most of the Star Wars starship fleet (A-Wings, X-Wings, Y-Wings, Tie Fighters, Star Destroyers, Starfighters and more) ostensibly possess the capacity for fully automated flight, however, they are mostly flown by organic lifeforms. In The Phantom Menace the locals on Tatooine have even taken to building and manually racing their own “pod racers”.
It seems likely that here on earth, humans too will continue to prefer to drive, fly, sail, and ride. Despite the ability to fully automate, most people will still want to be able to take full control.
Utopian visions often depict a future where sophisticated robots will perform highly skilled tasks, all but eradicating the costly errors that humans make. This is unlikely to be true.
A final message from the Star Wars universe is that the droids and advanced technologies are often far from perfect. In our own future, costly human errors may simply be replaced by robot designer errors.
The B1 Battle Droids seen in the first and second Star Wars films lack intelligence and frequently malfunction. C-3PO is notoriously error prone and his probability-based estimates are often wide of the mark.
In the fourth film, A New Hope, R5-D4 (another astromech droid) malfunctions and explodes just as the farmer Owen Lars is about to buy it. Other droids are slow and clunky, such as the GNK Power droid and HURID-327, the groundskeeper at the castle of Maz Kanata in The Force Awakens.
The much feared scenario, whereby robots become so intelligent that they eventually take over, is hard to imagine with this lot.
Perhaps the message from the Star Wars films is that we need to lower our expectations of robot capabilities, in the short term at least. Cars will still crash, mistakes will still be made, regardless of whether humans or robots are doing the work.
Paul Salmon, Professor of Human Factors, University of the Sunshine Coast
This article was originally published on The Conversation. Read the original article.
In this episode of Robots in Depth, Per Sjöborg speaks with Ian Bernstein, the founder of several robotics companies including Sphero. He shares his experience from completing 5 successful rounds of financing, raising 17 million dollars in the 5th one.
Ian also talks about building a world-wide distribution network and the complexity of combining software and hardware development. We then discuss what is happening in robotics and where future successes may come from, including the importance of Kickstarter and Indiegogo.
If you view this episode, you will also learn which day of the week people don’t play with their Sphero :-).
By Changliu Liu, Masayoshi Tomizuka
In modern factories, human workers and robots are two major workforces. For safety concerns, the two are normally separated with robots confined in metal cages, which limits the productivity as well as the flexibility of production lines. In recent years, attention has been directed to remove the cages so that human workers and robots may collaborate to create a human-robot co-existing factory.
Manufacturers are interested in combining human’s flexibility and robot’s productivity in flexible production lines. The potential benefits of industrial co-robots are huge and extensive, e.g. they may be placed in human-robot teams in flexible production lines, where robot arms and human workers cooperate in handling workpieces, and automated guided vehicles (AGV) co-inhabit with human workers to facilitate factory logistics. In the factories of the future, more and more human-robot interactions are anticipated to take place. Unlike traditional robots that work in structured and deterministic environments, co-robots need to operate in highly unstructured and stochastic environments. The fundamental problem is how to ensure that co-robots operate efficiently and safely in dynamic uncertain environments. In this post, we introduce the robot safe interaction system developed in the Mechanical System Control (MSC) lab.
Fig. 1. The factory of the future with human-robot collaborations.
Robot manufacturers including Kuka, Fanuc, Nachi, Yaskawa, Adept and ABB are providing or working on their solutions to the problem. Several safe cooperative robots or co-robots have been released, such as Collaborative Robots CR family from FANUC (Japan), UR5 from Universal Robots (Denmark), Baxter from Rethink Robotics (US), NextAge from Kawada (Japan) and WorkerBot from Pi4_Robotics GmbH (Germany). However, many of these products focus on intrinsic safety, i.e. safety in mechanical design, actuation and low level motion control. Safety during social interactions with humans, which are key to intelligence (including perception, cognition and high level motion planning and control), still needs to be explored.
Technically, it is challenging to design the behavior of industrial co-robots. In order to make the industrial co-robots human-friendly, they should be equipped with the abilities to: collect environmental data and interpret such data, adapt to different tasks and different environments, and tailor itself to the human workers’ needs. For example, during human-robot collaborative assembly shown in the figure below, the robot should be able to predict that once the human puts the two workpieces together, he will need the tool to fasten the assemble. Then the robot should be able to get the tool and hand it over to the human, while avoid colliding with the human.
Fig. 2. Human-robot collaborative assembly.
To achieve such behavior, the challenges lie in (1) the complication of human behaviors, and (2) the difficulty in assurance of real time safety without sacrificing efficiency. The stochastic nature of human motions brings huge uncertainty to the system, making it hard to ensure safety and efficiency.
The robot safe interaction system (RSIS) has been developed in the Mechanical System Control lab, which establishes a methodology to design the robot behavior to achieve safety and efficiency in peer-to-peer human-robot interactions.
As robots need to interact with humans, who have long acquired interactive behaviors, it is natural to let robot mimic human behavior. Human’s interactive behavior can result from either deliberate thoughts or conditioned reflex. For example, if there is a rear-end collision in the front, the driver of a following car may instinctively hit the brake. However, after a second thought, that driver may speed up to cut into the other lane to avoid chain rear-end. The first is a short-term reactive behavior for safety, while the second needs calculation on current conditions, e.g. whether there is enough space to achieve a full stop, whether there is enough gap for a lane change, and whether it is safer to change lane or do a full stop.
A parallel planning and control architecture has been introduced mimicking these kind of behavior, which included both long term and short term motion planners. The long term planner (efficiency controller) emphasizes efficiency and solves a long-term optimal control problem in receding horizons with low sampling rate. The short term planner (safety controller) addresses real time safety by solving a short-term optimal control problem with high sampling rate based on the trajectories planned by the efficiency controller. This parallel architecture also addresses the uncertainties, where the long term planner plans according to the most-likely behavior of others, and the short term planner considers almost all possible movements of others in the short term to ensure safety.
Fig. 3. The parallel planning and control architecture in the robot safe interaction system.
However, the robot motion planning problems in clustered environment are highly nonlinear and non-convex, hence hard to solve in real time. To ensure timely responses to the change of the environment, fast algorithms are developed for real-time computation, e.g. the convex feasible set algorithm (CFS) for the long term optimization, and the safe set algorithm (SSA) for the short term optimization. These algorithms achieve faster computation by convexification of the original non-convex problem, which is assumed to have convex objective functions, but non-convex constraints. The convex feasible set algorithm (CFS) iteratively solves a sequence of sub-problems constrained in convex subsets of the feasible domain. The sequence of solutions will converge to a local optima. It converges in fewer iterations and run faster than generic non-convex optimization solvers such as sequential quadratic programming (SQP) and interior point method (ITP). On the other hand, the safe set algorithm (SSA) transforms the non convex state space constraints to convex control space constraints using the idea of invariant set.
Fig. 4. Illustration of convexification in the CFS algorithm.
With the parallel planner and the optimization algorithms, the robot can interact with the environment safely and finish the tasks efficiently.
Fig. 5. Real time motion planning and control.
We now work on an advanced version of RSIS in the Mechanical System Control lab, the safe and efficient robot collaboration system (SERoCS), which is supported by National Science Foundation (NSF) Award #1734109. In addition to safe motion planning and control algorithms for safe human-robot interactions (HRI), SERoCS also consists of robust cognition algorithms for environment monitoring, optimal task planning algorithms for safe human-robot collaboration. The SERoCS will significantly expand the skill sets of the co-robots and prevent or minimize occurrences of human-robot collision and robot-robot collision during operation, hence enables harmonic human-robot collaboration in the future.
Fig. 6. SERoCS Architecture.
This article was initially published on the BAIR blog, and appears here with the authors’ permission.
| C. Liu, and M. Tomizuka, “Algorithmic safety measures for intelligent industrial co-robots,” in IEEE International Conference on Robotics and Automation (ICRA), 2016. |
| C. Liu, and M. Tomizuka, “Designing the robot behavior for safe human robot interactions”, in Trends in Control and Decision-Making for Human-Robot Collaboration Systems (Y. Wang and F. Zhang (Eds.)). Springer, 2017. |
| C. Liu, and M. Tomizuka, “Real time trajectory optimization for nonlinear robotic systems: Relaxation and convexification”, in Systems & Control Letters, vol. 108, pp. 56-63, Oct. 2017. |
| C. Liu, C. Lin, and M. Tomizuka, “The convex feasible set algorithm for real time optimization in motion planning”, arXiv:1709.00627. |
The Robot Launch global startup competition is over for 2017. We’ve seen startups from all over the world and all sorts of application areas – and we’d like to congratulate the overall winner Semio, and runners up Apellix and Mothership Aeronautics. All three startups met the judges criteria; to be an early stage platform technology in robotics or AI with great impact, large market potential and near term customer pipeline.
Semio from Southern California is a software platform for developing and deploying social robot skills. Ross Mead, founder and CEO of Semio said that “he was greatly looking forward to spending more time with The Robotics Hub, and is excited about the potential for Semio moving forward.”
Apellix from Florida provides software controlled aerial robotic systems that utilize tethered and untethered drones to move workers from harm’s way; such as window washers on skyscrapers (window washing drone, windmill blade cleaning and coating drone)
Robert Dahlstrom, founder and CEO of Apellix said, “As an entrepreneur I strongly believe in startup’s potential to improve lives, create jobs, and make the world a more exciting place. I also know first hand how difficult and challenging a startup can be (an emotional roller coaster ride) and how valuable the work Robot Launch is.”
Mothership Aeronautics from Silicon Valley have a solar powered drone capable of ‘infinity cruise’ where more power is generated than consumed. The drone can perform aerial surveillance and inspection for large scale infrastructures, like pipelines, railways and powerlines. Mothership may also fulfill the ‘warehouse in the sky’ vision that both Amazon and Walmart have tried to patent.
The other awardees are.
And as mentioned in our previous article, the three startups who won the Robohub Choice award were UniExo, BotsAndUs and Northstar Robotics. All the award winners will be featured on Robohub and get access to the Silicon Valley Robotics accelerator program and cowork space, where the award ceremony took place as part of a larger investor/startup showcase.
The Silicon Valley Robotics cowork space is at the newly opened Circuit Launch, and provides more than 30,000 sq ft of hot desks and office spaces with professional prototyping facilities. Access to the space is for interesting robotics, AI, AR/VR and sensor technologies, and can include access to the Silicon Valley Robotics startup accelerator program.
The other startups that pitched on the day were; Vecna, Twisted Field, RoboLoco, Dash Shipping, Tekuma, Sake Robotics and Kinema Systems.
Not all of the startups were from the Bay Area – Dash flew up from LA, and Vecna/Twisted Field from Boston, while Tekuma came from Australia as part of an Australian government startup program.
This latest video from the D’Andrea lab shows a quadrotor performing a looping trajectory while balancing an inverted pendulum at the same time.
The video is pretty self-explanatory and includes lots of the technical details – enjoy!
The work, which will be detailed in an upcoming paper, was done by Julien Kohler, Michael Mühlebach, Dario Brescianini, and Raffaello D’Andrea at ETH Zürich. You can learn more about the Flying Machine Arena here.
Our first few submissions have now arrived! Have a holiday robot video of your own that you’d like to share? Send your submissions to editors@robohub.org.
“I made 2000 ugly holiday cards with a $100k robot arm” by Simone Giertz
“Making Ideas Come True” by Danish Technological Institute
“Hey, Jibo. Welcome home for the holidays.” by Jibo
“Bake Together” and “Decorate Together” by iRobot
Keep them coming! Email us your holiday robot videos at editors@robohub.org!
Josh Tenenbaum, a professor of brain and cognitive sciences at MIT, directs research on the development of intelligence at the Center for Brains, Minds, and Machines, a multiuniversity, multidisciplinary project based at MIT that seeks to explain and replicate human intelligence.
Presenting their work at this year’s Conference on Neural Information Processing Systems, Tenenbaum and one of his students, Jiajun Wu, are co-authors on four papers that examine the fundamental cognitive abilities that an intelligent agent requires to navigate the world: discerning distinct objects and inferring how they respond to physical forces.
By building computer systems that begin to approximate these capacities, the researchers believe they can help answer questions about what information-processing resources human beings use at what stages of development. Along the way, the researchers might also generate some insights useful for robotic vision systems.
“The common theme here is really learning to perceive physics,” Tenenbaum says. “That starts with seeing the full 3-D shapes of objects, and multiple objects in a scene, along with their physical properties, like mass and friction, then reasoning about how these objects will move over time. Jiajun’s four papers address this whole space. Taken together, we’re starting to be able to build machines that capture more and more of people’s basic understanding of the physical world.”
Three of the papers deal with inferring information about the physical structure of objects, from both visual and aural data. The fourth deals with predicting how objects will behave on the basis of that data.
Two-way street
Something else that unites all four papers is their unusual approach to machine learning, a technique in which computers learn to perform computational tasks by analyzing huge sets of training data. In a typical machine-learning system, the training data are labeled: Human analysts will have, say, identified the objects in a visual scene or transcribed the words of a spoken sentence. The system attempts to learn what features of the data correlate with what labels, and it’s judged on how well it labels previously unseen data.
In Wu and Tenenbaum’s new papers, the system is trained to infer a physical model of the world — the 3-D shapes of objects that are mostly hidden from view, for instance. But then it works backward, using the model to resynthesize the input data, and its performance is judged on how well the reconstructed data matches the original data.
For instance, using visual images to build a 3-D model of an object in a scene requires stripping away any occluding objects; filtering out confounding visual textures, reflections, and shadows; and inferring the shape of unseen surfaces. Once Wu and Tenenbaum’s system has built such a model, however, it rotates it in space and adds visual textures back in until it can approximate the input data.
Indeed, two of the researchers’ four papers address the complex problem of inferring 3-D models from visual data. On those papers, they’re joined by four other MIT researchers, including William Freeman, the Perkins Professor of Electrical Engineering and Computer Science, and by colleagues at DeepMind, ShanghaiTech University, and Shanghai Jiao Tong University.
Divide and conquer
The researchers’ system is based on the influential theories of the MIT neuroscientist David Marr, who died in 1980 at the tragically young age of 35. Marr hypothesized that in interpreting a visual scene, the brain first creates what he called a 2.5-D sketch of the objects it contained — a representation of just those surfaces of the objects facing the viewer. Then, on the basis of the 2.5-D sketch — not the raw visual information about the scene — the brain infers the full, three-dimensional shapes of the objects.
“Both problems are very hard, but there’s a nice way to disentangle them,” Wu says. “You can do them one at a time, so you don’t have to deal with both of them at the same time, which is even harder.”
Wu and his colleagues’ system needs to be trained on data that include both visual images and 3-D models of the objects the images depict. Constructing accurate 3-D models of the objects depicted in real photographs would be prohibitively time consuming, so initially, the researchers train their system using synthetic data, in which the visual image is generated from the 3-D model, rather than vice versa. The process of creating the data is like that of creating a computer-animated film.
Once the system has been trained on synthetic data, however, it can be fine-tuned using real data. That’s because its ultimate performance criterion is the accuracy with which it reconstructs the input data. It’s still building 3-D models, but they don’t need to be compared to human-constructed models for performance assessment.
In evaluating their system, the researchers used a measure called intersection over union, which is common in the field. On that measure, their system outperforms its predecessors. But a given intersection-over-union score leaves a lot of room for local variation in the smoothness and shape of a 3-D model. So Wu and his colleagues also conducted a qualitative study of the models’ fidelity to the source images. Of the study’s participants, 74 percent preferred the new system’s reconstructions to those of its predecessors.
All that fall
In another of Wu and Tenenbaum’s papers, on which they’re joined again by Freeman and by researchers at MIT, Cambridge University, and ShanghaiTech University, they train a system to analyze audio recordings of an object being dropped, to infer properties such as the object’s shape, its composition, and the height from which it fell. Again, the system is trained to produce an abstract representation of the object, which, in turn, it uses to synthesize the sound the object would make when dropped from a particular height. The system’s performance is judged on the similarity between the synthesized sound and the source sound.
Finally, in their fourth paper, Wu, Tenenbaum, Freeman, and colleagues at DeepMind and Oxford University describe a system that begins to model humans’ intuitive understanding of the physical forces acting on objects in the world. This paper picks up where the previous papers leave off: It assumes that the system has already deduced objects’ 3-D shapes.
Those shapes are simple: balls and cubes. The researchers trained their system to perform two tasks. The first is to estimate the velocities of balls traveling on a billiard table and, on that basis, to predict how they will behave after a collision. The second is to analyze a static image of stacked cubes and determine whether they will fall and, if so, where the cubes will land.
Wu developed a representational language he calls scene XML that can quantitatively characterize the relative positions of objects in a visual scene. The system first learns to describe input data in that language. It then feeds that description to something called a physics engine, which models the physical forces acting on the represented objects. Physics engines are a staple of both computer animation, where they generate the movement of clothing, falling objects, and the like, and of scientific computing, where they’re used for large-scale physical simulations.
After the physics engine has predicted the motions of the balls and boxes, that information is fed to a graphics engine, whose output is, again, compared with the source images. As with the work on visual discrimination, the researchers train their system on synthetic data before refining it with real data.
In tests, the researchers’ system again outperformed its predecessors. In fact, in some of the tests involving billiard balls, it frequently outperformed human observers as well.