Archive 04.09.2017

Metallica’s European WorldWired tour, which opened to an ecstatic crowd of 15,000 in Copenhagen’s sold-out Royal Arena this Saturday, features a swarm of micro drones flying above the band. Shortly after the band breaks into their hit single “Moth Into Flame”, dozens of micro drones start emerging from the stage, forming a large rotating circle above the stage. As the music builds, more and more drones emerge and join the formation, creating increasingly complex patterns, culminating in a choreography of three interlocking rings that rotate in position.

This show’s debut marks the world’s first autonomous drone swarm performance in a major touring act. Unlike previous drone shows, this performance features indoor drones, flying above performers and right next to throngs of concert viewers in a live event setting. Flying immediately next to audiences creates a more intimate effect than outdoor drone shows. The same closeness also allows the creation of moving, three-dimensional sculptures like the ones seen in the video — an effect further enhanced by Metallica’s 360-degree stage setup, with concert viewers on all sides.

Flying drones close to and around people in such a setting is challenging. Unlike outdoors, indoor drones cannot rely on GPS signals, which are severely degraded in indoor settings and do not offer the required accuracy for autonomous drone navigation on stage. The safety aspects of flying dozens of drones close to crowds in the high-pressure, live-event environment impose further challenges. Robustness to the uncertainties caused by changing show conditions in a touring setting as well as variation in the drone systems’ components and sensors, including the hundreds of motors powering the drones, is another necessary condition for this drone show system.

“It’s all about safety and reliability first”, says Raffaello D’Andrea, founder of the company behind the drones used in the Metallica show, Verity Studios (full disclosure: I’m a co-founder). D’Andrea knows what he is talking about: In work with his previous company, which was snatched up by e-commerce giant Amazon for an eye-watering 775M USD in 2012, D’Andrea and his team created fleets of autonomous warehousing robots, moving inventory through the warehouse around the clock. That company, which has since been renamed Amazon Robotics, now operates up to 10,000 robots — in a single warehouse.

How was this achieved?

In a nutshell: Verity Studios’ drone show system is an advanced show automation system that uses distributed AI, robotics, and sophisticated algorithms to achieve the level of robust performance and safety required by the live entertainment industry. With a track record of >7,000 autonomous flights on Broadway, achieved with its larger Stage Flyer drones during 398 live shows, Verity Studios is no newcomer to this industry.

Many elements are needed to create a touring drone show; the drones themselves are just one aspect. Verity’s drones are autonomous, supervised by a human operator, who does not control drone motions individually. Instead, the operator only issues high-level commands such as “takeoff” or “land”, monitors the motions of multiple drones at a time, and reacts to anomalies. In other words, Verity’s advanced automation system takes over the role of multiple human pilots that would be required with standard, remote-controlled drones. The drones are flying mobile robots that navigate autonomously, piloting themselves, under human supervision. The autonomous drones’ motions and their lighting design are choreographed by Verity’s creative staff.

To navigate autonomously, drones require a reliable method for determining their position in space. As mentioned above, while drones can use GPS for their autonomous navigation in an outdoor setting, GPS is not a viable option indoors: GPS signals degrade close to large structures (e.g., tall buildings) and are usually not available, or severely degraded, in indoor environments. Since degraded GPS may result in unreliable or unsafe conditions for autonomous flight, the Verity drones use proprietary indoor localization technology.

It is the combination of a reliable indoor positioning system with intelligent autonomous drones and a suitable operator interface that allows the single operator of the Metallica show to simultaneously control the coordinated movement of many drones. This pilot-less approach is not merely a matter of increasing efficiency and effectiveness (who wants to have dozens of pilots on staff), but also a key safety requirement: Pilot errors have been an important contributing factor in dozens of documented drone accidents at live events. Safety risks rapidly increase as the number of drones increases, resulting in more complex flight plans and higher risks of mid-air collisions. Autonomous control allows safer operation of multiple drones than remote control by human pilots, especially when operating in a reduced airspace envelope.

Verity’s system also had to be engineered for safety in spite of other potential failures, including wireless interference, hardware or software component failures, power outages, or malicious disruption/hacking attacks. In its 398-show run on Broadway, the biggest challenge to safety turned out to be another factor: Human error. While operated by theater staff on Broadway, Verity’s system correctly identified human errors on five occasions and prevented the concerned drones from taking flight (on these occasions, the show continued with six or seven instead of the show’s planned eight drones; only one show proceeded without any drones as a safety precaution, i.e., the drone show’s “uptime” was 99.7%). As my colleagues and I have outlined in a recently published overview document on best practices for drone shows, when using drones at live event safety is a hard requirement.

Another key element for Verity’s show creation process are drone authoring tools. Planning shows like the Metallica performance requires tools for the efficient creation of trajectories for large numbers of drones. The trajectories must account for the drones’ actual flight dynamics, considering actuator limitations, as well as for aerodynamic effects, such as air turbulence or lift. Drone motions generated by these tools need to be collision-free and allow for emergency maneuvers. To create compelling effects, drone authoring tools also need to allow extracting all of the dynamic performance the drones are capable of — another area that D’Andrea’s team has gained considerable experience with prior to founding Verity Studios, in this case as part of research at the Swiss Federal Institute of Technology’s Flying Machine Arena.

Creating a compelling drone show requires more than the drone show system itself. For this tour, Verity Studios partnered with the world’s leading stage automation company TAIT Towers to integrate the drones into the stage floor as well as tackling a series of other technical challenges related to this touring show.

While technology is the key enabler, the starting point and the key driver of Verity’s shows are non-technological. Instead, the show is driven by the show designers’ creative intent. This comprises defining the role of show drones for the performance at hand as well as determining their integration into the visual and musical motifs of the show’s creative concept. For Metallica, the drones’ flight trajectories and lighting were created by Verity’s choreography team, incorporating feedback from Metallica’s production team and the band.

Metallica’s WorldWired tour
Metallica’s WorldWired Tour is their first worldwide tour after the World Magnetic Tour six years ago. The tour’s currently published European leg runs until 11 May 2018, with all general tickets sold out.

Further Robohub reading

• Drone shows: Creative potential and best practices
• Ben Einstein’s great tear-down of D’Andrea’s Amazon robots (although I disagree with Ben’s conclusion that the bots are “cooler than a flying drone” maybe Ben hasn’t seen Verity’s Lucie drones yet? You be the judge.)
• Verity’s 7,000 autonomous drone flights on Broadway

Some images for your viewing pleasure

Recording electrical signals from inside a neuron in the living brain can reveal a great deal of information about that neuron’s function and how it coordinates with other cells in the brain. However, performing this kind of recording is extremely difficult, so only a handful of neuroscience labs around the world do it.

To make this technique more widely available, MIT engineers have now devised a way to automate the process, using a computer algorithm that analyzes microscope images and guides a robotic arm to the target cell.

This technology could allow more scientists to study single neurons and learn how they interact with other cells to enable cognition, sensory perception, and other brain functions. Researchers could also use it to learn more about how neural circuits are affected by brain disorders.

“Knowing how neurons communicate is fundamental to basic and clinical neuroscience. Our hope is this technology will allow you to look at what’s happening inside a cell, in terms of neural computation, or in a disease state,” says Ed Boyden, an associate professor of biological engineering and brain and cognitive sciences at MIT, and a member of MIT’s Media Lab and McGovern Institute for Brain Research.

Boyden is the senior author of the paper, which appears in the Aug. 30 issue of Neuron. The paper’s lead author is MIT graduate student Ho-Jun Suk.

Precision guidance

For more than 30 years, neuroscientists have been using a technique known as patch clamping to record the electrical activity of cells. This method, which involves bringing a tiny, hollow glass pipette in contact with the cell membrane of a neuron, then opening up a small pore in the membrane, usually takes a graduate student or postdoc several months to learn. Learning to perform this on neurons in the living mammalian brain is even more difficult.

There are two types of patch clamping: a “blind” (not image-guided) method, which is limited because researchers cannot see where the cells are and can only record from whatever cell the pipette encounters first, and an image-guided version that allows a specific cell to be targeted.

Five years ago, Boyden and colleagues at MIT and Georgia Tech, including co-author Craig Forest, devised a way to automate the blind version of patch clamping. They created a computer algorithm that could guide the pipette to a cell based on measurements of a property called electrical impedance — which reflects how difficult it is for electricity to flow out of the pipette. If there are no cells around, electricity flows and impedance is low. When the tip hits a cell, electricity can’t flow as well and impedance goes up.

Once the pipette detects a cell, it can stop moving instantly, preventing it from poking through the membrane. A vacuum pump then applies suction to form a seal with the cell’s membrane. Then, the electrode can break through the membrane to record the cell’s internal electrical activity.

The researchers achieved very high accuracy using this technique, but it still could not be used to target a specific cell. For most studies, neuroscientists have a particular cell type they would like to learn about, Boyden says.

“It might be a cell that is compromised in autism, or is altered in schizophrenia, or a cell that is active when a memory is stored. That’s the cell that you want to know about,” he says. “You don’t want to patch a thousand cells until you find the one that is interesting.”

To enable this kind of precise targeting, the researchers set out to automate image-guided patch clamping. This technique is difficult to perform manually because, although the scientist can see the target neuron and the pipette through a microscope, he or she must compensate for the fact that nearby cells will move as the pipette enters the brain.

“It’s almost like trying to hit a moving target inside the brain, which is a delicate tissue,” Suk says. “For machines it’s easier because they can keep track of where the cell is, they can automatically move the focus of the microscope, and they can automatically move the pipette.”

By combining several imaging processing techniques, the researchers came up with an algorithm that guides the pipette to within about 25 microns of the target cell. At that point, the system begins to rely on a combination of imagery and impedance, which is more accurate at detecting contact between the pipette and the target cell than either signal alone.

The researchers imaged the cells with two-photon microscopy, a commonly used technique that uses a pulsed laser to send infrared light into the brain, lighting up cells that have been engineered to express a fluorescent protein.

Using this automated approach, the researchers were able to successfully target and record from two types of cells — a class of interneurons, which relay messages between other neurons, and a set of excitatory neurons known as pyramidal cells. They achieved a success rate of about 20 percent, which is comparable to the performance of highly trained scientists performing the process manually.

Unraveling circuits

This technology paves the way for in-depth studies of the behavior of specific neurons, which could shed light on both their normal functions and how they go awry in diseases such as Alzheimer’s or schizophrenia. For example, the interneurons that the researchers studied in this paper have been previously linked with Alzheimer’s. In a recent study of mice, led by Li-Huei Tsai, director of MIT’s Picower Institute for Learning and Memory, and conducted in collaboration with Boyden, it was reported that inducing a specific frequency of brain wave oscillation in interneurons in the hippocampus could help to clear amyloid plaques similar to those found in Alzheimer’s patients.

“You really would love to know what’s happening in those cells,” Boyden says. “Are they signaling to specific downstream cells, which then contribute to the therapeutic result? The brain is a circuit, and to understand how a circuit works, you have to be able to monitor the components of the circuit while they are in action.”

This technique could also enable studies of fundamental questions in neuroscience, such as how individual neurons interact with each other as the brain makes a decision or recalls a memory.

Bernardo Sabatini, a professor of neurobiology at Harvard Medical School, says he is interested in adapting this technique to use in his lab, where students spend a great deal of time recording electrical activity from neurons growing in a lab dish.

“It’s silly to have amazingly intelligent students doing tedious tasks that could be done by robots,” says Sabatini, who was not involved in this study. “I would be happy to have robots do more of the experimentation so we can focus on the design and interpretation of the experiments.”

To help other labs adopt the new technology, the researchers plan to put the details of their approach on their web site, autopatcher.org.

Other co-authors include Ingrid van Welie, Suhasa Kodandaramaiah, and Brian Allen. The research was funded by Jeremy and Joyce Wertheimer, the National Institutes of Health (including the NIH Single Cell Initiative and the NIH Director’s Pioneer Award), the HHMI-Simons Faculty Scholars Program, and the New York Stem Cell Foundation-Robertson Award.

In this episode, MeiXing Dong talks with Leon Kuperman, CTO of CUJO, about cybersecurity threats and how to guard against them. They discuss how CUJO, a smart hardware firewall, helps protect the home against online threats.

Leon Kuperman

Leon Kuperman is the CTO of CUJO IoT Security. He co-founded ZENEDGE, an enterprise web application security platform, and Truition Inc. He is also the CTO of BIDZ.com.

Links

• Download mp3 (14.4 MB)
• CUJO Website
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

Mike Salem from Udacity’s Robotics Nanodegree is hosting a series of interviews with professional roboticists as part of their free online material.

This week we’re featuring Mike’s interview with Cory Kidd. Dr. Kidd is focused on innovating within the rapidly changing healthcare technology market. He is the founder and CEO of Catalia Health, a company that delivers patient engagement across a variety of chronic conditions.

You can find all the interviews here. We’ll be posting them regularly on Robohub.

August fundings totaled $369 million but the number of August transactions, seven, was down from previous months, eg: both July and June had 19 fundings each. Acquisitions, on the other hand, remained steady with a big one pending: Snap has been negotiating all month to acquire Chinese drone startup Zero Zero Robotics for around $150M.

Fundings

1. Auris Medical Robotics, the Silicon Valley startup headed by Dr. Frederic H. Moll who previously co-founded Hansen Medical and Intuitive Surgical, raised $280 million in a Series D round led by Coatue Management and included earlier investors Mithril Capital Management, Lux Capital, and Highland Capital. Auris has raised a total of $530 million and is developing targeted, minimally invasive robotic-assisted therapies that treat only the diseased cells in order to prevent the progression of a patient’s illness. Lung cancer is the first disease they are targeting.
2. Oryx Vision, an Israeli startup, raised $50 million in a round led by Third Point Ventures and WRV with participation by Union Tech Ventures. They all join existing investors Bessemer Venture Partners, Maniv Mobility, and Trucks VC, a VC firm focused on the future of transportation. The company has raised a total of $67 million to date. Oryx is developing a LiDAR for self-driving automobiles using microscopic antennas to detect the light frequencies. The tiny antennas are made of silicon which allows them to put thousands in one sensor thereby lowering the cost of LiDAR distancing. The advantage is increased range and sensitivity for an autonomous vehicle that needs to know exactly what is surrounding it and what those things are doing and can see through fog and not get blinded by bright sunlight.
3. TuSimple, a Chinese startup developing driverless technologies for the trucking industry, raised $20 million in a Series B funding round led by Nvidia with participation by Sina. Nvidia will own a 3% stake in TuSimple while the startup will support the development of the Nvidia’s artificial intelligence computing platform for self-driving vehicles, Drive PX2.
4. Atlas Dynamics, a Latvian/Israeli drone startup, raised $8 million from investment groups in Israel and in Asia. The 3-rotor Atlas Pro drone operates autonomously with interchangeable payloads and offers 55 minutes of flight time.
5. Common Sense Robotics, an Israeli warehouse fulfillment robotics startup, raised $6 million from Aleph VC and Innovation Endeavors. CommonSense is developing small urban, automated spaces that combine the benefits of local distribution with the economics of automated fulfillment. In big cities these ‘micro-centers’ would receive, stock, and package merchandise of participating vendors based on predictive algorithms. Vendors would then arrange last-mile delivery solutions.
6. Sky-Futures, a London-based industrial inspection services with drones startup, raised $4 million in funding from Japanese giant Mitsui & Co. The announcement came as part of Theresa May’s just-concluded trip to Japan. Sky Futures and Mitsui plan to provide inspections and other services to Mitsui’s clients across a range of sectors. Mitsui, a trading, investment and service company, has 139 offices in 66 countries.
7. Ambient Intelligence Technology, a Japanese underwater drone manufacturer spin-off from the University of Tsukuba, raised $1.93 million from Beyond Next Ventures and Mitsui Sumitomo Insurance Venture Capital, SMBC Venture Capital, and Freebit Investment. Ambient’s ROVs can operate for prolonged periods of autonomous operation at depths of 300 meters.

Acquisitions

1. Dupont Pioneer has acquired farm management software platform startup Granular for $300 million. San Francisco-based Granular’s farm management software helps farmers run more profitable businesses by enabling them to manage their operations and analyze their financials for each of their fields in real time and to create reports for third parties like landowners and banks. Last year they partnered with the American Farm Bureau Insurance Services to streamline crop insurance data collection and reporting and also have a cross-marketing arrangement with Deere.
2. L3 Technologies acquired Massachusetts-based OceanServer Technology for an undisclosed amount. “OceanServer Technology positions L3 to support the U.S. Navy’s vision for the tactical employment of UUVs. This acquisition also enhances our technological capabilities and strengthens our position in growth areas where we see compelling opportunity,” said Michael T. Strianese, L3’s Chairman and CEO. “As a leading innovator and developer of UUVs, OceanServer Technology provides L3 with a new growth platform that is aligned with the U.S. Navy’s priorities.”
3. KB Medical, SA, a Swiss medical robotics startup, was acquired by Globus Medical, a musculoskeletal solutions manufacturer, for an undisclosed amount. This is the 2nd acquisition of a robotics startup by Globus. They acquired Excelsius Robotics in 2014. “The addition of KB Medical will enable Globus Medical to accelerate, enhance and expand our product portfolio in imaging, navigation and robotics. KB Medical’s experienced team of technology development professionals, its strong IP portfolio, and shared philosophy for robotic solutions in medicine strengthen Globus Medical’s position in this strategic area,” said Dave Demski of Emerging Technologies.
4. Jenoptik, a Germany-based laser components manufacturer of vision systems for automation and robotics, acquired Michigan-based Five Lakes Automation, an integrator and manufacturer of robotic material handling systems, for an undisclosed amount.
5. Honeybee Robotics, the Brooklyn-based robotic space systems provider, was acquired by Ensign-Bickford for an undisclosed amount. Ensign-Bickford is a privately held 181-year-old contractor and supplier of space launch vehicles and systems. “The timing is great,” said Kiel Davis, President of Honeybee Robotics. “Honeybee has a range of new spacecraft motion control and robotics products coming to market. And EBI has the experience and resources to help us scale up and optimize our production operations so that we can meet the needs of our customers today and in the near future.”

IPOs

1. Duke Robotics, a Florida and Israeli developer of advanced robotic systems that provide troops with aerial support and other technologies developed in Israel, has filed and been qualified for a stock offering of up to $15 million under SEC Tier II Reg A+ which allows anyone, not just wealthy investors, to be able to purchase stock from approved equity crowdfunding offers.

Failures

1. C&R Robotics (KR)
2. EZ-Robotics (CN)

by Thomas Kochan, MIT Sloan School of Management and Lee Dyer, Cornell University

The technologies driving artificial intelligence are expanding exponentially, leading many technology experts and futurists to predict machines will soon be doing many of the jobs that humans do today. Some even predict humans could lose control over their future.

While we agree about the seismic changes afoot, we don’t believe this is the right way to think about it. Approaching the challenge this way assumes society has to be passive about how tomorrow’s technologies are designed and implemented. The truth is there is no absolute law that determines the shape and consequences of innovation. We can all influence where it takes us.

Thus, the question society should be asking is: “How can we direct the development of future technologies so that robots complement rather than replace us?”

The Japanese have an apt phrase for this: “giving wisdom to the machines.” And the wisdom comes from workers and an integrated approach to technology design, as our research shows.

Lessons from history

There is no question coming technologies like AI will eliminate some jobs, as did those of the past.

More than half of the American workforce was involved in farming in the 1890s, back when it was a physically demanding, labor-intensive industry. Today, thanks to mechanization and the use of sophisticated data analytics to handle the operation of crops and cattle, fewer than 2 percent are in agriculture, yet their output is significantly higher.

But new technologies will also create new jobs. After steam engines replaced water wheels as the source of power in manufacturing in the 1800s, the sector expanded sevenfold, from 1.2 million jobs in 1830 to 8.3 million by 1910. Similarly, many feared that the ATM’s emergence in the early 1970s would replace bank tellers. Yet even though the machines are now ubiquitous, there are actually more tellers today doing a wider variety of customer service tasks.

So trying to predict whether a new wave of technologies will create more jobs than it will destroy is not worth the effort, and even the experts are split 50-50.

It’s particularly pointless given that perhaps fewer than 5 percent of current occupations are likely to disappear entirely in the next decade, according to a detailed study by McKinsey.

Instead, let’s focus on the changes they’ll make to how people work.

It’s about tasks, not jobs

To understand why, it’s helpful to think of a job as made up of a collection of tasks that can be carried out in different ways when supported by new technologies.

And in turn, the tasks performed by different workers – colleagues, managers and many others – can also be rearranged in ways that make the best use of technologies to get the work accomplished. Job design specialists call these “work systems.”

One of the McKinsey study’s key findings was that about a third of the tasks performed in 60 percent of today’s jobs are likely to be eliminated or altered significantly by coming technologies. In other words, the vast majority of our jobs will still be there, but what we do on a daily basis will change drastically.

To date, robotics and other digital technologies have had their biggest effects on mostly routine tasks like spell-checking and those that are dangerous, dirty or hard, such as lifting heavy tires onto a wheel on an assembly line. Advances in AI and machine learning will significantly expand the array of tasks and occupations affected.

Creating an integrated strategy

We have been exploring these issues for years as part of our ongoing discussions on how to remake labor for the 21st century. In our recently published book, “Shaping the Future of Work: A Handbook for Change and a New Social Contract,” we describe why society needs an integrated strategy to gain control over how future technologies will affect work.

And that strategy starts with helping define the problems humans want new technologies to solve. We shouldn’t be leaving this solely to their inventors.

Fortunately, some engineers and AI experts are recognizing that the end users of a new technology must have a central role in guiding its design to specify which problems they’re trying to solve.

The second step is ensuring that these technologies are designed alongside the work systems with which they will be paired. A so-called simultaneous design process produces better results for both the companies and their workers compared with a sequential strategy – typical today – which involves designing a technology and only later considering the impact on a workforce.

An excellent illustration of simultaneous design is how Toyota handled the introduction of robotics onto its assembly lines in the 1980s. Unlike rivals such as General Motors that followed a sequential strategy, the Japanese automaker redesigned its work systems at the same time, which allowed it to get the most out of the new technologies and its employees. Importantly, Toyota solicited ideas for improving operations directly from workers.

In doing so, Toyota achieved higher productivity and quality in its plants than competitors like GM that invested heavily in stand-alone automation before they began to alter work systems.

Similarly, businesses that tweaked their work systems in concert with investing in IT in the 1990s outperformed those that didn’t. And health care companies like Kaiser Permanente and others learned the same lesson as they introduced electronic medical records over the past decade.

Each example demonstrates that the introduction of a new technology does more than just eliminate jobs. If managed well, it can change how work is done in ways that can both increase productivity and the level of service by augmenting the tasks humans do.

Worker wisdom

But the process doesn’t end there. Companies need to invest in continuous training so their workers are ready to help influence, use and adapt to technological changes. That’s the third step in getting the most out of new technologies.

And it needs to begin before they are introduced. The important part of this is that workers need to learn what some are calling “hybrid” skills: a combination of technical knowledge of the new technology with aptitudes for communications and problem-solving.

Companies whose workers have these skills will have the best chance of getting the biggest return on their technology investments. It is not surprising that these hybrid skills are now in high and growing demand and command good salaries.

None of this is to deny that some jobs will be eliminated and some workers will be displaced. So the final element of an integrated strategy must be to help those displaced find new jobs and compensate those unable to do so for the losses endured. Ford and the United Auto Workers, for example, offered generous early retirement benefits and cash severance payments in addition to retraining assistance when the company downsized from 2007 to 2010.

Examples like this will need to become the norm in the years ahead. Failure to treat displaced workers equitably will only widen the gaps between winners and losers in the future economy that are now already all too apparent.

In sum, companies that engage their workforce when they design and implement new technologies will be best-positioned to manage the coming AI revolution. By respecting the fact that today’s workers, like those before them, understand their jobs better than anyone and the many tasks they entail, they will be better able to “give wisdom to the machines.”

Thomas Kochan, Professor of Management, MIT Sloan School of Management and Lee Dyer, Professor Emeritus of Human Resource Studies and Research Fellow, Center for Advanced Human Resource Studies (CAHRS), Cornell University

This article was originally published on The Conversation. Read the original article.

by Adam Conner-Simons & Rachel Gordon

Despite what you might see in movies, today’s robots are still very limited in what they can do. They can be great for many repetitive tasks, but their inability to understand the nuances of human language makes them mostly useless for more complicated requests.

For example, if you put a specific tool in a toolbox and ask a robot to “pick it up,” it would be completely lost. Picking it up means being able to see and identify objects, understand commands, recognize that the “it” in question is the tool you put down, go back in time to remember the moment when you put down the tool, and distinguish the tool you put down from other ones of similar shapes and sizes.

Recently researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have gotten closer to making this type of request easier: In a new paper, they present an Alexa-like system that allows robots to understand a wide range of commands that require contextual knowledge about objects and their environments. They’ve dubbed the system “ComText,” for “commands in context.”

The toolbox situation above was among the types of tasks that ComText can handle. If you tell the system that “the tool I put down is my tool,” it adds that fact to its knowledge base. You can then update the robot with more information about other objects and have it execute a range of tasks like picking up different sets of objects based on different commands.

“Where humans understand the world as a collection of objects and people and abstract concepts, machines view it as pixels, point-clouds, and 3-D maps generated from sensors,” says CSAIL postdoc Rohan Paul, one of the lead authors of the paper. “This semantic gap means that, for robots to understand what we want them to do, they need a much richer representation of what we do and say.”

The team tested ComText on Baxter, a two-armed humanoid robot developed for Rethink Robotics by former CSAIL director Rodney Brooks.

The project was co-led by research scientist Andrei Barbu, alongside research scientist Sue Felshin, senior research scientist Boris Katz, and Professor Nicholas Roy. They presented the paper at last week’s International Joint Conference on Artificial Intelligence (IJCAI) in Australia.

How it works

Things like dates, birthdays, and facts are forms of “declarative memory.” There are two kinds of declarative memory: semantic memory, which is based on general facts like the “sky is blue,” and episodic memory, which is based on personal facts, like remembering what happened at a party.

Most approaches to robot learning have focused only on semantic memory, which obviously leaves a big knowledge gap about events or facts that may be relevant context for future actions. ComText, meanwhile, can observe a range of visuals and natural language to glean “episodic memory” about an object’s size, shape, position, type and even if it belongs to somebody. From this knowledge base, it can then reason, infer meaning and respond to commands.

“The main contribution is this idea that robots should have different kinds of memory, just like people,” says Barbu. “We have the first mathematical formulation to address this issue, and we’re exploring how these two types of memory play and work off of each other.”

With ComText, Baxter was successful in executing the right command about 90 percent of the time. In the future, the team hopes to enable robots to understand more complicated information, such as multi-step commands, the intent of actions, and using properties about objects to interact with them more naturally.

For example, if you tell a robot that one box on a table has crackers, and one box has sugar, and then ask the robot to “pick up the snack,” the hope is that the robot could deduce that sugar is a raw material and therefore unlikely to be somebody’s “snack.”

By creating much less constrained interactions, this line of research could enable better communications for a range of robotic systems, from self-driving cars to household helpers.

“This work is a nice step towards building robots that can interact much more naturally with people,” says Luke Zettlemoyer, an associate professor of computer science at the University of Washington who was not involved in the research. “In particular, it will help robots better understand the names that are used to identify objects in the world, and interpret instructions that use those names to better do what users ask.”

The work was funded, in part, by the Toyota Research Institute, the National Science Foundation, the Robotics Collaborative Technology Alliance of the U.S. Army, and the Air Force Research Laboratory.