Kushal Kedia (left) and Prithwish Dan (right) are members of the development team behind RHyME, a system that allows robots to learn tasks by watching a single how-to video.
By Louis DiPietro
Cornell researchers have developed a new robotic framework powered by artificial intelligence – called RHyME (Retrieval for Hybrid Imitation under Mismatched Execution) – that allows robots to learn tasks by watching a single how-to video. RHyME could fast-track the development and deployment of robotic systems by significantly reducing the time, energy and money needed to train them, the researchers said.
“One of the annoying things about working with robots is collecting so much data on the robot doing different tasks,” said Kushal Kedia, a doctoral student in the field of computer science and lead author of a corresponding paper on RHyME. “That’s not how humans do tasks. We look at other people as inspiration.”
Kedia will present the paper, One-Shot Imitation under Mismatched Execution, in May at the Institute of Electrical and Electronics Engineers’ International Conference on Robotics and Automation, in Atlanta.
Home robot assistants are still a long way off – it is a very difficult task to train robots to deal with all the potential scenarios that they could encounter in the real world. To get robots up to speed, researchers like Kedia are training them with what amounts to how-to videos – human demonstrations of various tasks in a lab setting. The hope with this approach, a branch of machine learning called “imitation learning,” is that robots will learn a sequence of tasks faster and be able to adapt to real-world environments.
“Our work is like translating French to English – we’re translating any given task from human to robot,” said senior author Sanjiban Choudhury, assistant professor of computer science in the Cornell Ann S. Bowers College of Computing and Information Science.
This translation task still faces a broader challenge, however: Humans move too fluidly for a robot to track and mimic, and training robots with video requires gobs of it. Further, video demonstrations – of, say, picking up a napkin or stacking dinner plates – must be performed slowly and flawlessly, since any mismatch in actions between the video and the robot has historically spelled doom for robot learning, the researchers said.
“If a human moves in a way that’s any different from how a robot moves, the method immediately falls apart,” Choudhury said. “Our thinking was, ‘Can we find a principled way to deal with this mismatch between how humans and robots do tasks?’”
RHyME is the team’s answer – a scalable approach that makes robots less finicky and more adaptive. It trains a robotic system to store previous examples in its memory bank and connect the dots when performing tasks it has viewed only once by drawing on videos it has seen. For example, a RHyME-equipped robot shown a video of a human fetching a mug from the counter and placing it in a nearby sink will comb its bank of videos and draw inspiration from similar actions – like grasping a cup and lowering a utensil.
RHyME paves the way for robots to learn multiple-step sequences while significantly lowering the amount of robot data needed for training, the researchers said. They claim that RHyME requires just 30 minutes of robot data; in a lab setting, robots trained using the system achieved a more than 50% increase in task success compared to previous methods.
“This work is a departure from how robots are programmed today. The status quo of programming robots is thousands of hours of tele-operation to teach the robot how to do tasks. That’s just impossible,” Choudhury said. “With RHyME, we’re moving away from that and learning to train robots in a more scalable way.”
This research was supported by Google, OpenAI, the U.S. Office of Naval Research and the National Science Foundation.
One-Shot Imitation under Mismatched Execution, Kushal Kedia, Prithwish Dan, Angela Chao, Maximus Adrian Pace, Sanjiban Choudhury.
Photo credit: Muntaka Chasant, reproduced under a CC BY-SA 4.0 license.
By Kaja Šeruga
Just outside the historic German town of Goslar, a sprawling industrial complex receives an endless stream of discarded electronics. On arrival, this electronic waste is laboriously prepared for recycling.
Electrocycling GmbH is one of the largest e-waste recycling facilities in Europe. Every year, it processes up to 80 000 tonnes of electronic waste, which comes in all shapes and forms.
Despite an impressive array of machinery, more than half of the site’s employees manually prepare the discarded items for recycling. They do this by sorting the incoming waste and removing batteries, which are a fire hazard and a major challenge in e-waste recycling.
“There are more and more devices, they are getting smaller, and they all contain lithium batteries, some of which are permanently installed, soldered or glued in place,” said Hannes Fröhlich, Electrocycling’s managing director.
“It’s not a dream job, dismantling these appliances every day with hammers and pliers. I think we can do better.”
Some of these tedious tasks could be performed by robots. However, the problem is that every time there is a change in the product or the process, the hardware and software need to be restructured. This can be costly and time-consuming.
To address this issue, an EU-funded research initiative named ReconCycle has managed to automate the process by creating robots that can reconfigure themselves for different tasks.
Researchers from Slovenia, Germany and Italy worked together on this issue at the Jožef Stefan Institute, Slovenia’s leading research facility, from 2020 to 2024.
The team developed adaptable AI-supported robots that are able to remove batteries from smoke detectors and radiator heat metres.
These two products can be found in most households and are replaced every five to eight years, creating large amounts of waste.
“The main challenge is that there are so many different versions of each device. Just think how many different remote controls there are,” said Dr Aleš Ude. He is head of the Department of Automatics, Biocybernetics and Robotics at the Jožef Stefan Institute and coordinates the ReconCycle research team.
In industrial settings, robots are usually programmed for one specific task, repeating exactly the same series of movements in a predictable environment.
Instead, the researchers set out to create a robot that can adapt to many different tasks, using state-of-the-art AI.
“We wanted to expand robotics, introduce robots where there aren’t any yet,” Ude said.
Working with Electrocycling, Ude’s international research team created an adaptable robotic work cell. This is a workspace that consists of at least one robot, its tools and equipment, and its controller.
The novelty here is that this closed system autonomously adapts itself to various tasks, with the help of complex AI-driven software and modular hardware that can be quickly reconfigured. It also uses soft components like SoftHand, a human-like hand that can manipulate objects with great precision.
There are also safety features like collaborative robots and emergency stop buttons.
International collaboration was crucial in securing the right expertise, said Ude.
“Robotics is very interdisciplinary, so it’s difficult to find the right partners in one country.”
Thankfully, the new robots are arriving just at the right time, as the amount of e-waste produced every year continues to grow. Almost 5 million tonnes of e-waste are produced in the EU each year, amounting to about 11 kilograms per person. Less than 40% of that is recycled, the European Parliament has warned.
Globally, around 62 million tonnes of e-waste were produced in 2022 alone, enough to fill 1.5 million 40-tonne trucks, according to UN data. Even more worryingly, the amount of e-waste is rising five times faster than the amount that is being recycled.
The EU is working to reduce e-waste through the Waste from Electrical and Electronic Equipment Directive, which sets the standards for collection and recycling.
The work of Ude’s team is also aligned with the EU’s digital strategy, which encourages the use of AI in manufacturing to improve efficiency and help achieve climate neutrality by 2050.
E-waste also has serious economic implications. An estimated €84 billion is lost each year when valuable metals like copper, iron and gold are discarded instead of being reused, according to the UN’s global e-waste monitor.
At Electrocycling, 80% of the e-waste is recovered as raw materials, such as iron, zinc, gold, silver and palladium – some 35 materials in all.
“People need to understand that this is not just waste, but also raw materials that need to be recycled and kept in circulation, both for economic efficiency and a reduction of CO2,” said Fröhlich.
New technology can make it even more efficient, and Fröhlich sees a lot of potential in it.
“I was surprised by how far the technology and AI have already come,” he said. “They even recreated a human hand for the robot.”
Ude hopes to continue working with Electrocycling to improve e-waste solutions further. The hope is also that adaptable robots which can handle changing environments will have applications far beyond e-waste recycling.
Given more time and development, these robots could even handle general housekeeping, or support carers in senior homes, said Ude.
“Robotics could be of great help in such areas.”
This article was originally published in Horizon, the EU Research and Innovation magazine.