All posts by Cornell University

Andrew James (from left), Neil Morrison, Natalia Kurz and Michael Neiss work on a prototype of their weed-killing robot ahead of The Farm Robotics Challenge, which they won on May 21.

By Holly Hartigan

A team of Cornell undergraduates beat 95 other teams to take the grand prize at The Farm Robotics Challenge with their invention: an autonomous robot that kills weeds with electricity.

Their robot can travel through a vineyard or orchard without a human operator, zapping weeds with a small amount of electricity, saving labor and energy and preventing crop loss, without the use of herbicides.

Led by Andrew James, an agricultural sciences major in the College of Agriculture and Life Sciences (CALS), the team of agricultural specialists and engineers studied the existing electrical weeding technology, developed their own low-energy system and built a working prototype over the course of four intense months.

Natalia Kurz, a biological engineering major in CALS, said the project required a lot of late nights. “There were fears for us, like, was it just going to be for nothing?”

Now, James and his co-founders are using the $50,000 grand prize to form a company – Rootline Robotics – to continue working on the robot. Agricultural technology firm Reservoir sponsored the award and will host the startup at its incubator in Sonoma, California.

“I’ve always been interested in building a startup within the ag-robotics space,” James said. “So after winning this competition and seeing all the amazing support from so many different industry stakeholders in this really exciting collaboration, it makes a lot of sense to keep going.”

The problem of weeds

Weeds are a huge challenge in orchards and vineyards because they steal water and nutrients, especially in the spring, according to Steve Selin, owner of South Hill Cider in Ithaca.

“The weed pressure is very strong, and the grasses grow right up to the trees because we can’t afford to weed whack or mulch them as much as we would like to in order to control them,” he said. “If the trees get really stressed out, they’ll just drop the fruit.”

To control weeds, organic growers typically employ string trimming, mowing and mulching, which are all very labor intensive. Existing electric weeders require an operator, consume a lot of energy and cost $150,000, on average, which is out of reach for most growers, James said.

Selin provided feedback to the students as they developed their robot and said he is excited to try out Rootline’s new technology.

“In May and June, if you could do something to knock the weeds back enough that they’re not going to compete with the trees, then the rest of the year you wouldn’t have to worry about it,” he said. “You can let them come back and have positive impacts, like shading the soil, which would help the soil microbes to have a healthier ecosystem.”

The interdisciplinary nature of robots

“Robotics is many different systems in one,” Kurz said. “It is interdisciplinary by definition.”

Next, the Rootline team will work on improving its technology and validating it with growers before bringing it to market, all while Kurz and Neiss finish their degrees.

Neiss said the way the team came together felt a bit like destiny.

“If you have many different minds in one room that gives you the ability to reach solutions that are going to be the most feasible and effective,” he said.

By Syl Kacapyr

Cornell engineers have developed a robotic collective that behaves less like a machine and more like a material that flows, reshapes and adapts to its environment without centralized control.

The system, called the Cross-Link Collective, consists of dozens of small robots that have limited mobility individually, but together exhibit coordinated and sustained motion. The research, published May 20 in Science Robotics, demonstrates a robotic system that resembles soft matter, continuously deforming and reorganizing as it moves, driven by what researchers call mechanical intelligence.

“Instead of relying on explicit computation and communication, the system shifts the intelligence into the shape of the robots and their physical interactions,” said corresponding author Kirstin Petersen, associate professor of electrical and computer engineering and the Aref and Manon Lahham Faculty Fellow in the Cornell Duffield College of Engineering. “We’re leveraging the contact dynamics to let useful behaviors emerge, so the system naturally settles into configurations that reduce internal stresses and improve motion.”

Each robotic module measures about 200 millimeters in length and 20 millimeters in width, and contains a small motor that drives it to oscillate between two shapes, an “I” and a “U.” These oscillations generate forces against the ground, allowing the modules to inch forward and jostle into one another. At each end of the module are weak Velcro patches, enabling them to temporarily latch and unlatch onto neighboring modules.

On their own, the modules move slowly and inefficiently. But when they entangle into chains, they begin to move collectively, self-organizing into shifting configurations that prove resilient in challenging environments.

On incline surfaces, chains of robotic modules moved more reliably than individuals, which often stalled depending on their orientation. In obstacle fields, the collective behaved like a flowing material in which connections formed to maintain cohesion, then broke apart to prevent jamming.

“It doesn’t matter if one module has a compromised battery or fails for other reasons,” said lead author Danna Ma, visiting lecturer in electrical and computer engineering. “The system stays functional because it can adapt. It is redundant and doesn’t depend on any single module.”

Despite the minimal approach, the researchers showed that even a small amount of computation can improve system properties. To enhance cohesion, isolated modules emit an audible distress signal, prompting nearby modules to slow down and allow the straggler to reconnect.

“There is no centralized sensing or control,” Ma said. “Each module can infer when it has lost contact with the group by how much it’s being jostled and then use an audible buzz to slow down nearby modules while it catches up. It’s as simple as that.”

Co-authors at the Georgia Institute of Technology developed the original design of the module, which Petersen and Ma refined over years of experimentation and statistical analysis to improve its ability to entangle and operate in large numbers. That process revealed how even subtle changes in module size and other characteristics can influence how effectively they connect and move as a group.

The Cross-Link Collective draws inspiration from active gels – materials whose molecular links continually form and dissolve while maintaining overall structure. The findings could help inspire new forms of soft-matter engineering, though the researchers mostly see the system as a tool for studying how mechanical intelligence can give rise to resilient emergent behaviors in robot collectives.

“It’s helpful for us to start thinking about what we can encode into the physics of a system itself, as robots are increasingly applied to real-world scenarios that are highly unreliable and dynamic,” Petersen said. “Counterintuitively, by giving up exact control over configurations and coordination, we gain a surprising range of useful behaviors.”

Cornell researchers used stretchable fiber-optic sensors to create a soft robot gripper that can predict the ripeness of strawberries by touch. Credit: Anand Mishra.

By David Nutt

When assessing the ripeness of fruit, sight and smell can tell you a lot, but the best indicator is often how the fruit feels.

Cornell researchers used stretchable fiber-optic sensors to create a soft robot gripper that can predict the ripeness of strawberries by touch, then gently twist them off their branch or vine without causing any damage.

The technology, developed in the lab of Rob Shepherd, the John F. Carr Professor of Mechanical Engineering in the Cornell Duffield College of Engineering, could lead to more resilient and ecological food production and increase the availability of fruit species that are difficult to cultivate.

Shepherd’s Organic Robotics Lab previously demonstrated the potential of stretchable fiber-optic sensors to give soft robotic systems the ability to feel the same dynamic, tactile sensations that enable humans to navigate the natural world. In recent years, the team has expanded into agriculture, designing a soft robotic gripper that injects living plant leaves with sensors that help it detect and communicate with its environment.

“The great thing about Cornell is we’re a really great agriculture school, and a lot of avenues are opening up because of it,” Shepherd said. “It really allows us to uniquely combine our robotics expertise with our agricultural prominence.”

To develop a way to evaluate and handle fruit with care, Shepherd’s team partnered with Marvin Pritts, professor of horticulture and global development in the College of Agriculture and Life Sciences, who specializes in developing sustainable production methods for berry crops.

In order to train and test their gripper, they needed a model fruit. And for that, they turned to the strawberry.

“You can accurately tell when strawberries are ripe by their color,” Shepherd said. “So we could train our model to know if it’s ripe based on touch, then validate our model by looking at the color. And Anand was able to accurately estimate whether it was the right time to pick strawberries based off of the stiffness he measured.”

The soft robot gripper has an equally soft touch. The gripper is equipped with two different fiber-optic sensors, one to measure the curvature of the finger, and the other to measure the pressure at the fingertip. This way the robot can estimate the shape of an object and adjust its grip accordingly to grasp the ripe fruit without bruising it.

“The fiber-optic strain gauges have the same mechanical properties as the grippers that are using them. So it’s kind of like the flesh feels the fruit, rather than having separate sensors,” Shepherd said.

The researchers also included a planetary gear mechanism so, once the fruit is grasped, the robot wrist can rotate and twist the strawberry off its vine, instead of pulling or plucking it, which can strain and damage the fruit.

This soft-gripping technology, developed in the Organic Robotics Lab, could lead to more resilient and ecological food production and increase the availability of fruit species that are difficult to cultivate. Credit: Anand Mishra.

For cases in which touch isn’t enough, the researchers installed a camera in the gripper’s palm to find fruit that are occluded by leaves or other vegetation. However, the device will be particularly handy when ripeness can’t be detected visually, such as for avocadoes, pineapples or – Shepherd’s favorite – pawpaws.

“The problem with pawpaws is you can’t see when they’re ripe, and they ripen so fast that if you’re not there at the right time, you just miss them,” he said. “And you can’t harvest and ship them, because they don’t survive shipping very well, either. That’s one reason we don’t have pawpaws in grocery stores. But this can help with that.”

The technology could have an even greater impact on sustainable farming practices.

“Robots will allow us to do things we cannot do economically right now. We have row crops because row crops fit our machines. But if we have a larger amount of smaller robots, we can have mixed cropping of different species that support each other,” Shepherd said. “Instead of having soy one year and corn the next, you can have them both. Or you could have interspersed species that are resistant to pestilence, that help block infestations and reduce the amount of pesticides and fertilizer. You can have drought resistance from canopy species.

“It’s very complicated to manage a farm that way,” he said, “and robots could allow us to do that.”

The research was supported by the National Science Foundation Center for Research on Programmable Plant Systems (CROPPS) and the Cornell Institute for Digital Agriculture.

By David Nutt

The way bugs and birds flap their wings may look effortless, but the dynamics that keep them aloft are dizzyingly complex and difficult to quantify.

Cornell researchers created a computational model that shows the effect of insects’ morphology on stabilizing their flight. The findings could lead to a new way to understand the evolution of animal flight while also providing a blueprint for designing flapping-wing robots.

The study published May 1 in Proceedings of the National Academy of Sciences. The research was led by Z. Jane Wang, professor of physics and mechanical and aerospace engineering in the College of Arts and Sciences and Cornell Duffield College of Engineering, respectively.

The effort began more than a decade ago, when Wang set out to understand how the neural circuitry in fruit flies evolved to control flight stability. By creating a 3D computational simulation, Wang’s team showed that fruit flies sense the orientation of their bodies every time they beat their wings, about one beat every 4 milliseconds, in order to stabilize themselves.

However, in order to study flight stability in all insects, the researchers would need to build an efficient computational tool to simulate a huge number of species.

“Previous studies, including ours, have always started with models of real insects, so we’re limited by the things we observe,” Wang said. “We miss all the other configurations that are also possible for flight.”

Wang and Owen Wetherbee, the new paper’s first author, distilled the 3D model into a new version that retained the key physics of the body-wing coupling and unsteady aerodynamics. The resulting equations revealed the critical physical parameters: wing to body mass ratio, wing loading, wing hinge position, wing beat frequency and wing motion amplitude. Taken together, they form what Wang calls a “five-dimensional morphological and kinematic space.”

“The power of this model is to give us something much more explicit than what we had before,” she said. “We knew the fundamental physics. By capturing the essential physics in the new model, we can understand each piece conceptually as well as facilitate computation to explore a large parameter space.”

The analyses of the computational results in 5D resulted in two explicit formula that provide a succinct metric for stability. These criteria capture the subtle and often ignored coupling between wing inertia and the body, which depends on the interplay among wing flap frequency, hinge placement, and wing and body mass ratios in order to achieve a kind of anti-resonance state. This sweet spot allows the flapping winged animal to control its body oscillations and remain aloft – a state known as passively stable flight – despite air perturbations that would normally cause it to tumble.

“All of a sudden, we found that many forms of flapping flight have passive stability, which surprised us initially, because works so far showed that most insects, except one or two, are passively unstable, hence the necessity for neural circuitry to control them,” Wang said. “But when we expanded the morphological space, we realized that what we studied before are but a few dots in this new view.”

Now that the researchers can characterize the stability boundary, they can offer a concrete design principle for realizing stable flapping flight in robots – something that has stumped roboticists for decades.

“In principle, this offers a completely new route for designing a robotic flapping-winged machine,” Wang said. “Instead of relying on extensive feedback control, which is only partially successful, our results suggest that we can tune the shape and the frequency of the flapping devices such that, according to these two rules, we may find the flyers are passively stable already. This would greatly simplify flight control.”

The new model allows this design work to be done with faster and simpler computation, and the ability to model stability traits also points to a new way for classifying winged animals and charting their evolution.

“During evolution, various traits are selected, but we don’t have much idea about what they are, let alone understand why they are being selected and how they evolve, apart from a very few examples, such as an eye,” Wang said. “This project brings new quantitative methods to study these very big questions in both biology and robotics. Mathematical modeling allows us to go beyond our own ideas and preconceptions to tackle these large questions.”

The research was supported by the National Science Foundation.

Anton Grabolle / Autonomous Driving / Licenced by CC-BY 4.0

By Susan Kelley

Autonomous vehicles (AVs) have been tested as taxis for decades in San Francisco, Pittsburgh and around the world, and trucking companies have enormous incentives to adopt them.

But AV companies rarely share the crash- and safety-related data that is crucial to improving the safety of their vehicles – mostly because they have little incentive to do so.

Is AV safety data an auto company’s intellectual asset or a public good? It can be both – with a little tweaking, according to a team of Cornell researchers.

The team has created a roadmap outlining the barriers and opportunities to encourage AV companies to share the data to make AVs safer, from untangling public versus private data knowledge, to regulations to creating incentive programs.

“The core of AV market competition involves who has that crash data, because once you have that data, it’s much easier for you to train your AI to not make that error. The hope is to first make this data transparent and then use it for public good, and not just profit,” said Hauke Sandhaus, M.S. ’24, a doctoral candidate at Cornell Tech and co-author of “My Precious Crash Data,” published Oct. 16 in ACM on Human-Computer Interaction and presented at the ACM SIGCHI Conference on Computer-Supported Cooperative Work & Social Computing.

His co-authors are Qian Yang, assistant professor at the Cornell Ann S. Bowers College of Computing and Information Science; Wendy Ju, associate professor of information science and design tech at Cornell Tech, the Cornell Ann S. Bowers College of Computing and Information Science and the Jacobs Technion-Cornell Institute; and Angel Hsing-Chi Hwang, a former postdoctoral associate at Cornell and now assistant professor of communication at the University of Southern California, Annenberg.

The team interviewed 12 AV company employees who work on safety in AV design and deployment, to understand how they currently manage and share safety data, the data sharing challenges and concerns they face, and their ideal data-sharing practices.

The interviews revealed the AV companies have a surprising diversity of approaches, Sandhaus said. “Everyone really has some niche, homegrown data set, and there’s really not a lot of shared knowledge between these companies,” he said. “I expected there would be much more commonality.”

The research team discovered two key barriers to sharing data – both underscoring a lack of incentives. First, crash and safety data includes information about the machine-learning models and infrastructure that the company uses to improve safety. “Data sharing, even within a company, is political and fraught,” the team wrote in the paper. Second, the interviewees believed AV safety knowledge is private and brings their company a competitive edge. “This perspective leads them to view safety knowledge embedded in data as a contested space rather than public knowledge for social good,” the team wrote.

And U.S. and European regulations are not helping. They require only information such as the month when the crash occurred, the manufacturer and whether there were injuries. That doesn’t capture the underlying unexpected factors that often cause accidents, such as a person suddenly running onto the street, drivers violating traffic rules, extreme weather conditions or lost cargo blocking the road.

To encourage more data-sharing, it’s crucial to untangle safety knowledge from proprietary data, the researchers said. For example, AV companies could share information about the accident, but not raw video footage that would reveal the company’s technical infrastructure.

Companies could also come up with “exam questions” that AVs would have to pass in order to take the road. “If you have pedestrians coming from one side and vehicles from the other side, then you can use that as a test case that other AVs also have to pass,” Sandhaus said.

Academic institutions could act as data intermediaries with which AV companies could leverage strategic collaborations. Independent research institutions and other civic organizations have set precedents working with industry partners’ public knowledge. “There are arrangements, collaboration, patterns for higher ed to contribute to this without necessarily making the entire data set public,” Qian said.

The team also proposes standardizing AV safety assessment via more effective government regulations. For example, a federal policymaking agency could create a virtual city as a testing ground, with busy traffic intersections and pedestrian-heavy roads that every AV algorithm would have to be able to navigate, she said.

Federal regulators could encourage car companies to contribute scenarios to the testing environment. “The AV companies might say, ‘I want to put my test cases there, because my car probably has passed those tests.’ That can be a mechanism for encouraging safer vehicle development,” Yang said. “Proposing policy changes always feels a little bit distant, but I do think there are near-future policy solutions in this space.”

The research was funded by the National Science Foundation and Schmidt Sciences.

Kushal Kedia (left) and Prithwish Dan (right) are members of the development team behind RHyME, a system that allows robots to learn tasks by watching a single how-to video.

By Louis DiPietro

Cornell researchers have developed a new robotic framework powered by artificial intelligence – called RHyME (Retrieval for Hybrid Imitation under Mismatched Execution) – that allows robots to learn tasks by watching a single how-to video. RHyME could fast-track the development and deployment of robotic systems by significantly reducing the time, energy and money needed to train them, the researchers said.

“One of the annoying things about working with robots is collecting so much data on the robot doing different tasks,” said Kushal Kedia, a doctoral student in the field of computer science and lead author of a corresponding paper on RHyME. “That’s not how humans do tasks. We look at other people as inspiration.”

Kedia will present the paper, One-Shot Imitation under Mismatched Execution, in May at the Institute of Electrical and Electronics Engineers’ International Conference on Robotics and Automation, in Atlanta.

Home robot assistants are still a long way off – it is a very difficult task to train robots to deal with all the potential scenarios that they could encounter in the real world. To get robots up to speed, researchers like Kedia are training them with what amounts to how-to videos – human demonstrations of various tasks in a lab setting. The hope with this approach, a branch of machine learning called “imitation learning,” is that robots will learn a sequence of tasks faster and be able to adapt to real-world environments.

“Our work is like translating French to English – we’re translating any given task from human to robot,” said senior author Sanjiban Choudhury, assistant professor of computer science in the Cornell Ann S. Bowers College of Computing and Information Science.

This translation task still faces a broader challenge, however: Humans move too fluidly for a robot to track and mimic, and training robots with video requires gobs of it. Further, video demonstrations – of, say, picking up a napkin or stacking dinner plates – must be performed slowly and flawlessly, since any mismatch in actions between the video and the robot has historically spelled doom for robot learning, the researchers said.

“If a human moves in a way that’s any different from how a robot moves, the method immediately falls apart,” Choudhury said. “Our thinking was, ‘Can we find a principled way to deal with this mismatch between how humans and robots do tasks?’”

RHyME is the team’s answer – a scalable approach that makes robots less finicky and more adaptive. It trains a robotic system to store previous examples in its memory bank and connect the dots when performing tasks it has viewed only once by drawing on videos it has seen. For example, a RHyME-equipped robot shown a video of a human fetching a mug from the counter and placing it in a nearby sink will comb its bank of videos and draw inspiration from similar actions – like grasping a cup and lowering a utensil.

RHyME paves the way for robots to learn multiple-step sequences while significantly lowering the amount of robot data needed for training, the researchers said. They claim that RHyME requires just 30 minutes of robot data; in a lab setting, robots trained using the system achieved a more than 50% increase in task success compared to previous methods.

“This work is a departure from how robots are programmed today. The status quo of programming robots is thousands of hours of tele-operation to teach the robot how to do tasks. That’s just impossible,” Choudhury said. “With RHyME, we’re moving away from that and learning to train robots in a more scalable way.”

This research was supported by Google, OpenAI, the U.S. Office of Naval Research and the National Science Foundation.

Read the work in full

One-Shot Imitation under Mismatched Execution, Kushal Kedia, Prithwish Dan, Angela Chao, Maximus Adrian Pace, Sanjiban Choudhury.