Normally, students and scientists walk here, but today a drone is flying through a corridor on TU Delft Campus. Seemingly effortlessly, it whizzes past and between a variety of obstacles: rubbish bins, stacked boxes and poles. But then suddenly a person appears, walking straight towards the drone in the same space. This is not a stationery object but an actual moving person. ‘This is much more difficult for the drone to process. Because how fast is someone moving and are they going to make unexpected movements?’ asks Javier Alonso-Mora. This goes well too. As the drone approaches the walker, it moves to the side and flies on. The flying robot completes the obstacle course without a hitch.
This experiment, conducted by Alonso-Mora and his colleagues, is a good reflection of their research field. They investigate mobile robots that move on wheels or fly through the air and take their surroundings into account – which is why they can move safely alongside us in a corridor, room or hallway.
The new generation of robots that we’re now developing work with people and other robots. Dr. Javier Alonso Mora
That’s something new. For decades, robots were mainly used in factories, where they were separated from people and could assemble cars in a screened-off area or put something on a conveyor belt, for example. ‘The new generation of robots that we’re now developing work with people and other robots. So they have to take into account how others behave. They deliver packages, for example, or cooperate by assembling or delivering something. Moreover, they’re not fixed but can move freely in space,’ he says.
Adapting at lightning speed
For this to work, it’s important that the robots always make the right decision. This is achieved based on models that the scientists, including Alonso-Mora, are currently developing. ‘They perceive their environment on the basis of these models. It helps them to complete a task safely,’ he says.
Humans are highly skilled at this. We’re masters at correctly predicting what will happen and avoiding other people in time or adjusting our route accordingly to avoid a collision if things don’t happen as predicted after all. Busy intersections in major cities are good examples. Cyclists, cars, pedestrians, scooters, trams and trucks weave in and out of each other. It looks chaotic, but usually goes smoothly. ‘This is more difficult for a robot. It has to constantly predict what action is safe. There are many potential outcomes, and the robot has to calculate the various permutations. That takes time. Humans, on the other hand, have so much experience that we can assess a situation at lightning speed and adapt immediately if necessary. Our models are trying to achieve the same with robots.’
It’s important that the robots that complete difficult tasks work as well and safely as possible. Dr. Javier Alonso Mora
An interesting example of Alonso-Mora’s research is the Harmony project he’s working on. Scientists are developing robots that help nurses, doctors and patients. For example, they bring food to patients in bed or medicine to nurses. ‘They need to calculate the best route and how to behave on the way. How do they safely navigate an area with a moving bed, a patient on crutches or a surgeon who’s in a hurry?’
Testing the robots in hospitals
Not only do the robots have to cover small distances, but they also have to open doors and plan a route. Moreover, it’s important that they understand whether their task needs to be carried out quickly or can wait a while. ‘So the robots have to process a great deal of information, and on top of that, they have to analyse their environment and work with people and other robots. We’ve just launched this international project, and our goal is to start using demonstration robots in hospitals in Switzerland and Sweden in three years’ time. We’re creating and testing the algorithm that will allow the robots to move safely.’
It’s no coincidence that Alonso-Mora is working on this particular project. He was already fascinated by robots as a young boy. ‘I used to love playing with Lego; I could spend hours building things with it. We’re now doing something similar with robots, but in a more complicated way, because they too consist of many different parts. I’m interested in whether we can make the robots of the future intelligent. As a child, I loved reading Isaac Asimov’s science fiction, which often featured robots. The novel I, Robot, which was made into a film starring Will Smith, is a good example. It’s fiction, of course, but increasingly we’re getting closer to robots that are smart and can work with us. Whether they’re autonomous cars, drones or robots moving around in hospitals. It’s important that the robots that complete difficult tasks work as well and safely as possible.’
Dutch brewing company Heineken is one of the largest beer producers in the world with more than 70 production facilities globally. From small breweries to mega-plants, its logistics and production processes are increasingly complex and its machinery ever more advanced. The global beer giant therefore began looking for robotics solutions to make its breweries safer and more attractive for employees while enabling a more flexible organisation.
The environment is constantly changing and the robot has to be able to respond immediately.
Shobhit Yadav, mechatronics engineer smart industries and robotics at TNO
Automatically adapting to the situation
Dennis van der Plas, senior global lead packaging lines at Heineken, says, “We are becoming a high-tech company and attracting more and more technically trained staff. Repetitive tasks – like picking up fallen bottles from the conveyor belt will not provide them job satisfaction.” As part of the SMITZH innovation programme, Heineken and RoboHouse fieldlab, with support from the Netherlands Organisation for Applied Scientific Research (TNO), have developed a solution on the basis of flexible manufacturing: automated handling of unexpected situations.
According to Shobhit Yadav of TNO, flexible manufacturing is one of the most important developments in smart industry. “Today, manufacturing companies mainly produce small series on demand. It means that manufacturers have to be able to make many different products. This can be achieved either with a large number of production lines or with a small number that are flexible enough to adapt.” The Heineken project fell into the second category and involved developing a robot that could recognise different kinds of beer bottles that had fallen over on the conveyor. The robot had to pick them up while the belt was still moving. “The environment is constantly changing and the robot has to be able to respond immediately”, explains Shobhit. “This is a typical example of a flexible production line that automatically adapts to the situation.”
Robotics for a safe and enjoyable working environment
Industrial robots have obviously been around for a while. “The automotive industry deploys robots for welding car parts, whereas our sector uses them for automatically palletising products”, says Dennis. “But with this project we took a different approach. Our starting point was not a question of which robots exist and how they could be used. Instead, we focused on the needs and wishes of the people in the breweries, the operators who control and maintain the machines and how robots could support them in their work.”
The solution, in other words, had to lead not only to process optimisation but also – especially – to improved safety and greater job satisfaction. In addition, it would result in Heineken becoming a better employer. It is why Dennis and his colleague Wessel Reurslag, global lead packaging engineering & robotics, asked the operators what they would need to make their work safer and more interesting. One of the use cases that emerged was picking up bottles that had fallen over on the conveyor belt: repetitive but also unsafe as the glass bottles could break.
The lab is the place to meet for anyone involved in robotics.
Wessel Reurslag, global lead packaging engineering & robotics at Heineken
Experimenting without a business case
Heineken initially made contact with RoboHouse field lab through a sponsorship project with X!Delft, an initiative that strengthens corporate innovation and closes the gap between industry and Delft University of Technology. “The lab is the place to meet for anyone involved in robotics”, says Wessel. “It is also linked to SMITZH and thus connected to TNO.”
The parties soon realised that their ambitions overlapped. Heineken was seeking independent advice and both TNO and RoboHouse were looking for an applied research project that focused on flexible manufacturing. “This kind of partnership is very valuable to all involved”, says Shobhit. “SMITZH allows us at TNO to work with current issues in the industry and establish valuable contacts, which makes our research more relevant. In turn, manufacturers have somewhere to go with their questions and problems regarding smart technologies.”
Accessible way to do research and experiment
The guys at Heineken have nothing but praise for the innovative collaboration with TNO and RoboHouse. Dennis says, “The great thing is that all project partners were in it to learn something. At RoboHouse, we had access to the expertise of robotics engineers and state-of-the-art technologies like robotic arms. We supplied the bottle conveyor ourselves and TNO also added knowledge to the mix. It is an accessible way to do research and experiment. This would be a lot more difficult with a business case, which must involve an operational advantage from the start.”
Through close cooperation, we really developed a joint product.
Bas van Mil, mechanical engineer and robot gripper expert at RoboHouse
A joint product by TNO and RoboHouse
TNO and RoboHouse distilled two research goals from the use cases presented by Heineken: enabling real-time robot control and using vision technology to direct the robots with cameras. The main challenge involved devising a solution that could be applied to Heineken’s high-speed packaging lines. TNO worked on the control and movements of the robot, while RoboHouse took on the vision technology aspect. This entailed recognising the fallen bottles, developing the system’s software-based control and building the ‘gripper’ to pick up the bottles.
In many existing robotic systems, […] the robot will do a ‘blind pick.’
Bas van Mil, mechanical engineer and robot gripper expert at RoboHouse
“Communication between the robot and the computer is very important”, explains Bas van Mil, mechanical engineer at RoboHouse. “Our input and TNO’s work were complementary. For example, RoboHouse did not possess Shobhit’s knowledge of control technology, which is indispensable for controlling the robot. Through close cooperation, we really developed a joint product.”
Every millisecond counts
The biggest challenge in detecting and tracking fallen beer bottles is that they never stop moving. As Bas explains, “They do not just move in the direction of the bottle conveyor but can also roll around on the belt itself. In many existing robotic systems, the camera takes a single photo that informs the movements of the robot. The robot will do a ‘blind pick’ with no way of knowing whether anything has changed since the photo was taken. This only works if the environment stays the same – but in this case, it doesn’t.”
The solution involves a system in which the camera and the movements of the robot are constantly connected to each other. Bas says, “Every millisecond counts as the bottle will disappear from view and the robot will still try to pick it up from the spot where it was half a second ago.” A RoboHouse programmer developed the camera software to be as fast and efficient as possible. The field lab even purchased a powerful computer running an advanced AI system especially for the project.
We are receiving feedback as well as requests for the new robotic systems from breweries all over the globe.
Dennis van der Plas, senior global lead packaging lines at Heineken
TNO and RoboHouse then wrote a programme together that determines the speed of the robot from the moment a fallen bottle is detected. This enables the robot to move with the bottle based on its calculated speed. It is what makes this robot so different from existing ones. “The robot responds immediately to changes”, says Shobhit. “In fact, it is 30 per cent faster than the current top speed of Heineken’s bottle conveyors. As a result, it has a wide range of applications and can be used in a variety of environments with different production speeds.”
Smarter thanks to independent partners
Heineken valued not just the successful innovation but also the independent character of TNO and RoboHouse during the development process. Dennis says, “We now have a much better idea of what is technically feasible, what the challenges are and what we can realistically ask of our technical suppliers. Thanks to this project, we can act much more like a smart buyer and make smarter demands of our suppliers. This information is relevant to have, especially as we operate in such an innovative field where we do not just buy parts off the shelf. After all, if I ask too little, I will not get the best out of my project. But if I ask too much, it affects our relationship with the supplier.”
The project has also served as a source of inspiration for Dennis’s colleagues worldwide. “We have been sharing videos and reports from SMITZH on the intranet, building a kind of community within Heineken. We are receiving feedback as well as requests for the new robotic systems from breweries all over the globe.” To meet the demand, Dennis and Wessel want to supply breweries with a ready-to-use version. “We are currently looking for parties that can make the technology available and provide support services.”
Meanwhile, RoboHouse and TNO aim to continue optimising the robot. “This is only the pilot version”, says Bas. “We can still improve its flexibility, for example by installing a different vision module, thereby making the technology even more widely applicable.” Both organisations are therefore looking for use cases in which they can use the same technology to solve other problems. “We are looking at the bigger picture”, explains Shobhit. “This project could serve as a model for similar challenges in other industries.”
Platform for valuable connections
All parties emphasise the benefits of working together and sharing knowledge to achieve a successful and relevant innovation. “There are very few places like SMITZH where manufacturing companies can go with these kinds of questions”, says Shobhit. “SMITZH is quite unique in that sense and is vitally important because it offers a specific platform for appropriate collaborations.” Wessel agrees: “If there is one thing this project has made clear, it is that producers like Heineken, tech companies and knowledge institutions should collaborate more intensively to substantiate these kinds of projects. You won’t find a solution in a PDF or a presentation. There can be no genuine solution until it is made real and put into practice.”
Do you have a similar use case and are you, like Heineken, searching for a practical solution? Or would you first like to learn more about RoboHouse or this specific project? Feel free to contact our lead engineer Bas van Mil, or email SMITZH with questions about its collaboration programme, or contact Shobhit Yadav at TNO more information.
SMITZH and future of work fieldlab RoboHouse SMITZH is an innovation programme focused on smart manufacturing solutions in West Holland. It brings together supply and demand to stimulate the industrial application of smart manufacturing technologies and help regional companies innovate. Each SMITZH project consists of at least one manufacturing company and a fieldlab. RoboHouse served as the fieldlab in this project.
Combining drone imagery with weather data and planting schemes to forecast how much fresh vegetables a harvest is going to yield; that’s what predictive modelling intern Berend Klaver from TU Delft is sweating on at VanBoven, while his bosses are entertaining the American west coast.
“The market of fresh vegetables is one of constant shortages and surpluses. VanBoven predicts the harvest of fresh produce to perfectly align supply and demand. The result is decreased food waste, a resilient value chain and fair prices,” says the startup on its website.
The result is decreased food waste, a resilient value chain and fair prices.
So this firm from The Hague does not favour any particular party in the value chain, even though we do get a sense that it may have special empathy for farmers. What’s unique about the robot-powered predictions made by Klaver and his colleagues, is their cooperative deployment. The models are being used to foster symbiotic relations between all players in the system: growers of fresh vegetables, distributors and processors, agricultural service providers and retailers too. These parties can now all work together to anticipate fluctuations and coordinate a joint response to the unpredictability of nature and markets.
A refreshing proposition that could point towards a more positive future of work.
As an alternative to cutthroat, winner-takes-all capitalism, it seems a refreshing proposition that could point towards a more positive future of work. Maybe today’s tech startups from The Netherlands are not just soaking up insights during their missions to Silicon Valley, but are dishing them out too. ‘How Dutch-style Cosiness Breeds Resilience and Wellbeing’; we can already picture that headline in WIRED.
The competition aims to highlight the importance of valorisation in the academic world. In addition to a Silicon Valley tour hosted by Holland in the Valley, an ecosystem for Dutch entrepreneurs in the San Francisco Bay Area, the winners are also being showered with perks such as introductions to networks, coaching programmes and the right to carry the title ‘Best Academic Startup of 2020’.
Multidisciplinary teams of minor students have designed and built a functional robotic prototype for a project customer with a challenge to improve the lives of working people. Streaming live from RoboHouse on the TU Delft Campus, you can join a virtual celebration of ideas en technology with some of our communities finest talent.
How does it work?
On Thursday 28 January 3pm (CET) you can visit the streaming website that we have created for the occasion. There you will find live video streams for a programme with a project overview and demonstrations by each student team. This all happens between 3pm and 4:30pm.
Our hosts will be running around RoboHouse for you, going from robot to robot and from team to team, followed by highly mobile camera people, all with the purpose of giving you the most intimate and direct experience. We want you feel as if you are able to touch the robots yourself.
After toasting on another successful Demo Day, the programme will go into free flow. You are invited to plunge into the various projects and splash around with anyone you like during the Breakouts. These start around 4:40pm and can be joined via Zoom-links that are available on the website.
And of course, most appreciation and respect go to the eight talented student teams, who this edition go by the illustrious names of: Krill + ARMS + D.I.R.T + BikeBotics + Oxillia + BoxBot + Shino + Althea.
Welcome to the future of work!
About the TU Delft Minor Robotics
The Minor Robotics is a 5-month educational program for third year BSc students of Mechanical Engineering, Industrial Design, Computer Science, and Electrical Engineering from Delft University of Technology. A team of students from all above disciplines (to make sure they cover all knowledge needed in robotics) are working on building a robot for a customer.
Students follow multidisciplinary courses, and work in multidisciplinary teams to design, build, and program robots for customers. During the first months of the minor the students will focus on taking courses outside their own discipline (for example, the Industrial Design student will learn about programming and electronics, while a Computer Science student will learn about statics and prototyping). But they will also work on the design of their robot in close collaboration with the customer.
The last months are dedicated full time on building the robot and ends with a demo together with all other teams from the Minor Robotics.
Here’s a video of last year’s TU Delft Minor Robotics event at RoboHouse.