The smart city of Milton Keynes hosted the first edition of the European Robotics League (ERL)- Smart Cities Robotic Challenge (SciRoc Challenge). Ten European teams met in the shopping mall of Centre:mk to compete against each other in five futuristic scenarios in which robots assist humans serving coffee orders, picking products in a grocery shop or bringing medical aid. This robotics competition aims at benchmarking robots using a ranking system that allows teams to assess their performance and compare it with others. Find out the winning teams of the SciRoc Challenge 2019…
The ERL Smart Cities Robotics Challenge
The European Robotics League (ERL) was launched in 2016 under the umbrella of SPARC- the Partnership for Robotics in Europe. This pan-European robotics competition builds on the success of the EU-funded projects: RoCKIn, euRathlon, EuRoC and ROCKEU2. The SciRoc Horizon 2020 project took over the reins of the league in 2018, bringing in the expertise from the University of the West of England, Bristol, the Advanced Center for Aerospace Technologies (CATEC), the Association of Instituto Superior Técnico for Research and Development (IST-ID), the Centre for Maritime Research and Experimentation (CMRE), euRobotics aisbl, Politecnico di Milano, the Open University, the Sapienza University of Rome, the University of Applied Sciences Bonn-Rhein-Sieg and the Universitat Politecnica de Catalunya. The SciRoc consortium carefully designed the new biennial ERL Smart Cities Robotics challenge where robots from all three ERL leagues (Consumer, Professional and Emergency Service Robots) come together to interact with a smart infrastructure in a familiar urban setting.
Daniele Nardi, Professor of Artificial Intelligence at Sapienza University of Rome and Head of the Technical Committee of the SciRoc Challenge, explains that the project consortium chose the topic of smart cities because “Robotics competitions in a smart city are projected into the future, since it is likely that smart cities will be among the first places populated by robots”. Professor Nardi adds that they structured the competition through a series of episodes, each of them being a specific technical challenge for a robotic system and at the same time representing a situation that would be typically encountered in a smart city populated with robots.
Benchmarking through competitions
Benchmarking has been and still is a hot topic in the robotics community. How can we compare robots’ and robot systems’ performance? In 2013, the RoCKIn and euRathlon projects started exploring and developing a benchmarking methodology for robotics competitions. But, why benchmark robots through competitions? The answer from Matteo Matteuci, Assistant Professor at the Politecnico di Milano and one of the researchers behind the development of the ERL benchmarking methodology, is clear: “competitions are fun, they put you in realistic situations outside of your own lab”.
The European Robotics League approach to benchmarking is based on the definition of two separate, but interconnected, types of benchmarks: Functionality Benchmarks (FBMs) and Task Benchmarks (TBMs). A functionality benchmark evaluates a robot’s performance in specific functionalities, such as navigation, object perception, manipulation, etc. Whereas, a task benchmark assesses the performance of the robot system facing complex tasks that require using different functionalities. Matteucci comments “We have gone from a big once in a while costly competition to frequent, sustainable and repeatable competitions in a regional net of laboratories and hubs. The ERL is a big open lab running tournaments during the whole year. The competitions are structured in a way that makes them repeatable experiments you can compare.” In the case of the Sciroc Challenge he adds “it is somehow in the middle of the two, it’s a bigger event made up of small tasks in a public venue. Each task benchmark requires different functionalities, but it is more based on one than the others. For example, the episode of the elevator is mostly focused on HRI but also requires navigation and perception. The door is a mobile manipulation task but requires navigation and perception. SciRoc can be seen as a dry run of possible benchmarks that can be later introduced into the ERL local tournaments. In the case of the “Through the door” episode set up, it’s going to be deployed within 6 months in the facility of EUROBENCH project in Genova for benchmarking humanoid robots.”
The SciRoc challenge has also introduced a new term to the European Robotics League terminology: “the Episode”. Matteucci explains that this term refers more to the set up than to a category of benchmark. “The episode provides a narrative for the general public. SciRoc is a robotics event in the middle of a city in contact with people, so people are more interested in the story and perspective than the pure engineering benchmarking part. That’s why we came up with the short stories in the context of the smart cities.”
The ERL local tournaments have specific TBMs and FBMs for each of the leagues. With the purpose to align the ERL tournaments with the new SciRoc challenge, the leagues had to integrate new benchmarks. This was the case of the ERL Consumer Service robotics league, that takes place in a home or domestic environment. Pedro Lima, Professor in Robotics at IST university of Lisbon and Head of the Technical Committee of the ERL Consumer league explains that “the tasks the robots have to perform in the apartment (navigate around the house, detecting and picking objects, etc.) are very similar to the ones they have to perform in the shopping mall. We made changes in some of the TBMs to be more in line with the requirements of the coffee shop environment. Also, the task of opening a door is not new for the ERL Consumer league, but in the “through the door” episode the door has a handle to add complexity and also encourage humanoid robots to participate.”
SciRoc Challenge 2019 winners
The ERL Smart Cities Robotic challenge finals took place during the weekend and many visitors could see the robots successfully perform different complex tasks.
The awards ceremony was held at the Centre:MK competition arena on Saturday afternoon. Matthew Studley, SciRoc project coordinator, welcomed everyone and opened the ceremony. Then followed a short speech by Enrico Motta, Director of SciRoc Challenge 2019, thanking teams and sponsors Milton Keynes Council, Centre:mk, PAL Robotics, OCADO Technology, COSTA Coffee, Cranfield University and Catapult.
The winners of the SciRoc Challenge 2019 in each episode are:
Deliver coffee shop orders (E03)
- Winner: Leeds Autonomous Service Robots
- Runner up: eNTiTy
Take the elevator (E04)
- Winner: Gentlebots
- Runner up: eNTiTy
Shopping pick and pack (E07)
- Winner: b-it-bots
- Runner up: CATIE Robotics
Through the door (E10)
- Winner: b-it-bots
Fast delivery of emergency pills (E12)
- Winner: TeamBathDrones Research
- Runner up: UWE Aero
Public choice Award: Most social robot
- Winner: eNTiTy
For information on teams’ scoring, visit the websites of the European Robotics League and SciRoc Challenge.
Epilogue
This summer our colleague and friend Gerhard Kraetzschmar passed away. Gerhard was Professor for Autonomous Systems at Bonn-Rhein-Sieg University of Applied Sciences, RoboCup Trustee and Head of the ERL Professional Service Robots league. He believed robotics competitions are an excellent platform for challenging and showcasing robotics technologies, and for developing skills of future engineers and scientists. I am sure his legacy will inspire new generations of roboticist, the same way that he inspired us.
Get involved in robotics competitions, they are much more than fun.
See you all at SciRoc Challenge 2021!
Missed coverage of the teams participating in SciRoc Challenge? Find it here.
