Happy holidays everyone! Here are some more robot videos to get you into the holiday spirit.
Have a last minute holiday robot video of your own that you’d like to share? Send your submissions to daniel.carrillozapata@robohub.org
And as a late entry, here we have the video from PAL Robotics:
How can we make a robot learn in the real world while ensuring safety? In this work, we show how it’s possible to face this problem. The key idea to exploit domain knowledge and use the constraint definition to our advantage. Following our approach, it’s possible to implement learning robotic agents that can explore and learn in an arbitrary environment while ensuring safety at the same time.
Safety is a fundamental feature in real-world robotics applications: robots should not cause damage to the environment, to themselves, and they must ensure the safety of people operating around them. To ensure safety when we deploy a new application, we want to avoid constraint violation at any time. These stringent safety constraints are difficult to enforce in a reinforcement learning setting. This is the reason why it is hard to deploy learning agents in the real world. Classical reinforcement learning agents use random exploration, such as Gaussian policies, to act in the environment and extract useful knowledge to improve task performance. However, random exploration may cause constraint violations. These constraint violations must be avoided at all costs in robotic platforms, as they often result in a major system failure.
While the robotic framework is challenging, it is also a very well-known and well-studied problem: thus, we can exploit some key results and knowledge from the field. Indeed, often a robot’s kinematics and dynamics are known and can be exploited by the learning systems. Also, physical constraints e.g., avoiding collisions and enforcing joint limits, can be written in analytical form. All this information can be exploited by the learning robot.
Many reinforcement learning approaches try to solve the safety problem by incorporating the constraint information in the learning process. This approach often results in slower learning performances, while not being able to ensure safety during the whole learning process. Instead, we present a novel point of view to the problem, introducing ATACOM (Acting on the TAngent space of the COnstraint Manifold). Different from other state-of-the-art approaches, ATACOM tries to create a safe action space in which every action is inherently safe. To do so, we need to construct the constraint manifold and exploit the basic domain knowledge of the agent. Once we have the constraint manifold, we define our action space as the tangent space to the constraint manifold.
We can construct the constraint manifold using arbitrary differentiable constraints. The only requirement is that the constraint function must depend only on controllable variables i.e. the variables that we can directly control with our control action. An example could be the robot joint positions and velocities.
We can support both equality and inequality constraints. Inequality constraints are particularly important as they can be used to avoid specific areas of the state space or to enforce the joint limits. However, they don’t define a manifold. To obtain a manifold, we transform the inequality constraints into equality constraints by introducing slack variables.
With ATACOM, we can ensure safety by taking action on the tangent space of the constraint manifold. An intuitive way to see why this is true is to consider the motion on the surface of a sphere: any point with a velocity tangent to the sphere itself will keep moving on the surface of the sphere. The same idea can be extended to more complex robotic systems, considering the acceleration of system variables (or the generalized coordinates, when considering a mechanical system) instead of velocities.
The above-mentioned framework only works if we consider continuous-time systems, when the control action is the instantaneous velocity or acceleration. Unfortunately, the vast majority of robotic controllers and reinforcement learning approaches are discrete-time digital controllers. Thus, even taking the tangent direction of the constraint manifold will result in a constraint violation. It is always possible to reduce the violations by increasing the control frequency. However, error accumulates over time, causing a drift from the constraint manifold. To solve this issue, we introduce an error correction term that ensures that the system stays on the reward manifold. In our work, we implement this term as a simple proportional controller.
Finally, many robotics systems cannot be controlled directly by velocity or accelerations. However, if an inverse dynamics model or a tracking controller is available, we can use it and compute the correct control action.
We tried ATACOM on a simulated air hockey task. We use two different types of robots. The first one is a planar robot. In this task, we enforce joint velocities and we avoid the collision of the end-effector with table boundaries.
The second robot is a Kuka Iiwa 14 arm. In this scenario, we constrained the end-effector to move on the planar surface and we ensure no collision will occur between the robot arm and the table.
In both experiments, we can learn a safe policy using the Soft Actor-Critic algorithm as a learning algorithm in combination with the ATACOM framework. With our approach, we are able to learn good policies fast and we can ensure low constraint violations at any timestep. Unfortunately, the constraint violation cannot be zero due to discretization, but it can be reduced to be arbitrarily small. This is not a major issue in real-world systems, as they are affected by noisy measurements and non-ideal actuation.
The key question to ask is if we can ensure any safety guarantees with ATACOM. Unfortunately, this is not true in general. What we can enforce are state constraints at each timestep. This includes a wide class of constraints, such as fixed obstacle avoidance, joint limits, surface constraints. We can extend our method to constraints considering not (directly) controllable variables. While we can ensure safety to a certain extent also in this scenario, we cannot ensure that the constraint violation will not be violated during the whole trajectory. Indeed, if the not controllable variables act in an adversarial way, they might find a long-term strategy to cause constraint violation in the long term. An easy example is a prey-predator scenario: even if we ensure that the prey avoids each predator, a group of predators can perform a high-level strategy and trap the agent in the long term.
Thus, with ATACOM we can ensure safety at a step level, but we are not able to ensure long-term safety, which requires reasoning at trajectory level. To ensure this kind of safety, more advanced techniques will be needed.
The authors were best paper award finalists at CoRL this year, for their work: Robot reinforcement learning on the constraint manifold.
The use of industrial robots in factories around the world is accelerating at a high rate: 126 robots per 10,000 employees is the new average of global robot density in the manufacturing industries – nearly double the number five years ago (2015: 66 units). This is according to the 2021 World Robot Report.
By regions, the average robot density in Asia/Australia is 134 units, in Europe 123 units and in the Americas 111 units. The top 5 most automated countries in the world are: South Korea, Singapore, Japan, Germany, and Sweden.
“Robot density is the barometer to track the degree of automation adoption in the manufacturing industry around the world,” says Milton Guerry, President of the International Federation of Robotics.
The development of robot density in China is the most dynamic worldwide: Due to the significant growth of robot installations, the density rate rose from 49 units in 2015 to 246 units in 2020. Today, China’s robot density ranks 9th globally compared to 25th just five years ago.
Asia is also the home of the country with the world´s highest robot density in the manufacturing industry: the Republic of Korea has held this position since 2010. The country’s robot density exceeds the global average seven-fold (932 units per 10,000 workers). Robot density had been increasing by 10% on average each year since 2015. With its globally recognized electronics industry and a distinct automotive industry, the Korean economy is based on the two largest areas for industrial robots.
Singapore takes second place with a rate of 605 robots per 10,000 employees in 2020. Singapore’s robot density had been growing by 27% on average each year since 2015.
Japan ranked third in the world: In 2020, 390 robots were installed per 10,000 employees in the manufacturing industry. Japan is the world´s predominant industrial robot manufacturer: The production capacity of Japanese suppliers reached 174,000 units in 2020. Today, Japan´s manufacturers deliver 45% of the global robot supply.
Robot density in the United States rose from 176 units in 2015 to 255 units in 2020. The country ranks seventh in the world – ahead of Chinese Taipei (248 units) and China (246 units). The modernization of domestic production facilities has boosted robot sales in the United States. The use of industrial robots also aids to achieve decarbonization targets e.g. in the cost-efficient production of solar panels and in the continued transition towards electric vehicles. Several car manufacturers have announced investments to further equip their factories for new electric drive car models or to increase capacity for battery production. These major projects will create demand for industrial robots in the next few years.
Europe´s most automated country is Germany – ranking 4th worldwide with 371 units. The annual supply had a share of 33% of total robot sales in Europe 2020 – 38% of Europe’s operational stock is in Germany. The German robotics industry is recovering, mainly driven by strong overseas business rather than by the domestic or European market. Robot demand in Germany is expected to grow slowly, mainly supported by demand for low-cost robots in the general industries and outside traditional manufacturing.
France has a robot density of 194 units (ranking 16th in the world), which is well above the global average of 126 robots and relatively similar compared to other EU countries like Spain (203 units), Austria (205 units) or The Netherlands (209 units). EU members like Sweden (289 units), Denmark (246 units) or Italy (224 units), have a significantly higher degree of automation in the manufacturing segment.
As the only G7 country – the UK has a robot density below the world average of 126 units with 101 units, ranking 24th. Five years ago, the UK´s robot density was 71 units. The exodus of foreign labor after Brexit increased the demand for robots in 2020. This situation is expected to prevail in near future, the modernization of the UK manufacturing industry will also be boosted by massive tax incentives, the “super-deduction”: From April 2021 until March 2023, companies can claim 130% of capital allowances as a tax relief for plant and machinery investments.
That’s right! You better not run, you better not hide, you better watch out for brand new robot holiday videos on Robohub!
Drop your submissions down our chimney at daniel.carrillozapata@robohub.org and share the spirit of the season.
For inspiration, here are our two first videos: