Archive 09.02.2018

Humans physically interact with each other every day – from grabbing someone’s hand when they are about to spill their drink, to giving your friend a nudge to steer them in the right direction, physical interaction is an intuitive way to convey information about personal preferences and how to perform a task correctly.

So why aren’t we physically interacting with current robots the way we do with each other? Seamless physical interaction between a human and a robot requires a lot: lightweight robot designs, reliable torque or force sensors, safe and reactive control schemes, the ability to predict the intentions of human collaborators, and more! Luckily, robotics has made many advances in the design of personal robots specifically developed with humans in mind.

However, consider the example from the beginning where you grab your friend’s hand as they are about to spill their drink. Instead of your friend who is spilling, imagine it was a robot. Because state-of-the-art robot planning and control algorithms typically assume human physical interventions are disturbances, once you let go of the robot, it will resume its erroneous trajectory and continue spilling the drink. The key to this gap comes from how robots reason about physical interaction: instead of thinking about why the human physically intervened and replanning in accordance with what the human wants, most robots simply resume their original behavior after the interaction ends.

We argue that robots should treat physical human interaction as useful information about how they should be doing the task. We formalize reacting to physical interaction as an objective (or reward) learning problem and propose a solution that enables robots to change their behaviors while they are performing a task according to the information gained during these interactions.

Reasoning About Physical Interaction: Unknown Disturbance versus Intentional Information

The field of physical human-robot interaction (pHRI) studies the design, control, and planning problems that arise from close physical interaction between a human and a robot in a shared workspace. Prior research in pHRI has developed safe and responsive control methods to react to a physical interaction that happens while the robot is performing a task. Proposed by Hogan et. al., impedance control is one of the most commonly used methods to move a robot along a desired trajectory when there are people in the workspace. With this control method, the robot acts like a spring: it allows the person to push it, but moves back to an original desired position after the human stops applying forces. While this strategy is very fast and enables the robot to safely adapt to the human’s forces, the robot does not leverage these interventions to update its understanding of the task. Left alone, the robot would continue to perform the task in the same way as it had planned before any human interactions.

Why is this the case? It boils down to what assumptions the robot makes about its knowledge of the task and the meaning of the forces it senses. Typically, a robot is given a notion of its task in the form of an objective function. This objective function encodes rewards for different aspects of the task like “reach a goal at location X” or “move close to the table while staying far away from people”. The robot uses its objective function to produce a motion that best satisfies all the aspects of the task: for example, the robot would move toward goal X while choosing a path that is far from a human and close to the table. If the robot’s original objective function was correct, then any physical interaction is simply a disturbance from its correct path. Thus, the robot should allow the physical interaction to perturb it for safety purposes, but it will return to the original path it planned since it stubbornly believes it is correct.

In contrast, we argue that human interventions are often intentional and occur because the robot is doing something wrong. While the robot’s original behavior may have been optimal with respect to its pre-defined objective function, the fact that a human intervention was necessary implies that the original objective function was not quite right. Thus, physical human interactions are no longer disturbances but rather informative observations about what the robot’s true objective should be. With this in mind, we take inspiration from inverse reinforcement learning (IRL), where the robot observes some behavior (e.g., being pushed away from the table) and tries to infer an unknown objective function (e.g., “stay farther away from the table”). Note that while many IRL methods focus on the robot doing better the next time it performs the task, we focus on the robot completing its current task correctly.

Formalizing Reacting to pHRI

With our insight on physical human-robot interactions, we can formalize pHRI as a dynamical system, where the robot is unsure about the correct objective function and the human’s interactions provide it with information. This formalism defines a broad class of pHRI algorithms, which includes existing methods such as impedance control, and enables us to derive a novel online learning method.

We will focus on two parts of the formalism: (1) the structure of the objective function and (2) the observation model that lets the robot reason about the objective given a human physical interaction. Let $x$ be the robot’s state (e.g., position and velocity) and $u_R$ be the robot’s action (e.g., the torque it applies to its joints). The human can physically interact with the robot by applying an external torque, called $u_H$, and the robot moves to the next state via its dynamics, $\dot{x} = f(x,u_R+u_H)$.

The Robot Objective: Doing the Task Right with Minimal Human Interaction

In pHRI, we want the robot to learn from the human, but at the same time we do not want to overburden the human with constant physical intervention. Hence, we can write down an objective for the robot that optimizes both completing the task and minimizing the amount of interaction required, ultimately trading off between the two.

$$r(x,u_R,u_H;\theta) = \theta^{\top} \phi(x,u_R,u_H) - ||u_H||^2$$

Here, $\phi(x,u_R,u_H)$ encodes the task-related features (e.g., “distance to table”, “distance to human”, “distance to goal”) and $\theta$ determines the relative weight of each of these features. In the function, $\theta$ encapsulates the true objective – if the robot knew exactly how to weight all the aspects of its task, then it could compute how to perform the task optimally. However, this parameter is not known by the robot! Robots will not always know the right way to perform a task, and certainly not the human-preferred way.

The Observation Model: Inferring the Right Objective from Human Interaction

As we have argued, the robot should observe the human’s actions to infer the unknown task objective. To link the direct human forces that the robot measures with the objective function, the robot uses an observation model. Building on prior work in maximum entropy IRL as well as the Bolzmann distributions used in cognitive science models of human behavior, we model the human’s interventions as corrections which approximately maximize the robot’s expected reward at state $x$ while taking action $u_R+u_H$. This expected reward emcompasses the immediate and future rewards and is captured by the $Q$-value:

$$P(u_H \mid x, u_R; \theta) \propto e^{Q(x,u_R+u_H;\theta)}$$

Intuitively, this model says that a human is more likely to choose a physical correction that, when combined with the robot’s action, leads to a desirable (i.e., high-reward) behavior.

Learning from Physical Human-Robot Interactions in Real-Time

Much like teaching another human, we expect that the robot will continuously learn while we interact with it. However, the learning framework that we have introduced requires that the robot solve a Partially Observable Markov Decision Process (POMDP); unfortunately, it is well known that solving POMDPs exactly is at best computationally expensive, and at worst intractable. Nonetheless, we can derive approximations from this formalism that can enable the robot to learn and act while humans are interacting.

To achieve such in-task learning, we make three approximations summarized below:

1) Separate estimating the true objective from solving for the optimal control policy. This means at every timestep, the robot updates its belief over possible $\theta$ values, and then re-plans an optimal control policy with the new distribution.

2) Separate planning from control. Computing an optimal control policy means computing the optimal action to take at every state in a continuous state, action, and belief space. Although re-computing a full optimal policy after every interaction is not tractable in real-time, we can re-compute an optimal trajectory from the current state in real-time. This means that the robot first plans a trajectory that best satisfies the current estimate of the objective, and then uses an impedance controller to track this trajectory. The use of impedance control here gives us the nice properties described earlier, where people can physically modify the robot’s state while still being safe during interaction.

Looking back at our estimation step, we will make a similar shift to trajectory space and modify our observation model to reflect this:

$$P(u_H \mid x, u_R; \theta) \propto e^{Q(x,u_R+u_H;\theta)} \rightarrow P(\xi_H \mid \xi_R; \theta) \propto e^{R(\xi_H, \xi_R;\theta)}$$

Now, our observation model depends only on the cumulative reward $R$ along a trajectory, which is easily computed by summing up the reward at each timestep. With this approximation, when reasoning about the true objective, the robot only has to consider the likelihood of a human’s preferred trajectory, $\xi_H$, given the current trajectory it is executing, $\xi_R$.

But what is the human’s preferred trajectory, $\xi_H$? The robot only gets to directly measure the human’s force $u_H$. One way to infer what is the human’s preferred trajectory is by propagating the human’s force throughout the robot’s current trajectory, $\xi_R$. Figure 1. builds up the trajectory deformation based on prior work from Losey and O’Malley, starting from the robot’s original trajectory, then the force application, and then the deformation to produce $\xi_H$.

Fig 1. To infer the human’s prefered trajectory given the current planned trajectory, the robot first measures the human’s interaction force, $u_H$, and then smoothly deforms the waypoints near interaction point to get the human’s preferred trajectory, $\xi_H$.

3) Plan with maximum a posteriori (MAP) estimate of $\theta$. Finally, because $\theta$ is a continuous variable and potentially high-dimensional, and since our observation model is not Gaussian, rather than planning with the full belief over $\theta$, we will plan only with the MAP estimate. We find that the MAP estimate under a 2nd order Taylor Series Expansion about the robot’s current trajectory with a Gaussian prior is equivalent to running online gradient descent:

$$\theta^{t+1} = \theta^{t} + \alpha(\Phi(\xi^t_H) - \Phi(\xi^t_R))$$

At every timestep, the robot updates its estimate of $\theta$ in the direction of the cumulative feature difference, $\Phi(\xi) = \sum_{x^t \in \xi} \phi(x^t)$, between its current optimal trajectory and the human’s preferred trajectory. In the Learning from Demonstration literature, this update rule is analogous to online Max Margin Planning; it is also analogous to coactive learning, where the user modifies waypoints for the current task to teach a reward function for future tasks.

Ultimately, putting these three steps together leads us to an elegant approximate solution to the original POMDP. At every timestep, the robot plans a trajectory $\xi_R$ and begins to move. The human can physically interact, enabling the robot to sense their force $u_H$. The robot uses the human’s force to deform its original trajectory and produce the human’s desired trajectory, $\xi_H$. Then the robot reasons about what aspects of the task are different between its original and the human’s preferred trajectory, and updates $\theta$ in the direction of that difference. Using the new feature weights, the robot replans a trajectory that better aligns with the human’s preferences.

For a more thorough description of our formalism and approximations, please see our recent paper from the 2017 Conference on Robot Learning.

Learning from Humans in the Real World

To evaluate the benefits of in-task learning on a real personal robot, we recruited 10 participants for a user study. Each participant interacted with the robot running our proposed online learning method as well as a baseline where the robot did not learn from physical interaction and simply ran impedance control.

Fig 2. shows the three experimental household manipulation tasks, in each of which the robot started with an initially incorrect objective that participants had to correct. For example, the robot would move a cup from the shelf to the table, but without worrying about tilting the cup (perhaps not noticing that there is liquid inside).

Fig 2. Trajectory generated with initial objective marked in black, and the desired trajectory from true objective in blue. Participants need to correct the robot to teach it to hold the cup upright (left), move closer to the table (center), and avoid going over the laptop (right).

We measured the robot’s performance with respect to the true objective, the total effort the participant exerted, the total amount of interaction time, and the responses of a 7-point Likert scale survey.

In Task 1, participants have to physically intervene when they see the robot tilting the cup and teach the robot to keep the cup upright.

Task 2 had participants teaching the robot to move closer to the table.

For Task 3, the robot’s original trajectory goes over a laptop. Participants have to physically teach the robot to move around the laptop instead of over it.

The results of our user studies suggest that learning from physical interaction leads to better robot task performance with less human effort. Participants were able to get the robot to execute the correct behavior faster with less effort and interaction time when the robot was actively learning from their interactions during the task. Additionally, participants believed the robot understood their preferences more, took less effort to interact with, and was a more collaborative partner.

Fig 3. Learning from interaction significantly outperformed not learning for each of our objective measures, including task cost, human effort, interaction time.

Ultimately, we propose that robots should not treat human interactions as disturbances, but rather as informative actions. We showed that robots imbued with this sort of reasoning are capable of updating their understanding of the task they are performing and completing it correctly, rather than relying on people to guide them until the task is done.

This work is merely a step in exploring learning robot objectives from pHRI. Many open questions remain including developing solutions that can handle dynamical aspects (like preferences about the timing of the motion) and how and when to generalize learned objectives to new tasks. Additionally, robot reward functions will often have many task-related features and human interactions may only give information about a certain subset of relevant weights. Our recent work in HRI 2018 studied how a robot can disambiguate what the person is trying to correct by learning about only a single feature weight at a time. Overall, not only do we need algorithms that can learn from physical interaction with humans, but these methods must also reason about the inherent difficulties humans experience when trying to kinesthetically teach a complex – and possibly unfamiliar – robotic system.

---

Thank you to Dylan Losey and Anca Dragan for their helpful feedback in writing this blog post. This article was initially published on the BAIR blog, and appears here with the authors’ permission.

---

This post is based on the following papers:

• A. Bajcsy* , D.P. Losey*, M.K. O’Malley, and A.D. Dragan. Learning Robot Objectives from Physical Human Robot Interaction. Conference on Robot Learning (CoRL), 2017.

• A. Bajcsy , D.P. Losey, M.K. O’Malley, and A.D. Dragan. Learning from Physical Human Corrections, One Feature at a Time. International Conference on Human-Robot Interaction (HRI), 2018.

Last week my colleague Dieter Vanderelst presented our paper: The Dark Side of Ethical Robots at AIES 2018 in New Orleans.

I blogged about Dieter’s very elegant experiment here, but let me summarise. With two NAO robots he set up a demonstration of an ethical robot helping another robot acting as a proxy human, then showed that with a very simple alteration of the ethical robot’s logic it is transformed into a distinctly unethical robot – behaving either competitively or aggressively toward the proxy human.

Here are our paper’s key conclusions:

The ease of transformation from ethical to unethical robot is hardly surprising. It is a straightforward consequence of the fact that both ethical and unethical behaviours require the same cognitive machinery with – in our implementation – only a subtle difference in the way a single value is calculated. In fact, the difference between an ethical (i.e. seeking the most desirable outcomes for the human) robot and an aggressive (i.e. seeking the least desirable outcomes for the human) robot is a simple negation of this value.

On the face of it, given that we can (at least in principle) build explicitly ethical machines then it would seem that we have a moral imperative to do so; it would appear to be unethical not to build ethical machines when we have that option. But the findings of our paper call this assumption into serious doubt. Let us examine the risks associated with ethical robots and if, and how, they might be mitigated. There are three.

1. First there is the risk that an unscrupulous manufacturermight insert some unethical behaviours into their robots in order to exploit naive or vulnerable users for financial gain, or perhaps to gain some market advantage (here the VW diesel emissions scandal of 2015 comes to mind). There are no technical steps that would mitigate this risk, but the reputational damage from being found out is undoubtedly a significant disincentive. Compliance with ethical standards such as BS 8611 guide to the ethical design and application of robots and robotic systems, or emerging new IEEE P700X ‘human’ standards would also support manufacturers in the ethical application of ethical robots. 
2. Perhaps more serious is the risk arising from robots that have user adjustable ethics settings. Here the danger arises from the possibility that either the user or a technical support engineer mistakenly, or deliberately, chooses settings that move the robot’s behaviours outside an ‘ethical envelope’. Much depends of course on how the robot’s ethics are coded, but one can imagine the robot’s ethical rules expressed in a user-accessible format, for example, an XML like script. No doubt the best way to guard against this risk is for robots to have no user adjustable ethics settings, so that the robot’s ethics are hard-coded and not accessible to either users or support engineers. 
3. But even hard-coded ethics would not guard against undoubtedly the most serious risk of all, which arises when those ethical rules are vulnerable to malicious hacking. Given that cases of white-hat hacking of cars have already been reported, it’s not difficult to envisage a nightmare scenario in which the ethics settings for an entire fleet of driverless cars are hacked, transforming those vehicles into lethal weapons. Of course, driverless cars (or robots in general) without explicit ethics are also vulnerable to hacking, but weaponising such robots is far more challenging for the attacker. Explicitly ethical robots focus the robot’s behaviours to a small number of rules which make them, we think, uniquely vulnerable to cyber-attack.

Ok, taking the most serious of these risks: hacking, we can envisage several technical approaches to mitigating the risk of malicious hacking of a robot’s ethical rules. One would be to place those ethical rules behind strong encryption. Another would require a robot to authenticate its ethical rules by first connecting to a secure server. An authentication failure would disable those ethics, so that the robot defaults to operating without explicit ethical behaviours. Although feasible, these approaches would be unlikely to deter the most determined hackers, especially those who are prepared to resort to stealing encryption or authentication keys.

It is very clear that guaranteeing the security of ethical robots is beyond the scope of engineering and will need regulatory and legislative efforts. Considering the ethical, legal and societal implications of robots, it becomes obvious that robots themselves are not where responsibility lies. Robots are simply smart machines of various kinds and the responsibility to ensure they behave well must always lie with human beings. In other words, we require ethical governance, and this is equally true for robots with or without explicit ethical behaviours.

Two years ago I thought the benefits of ethical robots outweighed the risks. Now I’m not so sure. I now believe that – even with strong ethical governance – the risks that a robot’s ethics might be compromised by unscrupulous actors are so great as to raise very serious doubts over the wisdom of embedding ethical decision making in real-world safety critical robots, such as driverless cars. Ethical robots might not be such a good idea after all.

Here is the full reference to our paper:

Vanderelst D and Winfield AFT (2018), The Dark Side of Ethical Robots, AAAI/ACM Conf. on AI Ethics and Society (AIES 2018), New Orleans.

The two "fingers" of the robot gripper have built-in intelligence and advanced technology that mimics the way humans instinctively use our sense of touch when we grab things to move them.

While the vision systems of old were unreliable, clunky and expensive, today’s systems are anything but. Proper vision systems can make the difference between an efficient robotic system and one that is not working optimally.

Workers at the Fuji Xerox R&D Square in Japan are about to welcome a new workmate who will take on many of their mundane tasks and promote collaboration.

For more than a decade, biomimetic robots have been deployed alongside live animals to better understand the drivers of animal behavior, including social cues, fear, leadership, and even courtship. The encounters have always been unidirectional; the animals observe and respond to the robots. But in the lab of Maurizio Porfiri, a professor of mechanical and aerospace engineering at the NYU Tandon School of Engineering, the robots can now watch back.

Mariella Moon for EnGadget: While the team's UAVs are scanning the skies, their tactical plane will focus on what's happening down below. The plane is equipped with high-resolution CCTVs with facial recognition

As you can see, the continued growth in the creation of drones only shows the shocking improvement in the world of software and tech devices. Just like Artificial Intelligence, it’s another fantastic glimpse into the future!

Designing the right RTOS architecture requires delicate attention and decisions. Basic real-time operating system services such as inter-process communication, process synchronization, process management and interrupt handling must unfold efficiently.

In this episode, Audrow Nash speaks with Karl Weaver (魏卡爾), formerly the Original Equipment Manufacturer Business Development Director for Oasis Smart SIM. Weaver discusses how wearable technology is growing as a form of payment system in China. He speaks about wireless technology, including Near-Field Communications (NFC) and Embedded SIM cards (eSIM), in wearable technology and in other applications, such as bike rental.

Karl Weaver (魏卡爾)

Since the recording of this interview, Karl has begun working for ARM promoting eSIM and iUICC to Greater China and Asia.

Karl Weaver (魏卡爾) previously worked as the Original Equipment Manufacturer Business Development Director (North America and Northeast Asia) for Oasis Smart SIM. Prior to Oasis, Karl worked for Rivetz Corp, a Massachusetts-based start-up to promote developer tools for design-in of TEE-enables applications on Smartphones for payment and security. Karl also spent 5 years working in China for Gemalto (and Trustonic) as liaison and evangelist of embedded Mobile Near-Field Communications Payments & TEE security technologies to the OEM Smartphone/Tablet PC ecosystem. Weaver has a B.S degree in Business Management from Salve Regina University and a Certification in Mandarin Chinese Language, Customs, and Culture from National Taiwan Normal University.

 

Links

• Download mp3 (17.5 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

Katie Fehrenbacher for GreenTechMedia: For drone makers, that means a drone could fly farther and longer on a single charge. For example, a drone surveying acres of crop land could fly for an hour instead of just 20 minutes with a standard battery.

By Benjamin Boettner

Because of their high precision and speed, Delta robots are deployed in many industrial processes, including pick-and-place assemblies, machining, welding and food packaging. Starting with the first version developed by Reymond Clavel for a chocolate factory to quickly place chocolate pralines in their packages, Delta robots use three individually controlled and lightweight arms that guide a platform to move fast and accurately in three directions. The platform is either used as a stage, similar to the ones being used in flight simulators, or coupled to a manipulating device that can, for example, grasp, move, and release objects in prescribed patterns. Over time, roboticists have designed smaller and smaller Delta robots for tasks in limited workspaces, yet shrinking them further to the millimeter scale with conventional manufacturing techniques and components has proven fruitless.

Reported in Science Robotics, a new design, the milliDelta robot, developed by Robert Wood’s team at Harvard’s Wyss Institute for Biologically Inspired Engineering and John A. Paulson School of Engineering and Applied Sciences (SEAS) overcomes this miniaturization challenge. By integrating their microfabrication technique with high-performance composite materials that can incorporate flexural joints and bending actuators, the milliDelta can operate with high speed, force, and micrometer precision, which make it compatible with a range of micromanipulation tasks in manufacturing and medicine.

In 2011, inspired by pop-up books and origami, Wood’s team developed a micro-fabrication approach that enables the assembly of robots from flat sheets of composite materials. Pop-up MEMS (short for “microelectromechanical systems”) manufacturing has since been used for the construction of dynamic centimeter-scale machines that can simply walk away, or, as in the case of the RoboBee, can fly. In their new study, the researchers applied their approach to develop a Delta robot measuring a mere 15 mm-by-15 mm-by-20 mm.

“The physics of scaling told us that bringing down the size of Delta robots would increase their speed and acceleration, and pop-up MEMS manufacturing with its ability to use any material or combination of materials seemed an ideal way to attack this problem,” said Wood, who is a Core Faculty member at the Wyss Institute and co-leader of its Bioinspired Robotics platform. Wood is also the Charles River Professor of Engineering and Applied Sciences at SEAS. “This approach also allowed us to rapidly go through a number of iterations that led us to the final milliDelta.”

The milliDelta design incorporates a composite laminate structure with embedded flexural joints that approximate the more complicated joints found in large scale Delta robots. “With the help of an assembly jig, this laminate can be precisely folded into a millimeter-scale Delta robot. The milliDelta also utilizes piezoelectric actuators, which allow it to perform movements at frequencies 15 to 20 times higher than those of other currently available Delta robots,” said first-author Hayley McClintock, a Wyss Institute Staff Researcher on Wood’s team.

In addition, the team demonstrated that the milliDelta can operate in a workspace of about seven cubic millimeters and that it can apply forces and exhibit trajectories that, together with its high frequencies, could make it ideal for micromanipulations in industrial pick-and-place processes and microscopic surgeries such as retinal microsurgeries performed on the human eye.

The work by Wood’s team demonstrating the enhanced speed and control of their milliDelta robot at the millimeter scale opens entirely new avenues of development for industrial and medical robots, which are currently beyond the reach of existing technologies. Donald Ingber

Putting the milliDelta’s potential for microsurgeries and other micromanipulations to a first test, the researchers explored their robot as a hand tremor-cancelling device. “We first mapped the paths that the tip of a toothpick circumscribed when held by an individual, computed those, and fed them into the milliDelta robot, which was able to match and cancel them out,” said co-first author Fatma Zeynep Temel, Ph.D., a SEAS Postdoctoral Fellow in Wood’s team. The researchers think that specialized milliDelta robots could either be added on to existing robotic devices, or be developed as standalone devices like, for example, platforms for the manipulation of cells in research and clinical laboratories.

“The work by Wood’s team demonstrating the enhanced speed and control of their milliDelta robot at the millimeter scale opens entirely new avenues of development for industrial and medical robots, which are currently beyond the reach of existing technologies. It’s yet another example of how our Bioinspired Robotics platform is leading the way into the future,” said Wyss Institute Founding Director Donald Ingber, M.D., Ph.D., who is also the Judah Folkman Professor of Vascular Biology at HMS and the Vascular Biology Program at Boston Children’s Hospital, as well as Professor of Bioengineering at SEAS.

Other authors on the study are Neel Doshi, a SEAS Graduate Student, and Je-sung Koh, Ph.D., a former Postdoctoral Fellow on Wood’s Wyss Institute/SEAS team and now Assistant Professor at Ajou University in Korea. The work was funded by Harvard’s Wyss Institute for Biologically Inspired Engineering and a National Defense Science and Engineering Graduate Fellowship.

• PUBLICATION – Science Robotics: The milliDelta: A high-bandwidth, high-precision, millimeter-scale Delta robot