All posts by Oliver Mitchell

Last year, Intel partnered with Lady Gaga on the Super Bowl Halftime Show to showcase its latest aerial technology called “Shooting Star.” Intel did a reprise performance of its Shooting Star technology for Singapore’s 52nd birthday this past week. Instead of fireworks, the tech-savvy country celebrated its National Day Parade with a swarm of 300 LED drones animating the night sky with shapes, logos, and even a map of the country.

Intel’s global drone chief, Anil Nanduri, explained, “There’s considerably more operational complexity in handling a 300 drone fleet, compared with 100 drones in a show. It’s like juggling balls in your hand. You may be able to juggle three, but if you juggle nine, you may have to throw them higher and faster to get more time.” Earlier this year, Intel first showcased its 300 drone show at Coachella music festival on the heels of claiming the Guinness World Record of a 500 drone performance.

Choreographed drones are winning the hearts of Cirque du Soleil theatergoers with a fleet of flying acrobatic vehicles dancing around its human performers. These drones are the brain child of Professor Raffaello D’Andrea of the ETH Zurich, Switzerland and his new startup Verity Studios. D’Andrea is probably best known as one of the three founders of Kiva Systems and now he is taking the same machine intelligence that sorts and delivers goods within Amazon’s warehouses to safely wow audiences worldwide.  The flying lampshades (shown in the video below) are actually autonomous drones that magically synchronize with the dancers, without safety nets or human operators.

Verity’s customer, Cirque du Soleil’s Chief Creative Officer Jean-Francois Bouchard, said D’Andrea’s “flying machines are unquestionably one of the most important statements of the PARAMOUR show.” The key to the flying machines’ success over 7,000 autonomous flights on Broadway is its proprietary technologies that enable multiple self-piloted drones to be synchronized within flight. Verity’s drone is part of a larger performance system called “Stage Flyers.”

The Stage Flyer platform has proven itself in the field, flying next to thousands of people each evening by having built-in redundancy to any single failure. According to Verity’s website, the system is “capable of continuing operation in spite of a failed battery, a failed motor, a failed connector, a failed propeller, a failed sensor, or a failure of any other component. This is achieved through the duplication of critical components and the use of proprietary algorithms, which enable safe emergency responses to component failures.” This means that the drones can operate safely around audiences and performers alike, carrying payloads of cameras, mirrors, and special lighting effects. As shown in the above diagram, the drone system includes a fleet of self-piloted drones that utilize one positioning system and control unit. The company boasts that its system only takes a few hours to install, calibrate and learn how to operate.

Drone swarms are not just for entertainment, as today there are a number of upstarts and established players utilizing these mechanics for e-commerce fulfillment centers. Last June, Amazon was issued a patent for a “Multi-Level Fulfillment Center for Unmanned Aerial Vehicles.” The beehive-looking distribution center (above) is designed to facilitate traffic between inbound and outbound delivery drones. The patent illustration details “multiple levels with multiple landing and take-off locations, depending on local zoning regulations.”

This is all part of Amazon’s larger plan to grow its robotic workforce over the next two to three years. Instead of human truck drivers, the patent displays delivery bays that open and close automatically based on the direction of the drones and interior platforms that cycle around the hive. CB Insights reports that the patent describes “impact dampeners,” such as nets, for receiving inbound drones and “launch assist mechanisms,” such as fans, for launching outbound drones. It appears that Amazon will be looking again to technology like D’Andrea’s research to revolutionize its global network of warehouses with synchronized swarms of drones that safely soar above human workers.

Amazon’s competitor Walmart announced last summer its plans to utilize swarms of indoor drones for inventory management, replacing the need for people climbing dangerous ladders to manually scan labels. The New York Times first reported last July that the retailer applied for a FAA exemption to begin testing drones inside its massive distribution centers. Shekar Natarajan, the vice president of last mile and emerging science for Walmart, demonstrated for the newspaper how swarms of drones could easily move up and down aisles, from floor to ceiling, to scan 30 barcode images a second (an efficiency that would be impossible for even the most agile humans). Walmart has publicly boasted that it will spend close to $3 billion on new technology and other cost-saving investments to bolster its e-commerce business which is growing, but at a slower pace than its nemesis Amazon.

The race to dominate the warehouse has led to increased investments in the logistics sector and even an accelerator dedicated solely to technology around distribution centers. Chattanooga, Tennessee-based Dynamo Accelerator showcased last May its second cohort of startups. One of the most successful showings was Chicago-based Corvus Robotics, a software company that uses indoor aerial drones to scan inventory (similar to the Walmart example above). According to Dynamo’s managing directors, Corvus is building enabling tools that allow operators to fly drones autonomously, scan & sync barcodes, and enter the SKU data into the existing warehouse management system.

Santosh Sankar, the director at Dynamo, explained his accelerator’s mission succinctly in a recent blog post: “We believe our focus and hands-on approach is one of our value-adds. As such, we’re leaning into being seed investors and upholding our commitment to transforming our industry by focusing on our founders and our corporate partners. We’ve opted to not hold a quota for our programs and hone in on companies we can truly help because that ultimately makes for good seed investments.” Sankar added that several of the program participants “are already well on their way to generate ($1 million or more) in annual revenue and/or have raised their initial round of capital.”

Corvus may be the latest indoor drone startup to enter an already crowded warehouse market, which includes established players like the Hardis Group, Smartx, and DJI. Drones continue to amuse, amaze and evolve as the growing need for more unmanned systems in our lives appears to be almost insatiable. Next month, we plan to dig deeper into the drone market with our RobotLabNYC event series on September 19th at 6pm in WeWork Grand Central. Joining us in the discussions include thought-leaders from NASA, AUVSI, and Genius NY, RSVP today, as space is limited.

Jim Robinson of RRE Ventures said it best last month at the Silicon Dragon Conference when comparing Silicon Valley to New York, “There are two kinds of centers that have a lot of startups and technology, there are technology centers and commerce centers.” New York falls into the later category, while the Valley is the former. Sitting next to Jim, I reflected that Singapore might be in both groups, an Asian commerce hub and a leader in mechatronics. As an advocate for automation, I am often disheartened that the United States significantly lags behind its industrial counterparts in manufacturing autonomous machines. The key to a pro-job policy could be gleaning from the successes of countries like Singapore to implement America’s own ‘Robot First Plan.’

Last August, Singapore became the first country to permit autonomous taxis on its roads. Boston-based startup nuTonomy moved its operations to the Far East, which enabled the company to launch public trials weeks before Uber’s test in Pittsburgh. The secret to the company’s speed to market was to skip America’s 19,492 municipal government licensing departments to pilot its technology years before any of the other competing technologies, with the exception of Google’s Waymo. In addition to less regulatory hoops, the Singapore Land Transport Authority has partnered with nuTonomy on its rollout.

Pang Kin Keong, Singapore’s Permanent Secretary for Transport and the Chairman of its Committee on autonomous driving, said “We face constraints in land and manpower. We want to take advantage of self-driving technology to overcome such constraints, and in particular to introduce new mobility concepts which could bring about transformational improvements to public transport in Singapore.”

The company is on track to offer its driverless taxis throughout the country by 2018. Doug Park’s, nuTonomy’s COO, estimates that autonomous taxis could ultimately reduce the number of cars on Singapore’s roads from 900,000 to 300,000. Park explains, “When you are able to take that many cars off the road, it creates a lot of possibilities. You can create smaller roads, you can create much smaller car parks. I think it will change how people interact with the city going forward.” Park’s partnership with the city-state is made possible because Singapore is not straddled with the costs of aging infrastructure like many US and European cities.

Since first announcing its test in 2016, nuTonomy is on pace to expand globally with its recent partnership with ride-sharing company Lyft in a pilot in Boston. Karl Lagnemma, nuTonomy’s CEO, said: “By combining forces with Lyft in the U.S., we’ll be positioned to build the best passenger experience for self-driving cars. Both companies care immensely about solving urban transportation issues and the future of our cities, and we look forward to working with Lyft as we continue to improve our autonomous vehicle software system.”

Besides autonomous vehicles, drones have been widely embraced by Singapore’s infamously strict police department. Singaporean startup H3 Dynamics became the first company last year to launch a drone in the box solution that offers storage and charging stations in the field. H3’s “DRONEBOX” is a unique solar-based charging station that enables longer autonomous missions in areas that are typically hostile for humans. Since showcasing its technology above the streets of Singapore, H3 faces increased competition from formidable upstarts, including: Airobotics, EasyAriel, and HiveUAV.

According to its original press release, “DRONEBOX is an all-inclusive, self-powered system that can be deployed anywhere, including in remote areas where industrial assets, borders, or sensitive installations require constant monitoring. Designed as an evolution over today’s many unattended sensors and CCTV cameras installed in cities, borders, or large industrial estates, DRONEBOX innovates by giving sensors freedom of movement using drones as their vehicles. End-users can now deploy flying sensor systems at different locations, and measure just about anything, anywhere, anytime. They offer 24/7 reactivity, providing critical information to operators – even to those located thousands of miles away.”

In June, Dynamic H3 announced the opening of new drone operation centers to include Europe, America and the Middle East. Additionaly, H3 is now marketing its next generation of battery technology for extended high-value asset missions. H3’s HES Energy Systems is the product a decade-long research initiative between the company and the Singaporean government. Unlike typical drone lithium batteries that have a flight-time of 20-40 minutes, HES Energy’s developed its ground-breaking 6 hour battery (above) with a first of its kind “solid-hydrogen on demand powered system.” The combination of longer flights, self-charging stations and autonomous missions is a powerful value proposition for this Singaporean offering in differentiating itself in an already crowded unmanned flight market.

This past week, Dubai announced its plans to roll out a fleet of mini autonomous police cars for surveillance and crime prevention. This effort is part of the Middle Eastern city’s program to automate 25% of its police force over the next decade. The Gulf State’s ambitious plans were a perfect fit for Singaporean OTSAW Digital, a division of ActiV-a global tech powerhouse. Similar to nuTonomy and H3, OTSAW’s O-R3 grew out of the innovation friendly environment of the Asian republic.

OR-3 is smaller than a golf cart, and not meant actually to capture nefarious actors, but to identify suspicious activity while it is happening. Using facial recognition technology and a built-in ariel drone, the robot will begin patrolling the Dubai beat by the end of the year. The autonomous car/drone combo is almost a hybrid of nuTonomy and H3, with an array of sensors and machine intelligence technologies to survey the area via thermal imaging, license plate readers and cloud-based computing.

According to Abdullah Khalifa Al Marri, the head of the Dubai police department, the OR-3 isn’t meant to replace officers but rather to augment their skills with more efficient resources. “We seek to augment operations with the help of technology such as robots. Essentially, we aim for streets to be safe and peaceful without heavy police patrol,” said Al Marri. Last month, Dubai even deployed a humanoid-looking robot to monitor tourist attractions, dubbed Robocop, that speaks English and Arabic. According to Dubai, it plans to deploy larger humanoids that stand over 10 feet tall and go over 50 mph, while the human controller operates the device inside the robots.

Brigadier-General Khalid Nasser Al Razzouqi, Director-General of Smart Services at Dubai Police’s department, boasted, “The launch of the world’s first operational Robocop is a significant milestone for the Emirate and a step towards realizing Dubai’s vision to be a global leader in smart cities technology adoption.” In 2015, The World Economic Forum ranked the United Arab Emirates as the second most tech savvy government in the world, just behind Singapore.

As I write this article I find myself at another conference, “The State of New York: Smart Cities.” Hoping for insights about how my city will compete with the likes of these tech savvy counterparts, I was met by a group of bureaucrats touting App Competitions and Free WiFi. One speaker even suggested that the Metropolitan Transportation Authority (MTA) is run by the best executive team, even though New York’s Governor has politely called the organization dysfunctional due to multiple train derailments, signal problems, and overcrowded stations (see: The Summer of Hell). Science is not just about the possible, but the willing. Singapore’s ability to reinvent the very nature of how a city operates and partners with the private sector is proof positive that even a country founded 50 years ago can climb to the “top of the heap.”

The biblical narrative of the Garden of Eden describes how the snake became the most cursed of all beasts: “you shall walk on your belly, and you shall eat dust all the days of your life.” The reptile’s eternal punishment is no longer feared but embraced for its versatility and flexibility. The snake is fast approaching as one of the most celebrated robotic creatures for roboticists worldwide in out maneuvering rovers and humanoids.

Last week, while General Electric experienced a tumult in its management structure, its Aviation unit completed the acquisition of OC Robotics – a leader in serpent arm design. GE stated that it believes OC’s robots will be useful for jet engine maintenance, enabling repairs to be conducted while the engine is still attached to the wing by wiggling into parts where no human hand could survive. This promise translates into huge cost and time savings for maintenance and airline companies alike.

OC robots have use cases beyond avionics, including inspections of underground drilling and directional borings tens of feet below the Earth. In addition to acquiring visual data, OC’s snake is equipped with a high-pressure water jet and laser to measures the sharpness of the cutting surface. According to OC’s founder Andrew Graham, “This is faster and easier, and it keeps people safe. ” Graham seems to hit on the holy grail of robotics by combining profit and safety.

GE plans to expand the use case for its newest company. Lance Herrington, a leader at GE Aviation Services, says “Aviation applications will just be the starting point for this incredible technology.” Herrington implied that the snake technology could be adopted in the future to inspect power plants, trains, and even healthcare robots. As an example of its versatility, OC Robotics was awarded a prestigious prize by the U.K.’s Nuclear Decommissioning Authority for its LaserSnake. OC’s integrated snake-arm laser cutter was able to disassemble toxic parts of a nuclear fuel processing facility in a matter of weeks which would have taken years by humans while risking radiation exposure.

One of the most prolific inventors of robotic snake applications is Dr. Howie Choset of Carnegie Mellon University. Choset is the co-director of CMU’s Biorobotics Lab that has birthed severals startups based upon his snake technology, including Medrobotics (surgical systems); Hebi Robotics (actuators for modular robots); and Bito Robotics‘ (autonomous vehicles). Choset claims that his menagerie of metal reptiles are perfect for urban search and rescue, infrastructure repairs and medicine.

Recently, Medrobotics received FDA Clearance for its Flex Robotic System for colorectal procedures in the United States. According to the company’s press release, “Medrobotics is the first and only company to offer minimally invasive, steerable and shapeable robotic products for colorectal procedures in the U.S.” The Flex system promises a “scarfree” experience in accessing “hard-to-reach anatomy” that is just not possible with straight, rigid instruments.

“The human gastrointestinal system is full of twists and turns, and rigid surgical robots were not designed to operate in that environment. The Flex® Robotic System was. Two years ago Medrobotics started revolutionizing treatment in the head and neck in the U.S. We can now begin doing that in colorectal procedures,” said Dr. Samuel Straface, CEO.

Dr. Alessio Pigazzi, Professor of Surgery at the University of California, Irvine, exclaimed that “Medrobotics is ushering in the first of a new generation of shapeable and steerable robotic surgical systems that offer the potential to reduce the invasiveness of surgical procedures for more patients.” While Medrobotics’ system is currently only approved for use through the mouth and anus, Pigazzi looks forward to future applications whereby any natural orifices could be an entry point for true incision-less surgery.

Medrobotics was the brainchild of the collaboration of Choset with Dr. Alon Wolf of Israel’s prestigious Technion Institute of Technology. One of the earliest use cases for snake robots was by Wolf’s team in 2009 for military surveillance. As director of Technion’s BioRobotics and BioMechanics Laboratory (BRML) Wolf’s lab created the next generation of defensive snake robots for the latest terror threat, subterranean tunnels transporting suicide bombers and kidnappers.  Since the discovery of tunnels between the Gaza Strip and Israel in 2015, BRML has been working feverishly to deploy snake robots in the field of combat.

The vision for BRML’s hyper-redundant robots is to utilize its highly maneuverable actuators to sneak through tough terrain into tunnels and buildings. Once inside, the robot will provide instant scans of the environment to the command center and then leave behind sensors for continued surveillance. The robots are equipped with an array of sensors, including thermal imagers, miniature cameras, laser scanners, and laser radar with the ability of stitching seamlessly 360-degree views and maps of the targeted subterranean area. The robots, of course, would have dual uses for search & rescue and disaster recovery efforts.

Long term, Wolf would like to deploy his fleet of crawlers in search and rescue missions in urban locations and earthquakes.

“The robots we are creating at the Technion are extremely flexible and are able to manipulate delicate objects and navigate around walls. Over 400 rescue workers were killed during 9/11 because of the dangerous and unstable environment they were attempting to access and our objective is to ensure that robots are able to replace humans in such precarious situations,” explains Wolf.

It is no wonder why on his last visit to Israel, President Obama called Wolf’s vision “inspiring.”

The key takeaway from Tuesday’s RobotLabNYC forum, on “Exploring The Autonomous Future,” was humans are the key to robot adoption. Dr. Howard Morgan of First Round Capital expressed to the audience of more than 100 innovators working within the automation ecosystem, the necessity of embracing “entrepreneurial marketing” to reach customers. Tom Ryden echoed Morgan’s sentiment in his presentation about Mass Robotics, conveying his startups’ frustrations with the pace of adoption. Dr. Eric Daimler, formerly of the Obama Administration, concluded the evening succinctly by exclaiming, “we only adopt what we trust.” Trust is key for crossing the chasm.

Intuitive Surgical this past year celebrated its 17 year of operations with close to a million robotic surgeries completed last year. According to a recent Gallup Poll, medical professionals are the most trusted individuals in our society, even more than one’s clergy. The fact that robot-assisted surgery has become so routine and accepted by doctors and their patients is proof positive that in some industries we have already crossed the trust threshold.

Understanding how Intuitive’s Da Vinci robot built trust within the medical community could offer parallels to other areas of the automation industry. Robotic-assisted surgery or “telerobotics,” is the evolution of two modern technologies: 1) telepresence or telemanipulation; and 2) laparoscopic surgery. In 1987, French physician Dr. Philippe Mouret performed the first minimally invasive gallbladder surgery using an endoscope-like device to remotely guide his instruments via video to remove the damaged organ. By the 1990’s, laparoscopic surgery became commonplace, driving the demand for more precision through mechanics and computer-aided techniques. A decade later, Intuitive received FDA approval for its Da Vinci robot for general surgery, which has since been expanded for prostate, neurological, and thoracic procedures. Telerobotics evolved not just from the availability of advanced technologies, but from the demand for less invasive procedures by the most trusted people in America.

Last month, the FDA approved the Da Vinci Xi Systems, enabling Intuitive Surgical to market less expensive systems and gain marketshare with smaller medical institutions globally. “This new system enables access to Intuitive’s leading and proven robotic-assisted surgical technology at a lower price point. Customers around the globe have different needs from a clinical, cost and technology perspective; Intuitive’s goal is to meet those needs by providing a range of products and solutions: the da Vinci X System helps us continue to do so,” said CEO Dr. Gary Guthart.

According to the press release, Da Vinci X System is “a focused-quadrant surgery and features flexible port placement and 3D digital optics, while also including advanced instruments and accessories from its Xi system.” Another determining factor of Intuitive Surgical’s success is the interoperability of the instruments. Rather than just an endoscope that provides video feeds, Da Vinci is equipped with multiple end effectors that mimics traditional instruments guided by experienced surgeons telerobotically.

Patients trust the robot because it is simply augmenting their doctor’s skills with greater precision. This is reinforced by shorter recovery periods and better outcomes. Recently, Oxford published a research study which took place over a nine-year period that concluded patients who opted for robotic lobectomies had better lung cancer outcomes. As a new generation of surgeons embraces the robotic future, the market for abdominal surgical robots is expected to grow from $2.9 billion in 2017 to $12.9 billion by 2022.

Trusting robots with our bodies might seem like a difficult premise to uphold, but robots have been saving lives on the front lines since 1972. Bomb-defusing machines have been utilized in the most dangerous situations worldwide from Afghanistan to Jerusalem to New York City. Today, almost every police department and military have an arsenal of remote-controlled explosive removal devices.

Dr. Sethu Vijayakumar, director of the Edinburgh Centre for Robotics in the United Kingdom, explains, “One of the target areas, in terms of [the] use of robots, is for going into dangerous situations. Robots can go in, be operated from a safe distance, and, in a worst-case scenario, be sacrificed.”

Similar to robotic medicine, trust-based systems for the military are built by teleporting human expertise into dangerous situations. Pittsburgh-based RE2 Robotics took this concept to a new level with its Robotic Manipulation System announced last week. The RE2 System now enables users to actually use their limbs and hands to control the robot’s movements and grippers to quickly defuse explosives.

RE2 CEO explained the rationale for his new product, “Often times, you still need the human intellect to perform those tasks. But they’re dangerous, so the question is, how can we project that human capability remotely, so they’re still able to do their job and leverage the human intellect to solve a really big problem? That’s what we’re trying to do — keep the human safe, but allowing them to still do their job.”

While rover looks remarkably similar to Endeavor’s (formerly iRobot) Packbot that has been widely deployed by the US military in Iraq, Afghanistan and elsewhere, the control system is novel and more reliable in the high-pressure situations. Pedersen says, “If you’re going to project that human capability, the most human way to control it is to have it be as much like you as possible. That’s where we’ve come over the past decade, having true human-like capability. It’s no coincidence that these robots look like human torsos. These systems are a projection of you, remotely. It’s almost like an avatar, where you’re dealing with a threat out of harm’s way.”

While today the operator of RE2’s robot stands at a safe distance watching the video feed on a laptop, the company is developing a virtual reality headset accessory for the control system to enable the professional to completely immerse himself into the situation. Pedersen also plans to expand the use cases for his technology to civilian markets such as search & rescue, disaster recovery (like Fukushima) and medicine.

“Yes, people could use this technology for other means. But our charter is saving lives and extending it into new markets like health care, where we can do patient assist. [We can] help a person from a wheelchair to a bed or a wheelchair to a toilet, as the brawn for a caregiver,” touts Pedersen.

While we are years away from fully trusting autonomous systems with our lives, it appears from these two examples that the first step is enabling machines to augment our most trusted citizens. As yesterday was Father’s Day it is only appropriate I share with my readers my gift – GrillBot. The disclaimer on the box, however, does make me question when I plan to use it; fear of death is kind of a big deal!

It is unclear if Masayoshi Son, Chairman of Softbank, was one of the 17 million YouTube viewers of Boston Dynamic’s Big Dog before acquiring the company for an undisclosed amount this past Thursday. What is clear is the acquisition of Boston Dynamics by Softbank is a big deal. Softbank’s humanoid robot Pepper is trading up her dainty wheels for a pair of sturdy legs.

In expressing his excitement for the acquisition, Masayoshi Son said, “Today, there are many issues we still cannot solve by ourselves with human capabilities. Smart robotics are going to be a key driver of the next stage of the Information Revolution, and Marc and his team at Boston Dynamics are the clear technology leaders in advanced dynamic robots. I am thrilled to welcome them to the SoftBank family and look forward to supporting them as they continue to advance the field of robotics and explore applications that can help make life easier, safer and more fulfilling.”

Marc Raibert, CEO of Boston Dynamics, previously sold his company to Google in 2013. Following the departure of Andy Rubin from Google, the internet company expressed buyers remorse. Raibert’s company failed to advance from being a military contractor to a commercial enterprise. It became very challenging incorporating Boston Dynamic’s zoo of robots (mechanical dogs, cheetahs, bulls, mules and complex humanoids) into Google’s autonomous strategy. Since Rubin’s exit in 2014, rumors of buyers acquiring Boston Dynamics from Google have ranged from Toyota Research to Amazon Robotics. Softbank represents a new chapter for Raibert, and possible the entire industry.

Raibert’s statement to the press gave astute readers a peek of what to expect: “We at Boston Dynamics are excited to be part of SoftBank’s bold vision and its position creating the next technology revolution, and we share SoftBank’s belief that advances in technology should be for the benefit of humanity. We look forward to working with SoftBank in our mission to push the boundaries of what advanced robots can do and to create useful applications in a smarter and more connected world.” A quick study of the assets of both companies reveals how Boston Dynamics could help Softbank in its mission to build robots to benefit humanity.

Softbank’s premier robot is Pepper, a four-foot tall social robot that has been mostly deployed in Asia as a customer service agent. Recently, Pepper, as part of Softbank’s commitment to the Trump administration to invest $50 billion in the United States has been spotted in stores in California. As an example, Pepper proved itself as a valuable asset last year to Palo Alto’s premier tech retailer B8ta, accounting for a 6 times increase in sales. To date, there are close to 10,000 Pepper robots deployed worldwide, mostly in Asian retail stores. However, Softbank is also the owner of Sprint with 4,500 cell phone stores across the USA, and a major investor in WeWork with 140 locations globally servicing 100,000 members – could Pepper be the customer service agent or receptionist of the future?

According to Softbank’s website, Pepper is designed to be a “day-to-day companion,” with its most compelling feature being the ability to perceive emotions. Softbank boasts that their humanoid is the first robot ever to recognize moods of homo sapiens and adapt its behavior accordingly. While this is extremely relevant for selling techniques, Softbank is most proud of Pepper being the first robot to be adopted into homes in Japan. It is believed that Pepper is more than a point-of-purchase display gimmick, but an example of the next generation of caregivers for the rising elderly populations in Japan and the United States. According to the Daily Good, “Pepper could do wonders for the mental engagement and continual monitoring of those in need.” Its under $2,000 price point also provides an attractive incentive to introduce the robot into new environments, however wheel-based systems are a limitation in the home with clutter floors, stairs and other unforeseen obstacles.

Boston Dynamics is almost the complete opposite of Softbank; it is a research group spun out of MIT. Its expertise is not in social robots but in military “proofs of concepts” like futuristic combat mules. The company has touted some of the most frightening mechanical beasts to ever walk the planet from metal Cheetahs that sprint at over 25 miles per hour to mechanized dogs that scale mountains with ease to one of the largest humanoids every built that has an uncanny resemblance to Cyberdyne’s T-800. In a step towards commercialization, Boston Dynamics released earlier this year its newest monster – a wheel-biped leg robot named Handle that can easily lift over a hundred pounds and jump higher than Lebron James. Many analysts pontificated that this appeared to be Boston Dynamics attempt to prove its relevance to Google with a possible last mile delivery bot.

In an IEEE interview when Handle debuted last February, Raibert exclaimed, “Wheels are a great invention. But wheels work best on flat surfaces and legs can go anywhere. By combining wheels and legs, Handle can have the best of both worlds.” IEEE writer Evan Ackerman questioned, after seeing Handle, if the next generation of Boston Dynamic’s humanoids could feature legs with roller-skate like shoes. One is certain that Boston Dynamics is the undisputed leader of dynamic control and balance systems for complex mechanical designs.

Leading roboticist Dan Kara of ABI Research confirmed that “these [Boston Dynamics] are the world’s greatest experts on legged mobility.”

If walking is the expertise of Raibert’s team and Softbank is the leader of cognitive robotics with a seemingly endless supply of capital, the combination could be the first real striding humanoid capable of human-like emotions. By 2030 there will be 70 million people over the age of 65 years in America, with a considerable smaller amount of caregivers. To answer this call researchers are already converting current versions of Pepper into sophisticated robotic assistants. Last year, Rice University unveiled a “Multi-Purpose Eldercare Robot Assistant (MERA)” which is essentially a customized version of the Softbank’s robot. MERA is specifically designed to be a home companion for seniors that “records and analyzes videos of a person’s face and calculates vital signs such as heart and breathing rates.” Rice University partnered with IBM’s Aging-in-Place Research Lab to create MERA’s speech technology. IBM’s Lab founder, Susann Keohane, explained that Pepper “has everything bundled into one adorable self.” Now with Boston Dynamic’s legs Pepper could be a friend, physical therapist, and life coach walking side by side with its human companion.

Daniel Theobald, founder of Vecna Technologies – a healthcare robotic company, summed it best last week, “I think Softbank has made a major commitment to the future of robotics. They understand that the world economy is going to be driven by robotics more and more.”

Next Tuesday we will dive further into the implications of Softbank’s purchase of Boston Dynamics with Dr. Howard Morgan/First Round Capital, Tom Ryden/MassRobotics and Dr. Eric Daimler/Obama White House at RobotLabNYC’s event on 6/13 @ 6pm WeWork Grand Central (RSVP).

The device named “Spark” flew high above the man on stage with his hands waving in the direction of the flying object. In a demonstration of DJI’s newest drone, the audience marveled at the Coke can-sized device’s most compelling feature: gesture controls. Instead of a traditional remote control, this flying selfie machine follows hand movements across the sky. Gestures are the most innate language of mammals, and including robots in our primal movements means we have reached a new milestone of co-existence.

Madeline Gannon of Carnegie Mellon University is the designer of Mimus, a new gesture controlled robot featured in an art installation at The Design Museum in London, England. Gannon explained: “In developing Mimus, we found a way to use the robot’s body language as a medium for cultivating empathy between museum-goers and a piece of industrial machinery. Body language is a primitive yet fluid means of communication that can broadcast an innate understanding of the behaviors, kinematics and limitations of an unfamiliar machine.” Gannon wrote about her experiences recently in the design magazine Dezeen: “In a town like Pittsburgh, where crossing paths with a driverless car is now an everyday occurrence, there is still no way for a pedestrian to read the intentions of the vehicle…it is critical that we design more effective ways of interacting and communicating with them.”

So far, the biggest commercially deployed advances in human-robot interactions have been conversational agents by Amazon, Google and Apple. While natural language processing has broken new ground in artificial intelligence, the social science of its acceptability in our lives might be its biggest accomplishment. Japanese roboticist Masahiro Mori described the danger of making computer generated voices too indistinguishable from humans as the “uncanny valley.” Mori cautioned inventors from building robots that are too human sounding (and possibly looking) as the result elicits negative emotions best described as “creepy” and “disturbing.”

Recently, many toys have embraced conversational agents as a way of building greater bonds and increasing the longevity of play with kids. Barbie’s digital speech scientist, Brian Langner of ToyTalk, detailed his experiences with crossing into the “Uncanny Valley” as: “Jarring is the way I would put it. When the machine gets some of those things correct, people tend to expect that it will get everything correct.”

Kate Darling of MIT’s Media Lab, whose research centers on human-robot interactions, suggested that “if you get the balance right, people will like interacting with the robot, and will stop using it as a device and start using it as a social being.”

This logic inspired Israeli startup Intuition Robotics to create ElliQ—a bobbing head (eyeless) robot. The purpose of the animatronics is to break down barriers between its customer base of elderly patients and their phobias of technology. According to Intuition Robotics’ CEO, Dor Skuler, the range of motion coupled with a female voice helps create a bond between the device and its user. Don Norman, usability designer of ElliQ, said: “It looks like it has a face even though it doesn’t. That makes it feel approachable.”

Mayfield Robotics decided to add cute R2D2-like sounds to its newest robot, Kuri. Mayfield hired former Pixar designers Josh Morenstein and Nick Cronan of Branch Creative with the sole purpose of making Kuri more adorable. To accomplish this mission, Morenstein and Cronan gave Kuri eyes, but not a mouth as that would be, in their words “creepy.” Conan shares the challenges with designing the eyes: “Just by moving things a few millimeters, it went from looking like a dumb robot to a curious robot to a mean robot. It became a discussion of, how do we make something that’s always looking optimistic and open to listen to you?” Kuri has a remarkable similarity to Morenstein and Cronan’s former theatrical robot, EVA.

In the far extreme of making robots act and behave human, RealDoll has been promoting six thousand dollar sex robots. To many, RealDoll has crossed the “Uncanny Valley” of creepiness with sex dolls that look and talk like humans. In fact, there is a growing grassroots campaign to ban RealDoll’s products globally, as it endangers the very essence of human relationships. Florence Gildea writes on the organization’s blog: “The personalities and voices that doll owners project onto their dolls is pertinent for how sex robots may develop, given that sex doll companies like RealDoll are working on installing increasing AI capacities in their dolls and the expectation that owners will be able to customize their robots’ personalities.” The example given is how the doll expresses her “feelings” for her owner on Twitter:

Obviously a robot companion has no feelings, however it is a projection of the doll owners’. “To anthropomorphize their dolls to sustain the fantasy that they have feelings for the owner. The Twitter accounts seemingly manifest the dolls’ independent existence so that their dependence on their owners can seem to signify their emotional attachment, rather than it following inevitable from their status as objects. Immobility, then, can be misread as fidelity and devotion.” The implications of this behavior is that their female companion, albeit mechanical, enjoys “being dominated.” The fear that the Campaign Against Sex Robots expresses is the objectification of women (even robotic ones) reinforces problematic human sexual stereotypes.

Today, with technology at our fingertips, there is growing phenomena of preferring one-directional device relationships over complicated human encounters. MIT Social Sciences Professor Sherry Turkle writes in her essay, Close Engagements With Artificial Companionship, that “over-stressed, overworked, people claim exhaustion and overload. These days people will admit they’d rather leave a voice mail or send an email than talk face-to-face. And from there, they say: ‘I’d rather talk to the robot. Friends can be exhausting. The robot will always be there for me. And whenever I’m done, I can walk away.’”

In the coming years humans will communicate more with robots in their lives from experiences in the home to the office to their leisure time. The big question will be not the technical barriers, but the societal norms that will evolve to accept Earth’s newest species.

“What do we think a robot is?” asked robot designer Norm. “Some people think it should look like an animal or a person, and it should move around. Or it just has to be smart, sense the environment, and have motors and controllers.”

Norm’s answer, like beauty, could be in the eye of the beholder.

In 1985, a twenty-two year old Garry Kasparov became the youngest World Chess Champion. Twelve years later, he was defeated by the only player capable of challenging the grandmaster, IBM’s Deep Blue. That same year (1997), RoboCup was formed to take on the world’s most popular game, soccer, with robots. Twenty years later, we are on the threshold of the accomplishing the biggest feat in machine intelligence, a team of fully autonomous humanoids beating human players at FIFA World Cup soccer.

Many of the advances that have led to the influx of modern autonomous vehicles and machine intelligence are the result of decades of competitions. While Deep Blue and AlphaGo have beat the world’s best players at board games, soccer requires real-world complexities (see chart) in order to best humans on the field. This requires RoboCup teams to combine a number of mechatronic technologies within a humanoid device, such as real-time sensor fusion, reactive behavior, strategy acquisition, deep learning, real-time planning, multi-agent systems, context recognition, vision, strategic decision-making, motor control, and intelligent robot control.

Professor Daniel Lee of University of Pennsylvania’s GRASP Lab described the RoboCup challenges best, “Why is it that we have machines that can beat us in chess or Jeopardy but we can beat them in soccer? What makes it so difficult to embody intelligence into the physical world?” Lee explains, “It’s not just the soccer domain. It’s really thinking about artificial intelligence, robotics, and what they can do in a more general context.”

RoboCup has become so important that the challenge of soccer has now expanded into new leagues that focus on many commercial endeavors from social robotics, to search & rescue, to industrial applications. These leagues have a number of subcategories of competition with varying degrees of difficulty. In less than two months, international teams will convene in Nagoya, Japan for the twenty-first games. As a preview of what to expect, let’s review some of last year’s winners. And just maybe it could give us a peek at the future of automation.

RoboCup Soccer

While Iran’s human soccer team is 28th in the world, their robot counterparts (Baset Pazhuh Tehran) won 1st place in the AdultSize Humanoid competition. Baset’s secret sauce is its proprietary algorithms for motion control, perception, and path planning. According to Baset’s team description paper, the key was building a “fast and stable walk engine” based upon the success of past competitions. This walk engine was able to combine “all actuators’ data in each joint, and changing the inverse and forward kinematics” to “avoid external forces affecting robot’s stability, this feature plays an important role to keep the robot standing when colliding to the other robots or obstacles.” Another big factor was their goalkeeper that used a stereo vision sensor to detect incoming plays and win the competition by having “better percepts of goal poles, obstacles, and the opponent’s goalkeeper. To locate each object in real self-coordinating system.” The team is part of a larger Iranian corporation, Baset, that could deploy this perception in the field. Baset’s oil and gas clients could benefit from better localization techniques and object recognition for pipeline inspections and autonomous work vehicles. If by 2050 RoboCup’s humanoids will be capable of playing humans in soccer, one has to wonder if Baset’s mechanical players will spend their offseason working in the Arabian peninsula?

RoboCup Rescue 

In 2001 the RoboCup organization added simulated rescue to the course challenge, paving the way for many life-saving innovations already being embraced by first responders. The course starts with a simulated earthquake environment whereby the robot performs a search and rescue mission lasting 20 minutes. The skills are graded by overcoming a number of obstacles that are designed to assess the robot’s autonomous operation, mobility, and object manipulation. Points are given by the number of victims found by the robot, details about the victims, and the quality of the area mapped. In 2016, students from the King Mongkut’s University of Technology North Bangkok won first place with their Invigorating Robot Activity Project (or iRAP).

Similar to Baset, iRap’s success is largely based upon solving problems from previous contests where they placed consistently in the top tier. The team had a total of four robots: one autonomous robot, two tele-operative robots, and one aerial drone. Each of the robots had multiple sensors related to providing critical data, such as CO2 levels, temperature, positioning, 2D mapping, images, and two-way communications. iRap’s devices managed to navigate with remarkable ease the test environment’s rough surfaces, hard terrains, rolling floor, stairs, and inclined floor. The most impressive performer was the caged quadcopter using enhanced sensors to localize itself within an outdoor search perimeter. According to the team’s description paper, “we have developed the autonomously outdoor robot that is the aerial robot. It can fly and localize itself by GPS sensor. Besides, the essential sensors for searching the victim.” It is interesting to note that the Thai team’s design was remarkably similar to Flyability’s Gimball that won first place in the UAE’s 2015 Drones for Good Competition. Like the RoboCup winner, the Gimball was designed specifically for search & rescue missions using a lightweight carbon fiber cage.

As RoboCup contestants push the envelope of navigation mapping technologies, it is quite possible that the 2017 fleet could develop subterranean devices that could actively find victims within minutes of being buried by the earth.

RoboCup @Home

The home, like soccer, is one of the most chaotic conditions for robots to operate successfully. It is also one of the biggest areas of interest for consumers. Last year, RoboCup @Home celebrated its 10th anniversary by bestowing the top accolade to Team-Bielefeld (ToBI) of Germany. ToBi built a humanoid-like robot that was capable of learning new skills through natural language within unknown environments. According to the team’s paper, “the challenge is two-fold. On the one hand, we need to understand the communicative cues of humans and how they interpret robotic behavior. On the other hand, we need to provide technology that is able to perceive the environment, detect and recognize humans, navigate in changing environments, localize and manipulate objects, initiate and understand a spoken dialogue and analyse the different scenes to gain a better understanding of the surrounding.” In order to achieve these ambitious objectives the team created a Cognitive Interaction Toolkit (CITK) to support an “aggregation of required system artifacts, an automated software build and deployment, as well as an automated testing environment.”

Infused with its proprietary software the team’s primary robot, the Meka M1 Mobile Manipulator (left) demonstrated the latest developments in human-robot-interactions within the domestic setting. The team showcased how the Meka was able to open previously shut doors, navigate safely around a person blocking its way, and recognize and grasp many household objects. According to the team, “the robot skills proved to be very effective for designing determined tasks, including more script-like tasks, e.g. ’Follow-Me’ or ’Who-is-Who’, as well as more flexible tasks including planning and dialogue aspects, e.g. ’General-PurposeService-Robot’ or ’Open-Challenge’.”

RoboCup @Work

The @Work category debuted in 2016 with the German team from the Leibniz Universität Hannover (LUHbots) winning first place. While LUHbots’ hardware was mostly off the shelf parts (a mobile robot KUKA youBot), the software utilized a number of proprietary algorithms. According to the paper, “in the RoboCup we use this software e.g. to grab objects using inverse kinematics, to optimize trajectories and to create fast and smooth movements with the manipulator. Besides the usability the main improvements are the graph based planning approach and the higher control frequency of the base and the manipulator.” The key to using this approach within a factory setting is its robust object recognition. The paper explains, “the robot measures the speed and position of the object. It calculates the point and time where the object reaches the task place. The arm moves above the calculated point. Waits for the object and accelerates until the arm is directly above the moving-object with the same speed. Overlapping the down movement with the current speed until gripping the object. The advantage of this approach is that while the calculated position and speed are correct every orientation and much higher objects can be gripped.”

Similar to other finalists, LUHbots’ object recognition software became the determining factor to its success. RoboCup’s goal of playing WorldCup soccer with robots may seem trivial, but its practice is anything but meaningless. In each category, the advances developed on the field of competitive science are paying real dividends on a global scale across many industries.

In the words of the RoboCup mission statement: “The ultimate goal is to ‘develop a robot soccer team which beats the human world champion team.’ (A more modest goal is ‘to develop a robot soccer team which plays like human players.’) Needless to say, the accomplishment of the ultimate goal will take decades of effort. It is not feasible with current technologies to accomplish this in the near future. However, this goal can easily lead to a series of well-directed subgoals. Such an approach is common in any ambitious or overly ambitious project. In the case of the American space program, the Mercury project and the Gemini project, which manned an orbital mission, were two precursors to the Apollo mission. The first subgoal to be accomplished in RoboCup is ‘to build real and software robot soccer teams which play reasonably well with modified rules.’ Even to accomplish this goal will undoubtedly generate technologies, which will impact a broad range of industries.”

* Editor’s note: thank you to Robohub for providing a twenty-year history of RoboCup videos. 

JD Claridge’s story epitomizes the current state of the drone industry. Claridge, founder of xCraft, is best known for being the first contestant on Shark Tank to receive money from all the Sharks – even Kevin O’Leary! Walking the floor of Xponential 2017, the annual convention of the Association for Unmanned Vehicle Systems Integration (AUVSI), Claridge remarked to me how the drone industry has grown up since his TV appearance.

Claridge has gone from pitching cellphone cases that turn into drones (aka phonedrone) to solving mission critical problems. The age of fully autonomous flight is near and the drone industry is finally recovering from the hangover of overhyped Kickstarter videos (see Lily drone’s $34 million fraud). xCraft’s pivot to lightweight, power efficient, enterprise drones is an example of this evolved marketplace. During the three days of Xponential 2017, several far-reaching announcements were made between stalwarts of the tech industry and aviation startups. Claridge introduced me to his new partner, Rajant, which is a leader in industrial wireless networks. xCraft’s latest models utilize Rajant’s mesh networks to launch swarms of drones with one controller. More drones flying simultaneously enables users to maximize the flight time limitations of lithium batteries by covering greater areas within a single mission.

Bob Schena, Rajant’s CEO, said, “Rajant’s network technology now makes it possible for one pilot to operate many aircrafts concurrently, with flight times of 45 minutes. We’re pleased to partner with xCraft and bring more intelligence, mobility and autonomy to UAV communication infrastructures covering greater aerial distances while supporting various drone payloads.”

The battery has been the Achilles heel of the small drone industry since inception. While large winged craft relies heavily on fossil fuels, multirotor battery-operated drones have been plagued with shorter missions of under 45 minutes. Innovators like Claridge are leading the way for a new wave of creative solutions:

Solar Powered Wings 

Airbus showcased its Zephyr drone products or HAPS (High Altitude Pseudo-Satellite) UAVs using solar-winged craft for power. Zephyr UAVs can fly for months at a time, saving thousands of tons of fuel. The HAPS also offers a number of lightweight payload options from voice communications to persistent internet to real-time surveillance. Airbus was not the only solar solution on display; there were a handful of Chinese upstarts and solar cell purveyors for retrofitting existing aircrafts.

Hybrid Fuel Solutions  

In the Startup Pavilion, William Fredericks of the Advanced Aircraft Company (AAC) demoed a novel technology using a hybrid of diesel fuel and lithium batteries with flexible fixed wings and multirotors, resulting in over 3 hours of flying time. AAC’s prototype, the Hercules (above) is remarkably lightweight and fast. Fredricks is an aircraft designer by trade with 12 designs flying in the air, including NASA’s Greased Lightning that looks remarkably similar to Boeing’s Osprey. The Hercules is available for sale on the company’s website for multiple use cases, including: agricultural, first responders, and package delivery. It is interesting to note that a few rows from Frederick’s booth was his former employer, NASA, promoting their new Autonomy Incubator for “intelligent flight systems” and its “autonomy innovation lab,” (definitely an incubator to watch).

Vertical Take Off & Landing

In addition to hybrid fuel strategies, entrepreneurs are also rethinking the launch procedures. AAC’s Hercules and XCraft’s commercial line of drones vertically takeoff to reduce wind resistance and maximize energy consumption. Australian Startup Iridium Dynamics takes this approach to a new level with astonishing results. Its winged craft, Halo, uses a patent-pending “hover thrust” of its entire craft so its wings actually create the vertical lift to hover with minimal power. The drone also has two rotors to fly horizontally. According to Dion Gonano, Control Systems Engineer, it can fly for over 2 hours. The Halo also lands vertically into a stationary mechanical arm. While the website lists a number of commercial applications for this technology, it was unclear in my discussions with Gonano if they have deployed this technology in real tests.

New Charging Efficiencies

Prior to Xponential, Seattle-based WiBotic announced the closing of its $2.5 seed round to fund its next generation of battery charging technologies. The company has created a novel approach to wireless inductive charging for robotics. Its wireless inductive charging platform includes a patent-pending auto detect feature that can begin recharging once the robot enters the proximity of the base station, even during flight. According to Dr. Ben Waters, (CEO), its charge is faster than traditional solutions presently on the market. Dr. Waters demonstrated for me its suite of software tools that monitor battery performance, providing clients with a complete power management analytics platform. WiBotic is already piloting its technology with leading commercial customers in the energy and security sectors. WiBotic is the first inductive charging platform; other companies have created innovating battery-swapping techniques. Airobotics unique drone storage box that is deployed currently at power plants in Israel, includes a robotic arm, housed inside, that services the robot post flight by switching out the payload and battery:

Reducing Payload Weight

In addition to aircraft design, payload weight is a big factor of battery drain. A growing trend within the industry is miniaturizing the size and cost of the components. Ultimately, the mission of a drone is directly related to the type of payload from cameras for collecting images to precise measurements using Light Detection and Ranging sensors (or Lidar). Lidar is typically deployed in autonomous vehicles to provide the most precise position for the robot in a crowded area, like a self-driving car on the road. However, Lidar is currently extremely expensive and large for many multirotor surveys. Chris Brown of Z-Senz, a former scientist with the The National Institute of Standards and Technology (NIST), hopes to change the landscape of drones with his miniaturized Lidar sensor. Brown’s reduced sensor, SKY1, offers major advantages for size, weight, and power consumption without losing accuracy of high distance sensing. A recent study estimates the Lidar market is expected to exceed $5 billion by 2022, with Velodyne and Quanergy already gaining significant investment. Z-Senz is aiming to be commercially available by 2018.

Lidar is not the only measuring methodology, Global Positioning Solutions (GPS) have been deployed widely. Two of the finalists of the Xponetial Startup Showdown were startups focused on reducing GPS chip sizes and increasing functionality. Inertial Sense has produced a chip the size of a dime that is capable of housing an Inertial Measurement Unit (IMU), Attitude Heading Reference System (AHRS), and GPS-aided Inertial Navigation System (INS). Their website claims that their “advanced algorithms fuse output from MEMs inertial sensors, magnetometers, barometric pressure, and a high-sensitivity GPS (GNSS) receiver to deliver fast, accurate, and reliable attitude, velocity, and position even in the most dynamic environments.” The chips and micro navigation accessories are available on the company’s e-store.

The winner of the Showdown, uAvionix, is a leading developer of avionics for both manned and unmanned flight. Their new transceivers and transponders claim to be “the smallest, and lightest and most affordable on the market” (already GPS is a commodity). uAvionix presented its “Ping Network System that reduces weight on average by 40% as compared to the two-piece installations.” The Ping products also claim barometric altitude precision with accuracy beyond 80,000 ft.

Paul Beard, CEO of uAvionix, said, “our customers have asked for even smaller and lighter solutions; integrating the transceivers, GPS receivers, GPS antennas, and barometric pressure sensors into a single form factor facilitates easier installation and lowers weight and power draw requirements resulting in a longer usable flight time.”

As I rushed to the airport to catch my manned flight, I felt reenergized about the drone industry, although follies will persist. I mean who wouldn’t want a pool deckchair drone this summer?

This and all other autonomous subjects will be explored at RobotLabNYC’s next event with Dr. Howard Morgan (FirstRound Capital) and Tom Ryden (MassRobotics) – RSVP.