Synthetic data is on the rise in artificial intelligence. It's going to make AI cheaper, better and less biased.
It's also very obtainable and usable. In a short while, it has gone from being an experimental technology. To something, I would not hesitate to use for production AI solutions.
To illustrate that, I will build an AI that can classify the difference between apples and bananas. I will only use images of the two classes generated by another AI - In this case, using DALL-E Mini.
I will build an image classifier using only easy-to-access, free AutoAI tools.
We need around 30 images of each label, bananas and apples.
We will be using DALL-E Mini, an open-source version of NVIDIAs text-to-image model DALL-E 2.
To generate the images, you can go to https://huggingface.co/spaces/dalle-mini/dalle-mini. Here you can prompt the text-to-image model with queries such as:
"Banana on table"
"Banana on random background"
"Apple on table"
"Apple on random background"
Try to match the background you will be testing on.
The response should look like this:
Generate around 30 images of each label and save them.
To create the model, we will be using the tool Teachable Machine. Teachable machine has
Open Teachable Machine and choose Image project and then Standard image model.
Names your classes Banana and Apple.
Upload your images from DALL-E Mini to the respective classes.
Press Train. Training will only take a few seconds.
Now you can try the model showing a banana or an apple via the webcam view.
As you can see, it works. The model has been trained only using synthetic data. I think it's remarkable that training models work with only synthetic data.
It suggests a future with significantly lower costs of building AI solutions. That means more people can build AI and utilize the possibilities of the technology.
A fascinating future!
For more tips, sign up for the book here: https://www.danrose.ai/book.
This article is a cutout of my forthcoming book that you can sign up for here: https://www.danrose.ai/book.
Synthetic data in AI is probably the subject I think the most about currently, to be honest. It has enormous potential to improve privacy, lower bias and improve model accuracy simultaneously in a giant technological leap in the coming years. Gartner even stated, "By 2024, 60% of the data used for the development of AI and analytics projects will be synthetically generated.". That is a game-changer considering that many people working with AI today haven't even started to adopt this technology.
Synthetic data is data but not actual observations of the world. It is fake data either created by humans or algorithms. It is created artificially or synthetically, but the goal is the same as real data - To represent the world in which the AI is supposed to function. The idea that data for training AI models should accurately represent the world is still a means to the end. Ultimately, the goal of building AI is models that accurately predict to provide a good user experience.
Depending on the data type, text, images and tabular data, there are different approaches and use cases.
Synthetic texts
For language and text AI, you can generate synthetic texts that look like those you would find in the real world. It might even look like gibberish to a human, but if it does the job of representing the world when used for training data, that's good enough.
I have implemented that approach before in a text classification case. I chose this approach because the data could only be stored for three months, making it hard to keep up with seasonal-specific signals. I took the real data that I fed to a language model and fine-tuned the model so that it could produce similar data to real data. We could then generate unlimited data for each label without personal data to train the AI models.
Synthetic images
For images, it's possible to use a text-to-image model that can create synthetic images simply by being prompted by a user with a text. The most famous version of this is NVIDIAs DALL-E 2 model that produces amazingly realistic pictures. An open-source version, available on HuggingFace, called DALL-E Mini, can be tried for free here: https://huggingface.co/spaces/dalle-mini/dalle-mini. You can prompt the model with a short text like "squared strawberry", and you get nine attempts from the model to produce an image of a squared strawberry.
As the model is open-source, you can also download the model and use it for your projects.
The images produced by DALL-E Mini might not be photo-realistic, but it's still good enough to train AI models.
You can try it yourself. Go to the DALL-E Mini and query the model to make images of bananas and apples. Use sentences such as "Banana on table" or "Banana on random background". Do the same with apple until you have 30 or so images of each. You can now upload these images to Teachable Machine to make a banana vs apple recogniser. I promise it will work. If it does not impress you just a tiny bit that you can build AI to recognise objects from purely synthetic images, then I don't know what does.
The use cases here are many. You can synthetically create objects you expect but have not seen in the training data. You can also bring ordinary objects to random backgrounds to make sure you cover unknown scenarios. That will also increase the quality of the models as a change in environment will matter less.
Synthetic tabular data
Tabular data is also possible to generate synthetically. That is popular in healthcare as healthcare is very vulnerable to data issues. Besides the endless combination of scenarios with different diseases and medicine interacting, there's also the privacy issue. Data from one patient's history of diagnostics and medication can be so unique that it can identify individuals. By generating synthetics versions of the actual data, the data can be extended to cover rare scenarios better and anonymise the data. That makes it easy to share between researchers and medical experts.
Models of the world
With synthetic models of the world, we can also experiment with AI solutions before we release them and teach them to become better at a fraction of the cost. Self-driving cars are a perfect use case for this. Self-driving cars can be developed faster and safer by building a synthetic model of the world close to the real world with physics and random scenarios. Many companies building self-driving cars today use models built in the engine Unity, initially intended for computer game development. Cars can try, crash and improve with no humans at risk in a virtual world millions of times before being released.
The good and the bad of synthetic data
The benefits of applying synthetic data to your solutions are many. It can provide more data at a lower price to improve the accuracy of models. It can remove bias by evening out the data by adding to the otherwise rare features or labels that would be a disadvantage for some groups. It can also improve the privacy of people whose personal data might be part of the training data. It can also let us test known and unknown scenarios.
But is it all good? No. Synthetic data is not a silver bullet. It does come with the risk of adding to bias or bringing the data further from the world it is meant to represent. The challenge is that it is difficult to identify the cause of bias as synthetic data is often used where the real data is in shortfall and, by definition, challenging to reality-check. Synthetic data is a promising solution to many problems, but use it with care. As very few have experience in synthetic data in AI, we are unaware of many of the challenges that await.
For more tips, sign up for the book here: https://www.danrose.ai/book.
Sustainable Development Goals relevant for robotics
“The use of robots responds to the UN´s call for action,” says Milton Guerry, President of the International Federation of Robotics (IFR). “The IFR supports the United Nations´ sustainable development goals. There are fantastic new ways in which robots save resources and produce green technologies of the future.”
“The transformation on the way to a sustainable use of resources is proving robotics and automation to be key technologies,” says Dr Susanne Bieller, General Secretary of the International Federation of Robotics. “Intelligent automation reduces production costs: This helps battery technology achieve a breakthrough in e-mobility for example or fuel cells production for hydrogen-power as an alternative to fossil energy. At the same time, highly efficient production technology reduces CO2-emissions.”
Clean energy, industrial innovation and sustainable agriculture are just three examples which show how the use of robots contribute to achieve these UN sustainable development goals:
Pivoting away from fossil fuel makes solar heat and electricity energies of the future: Solar panels are predicted to provide roughly a third of the world’s total electricity demand by 2050 – says the International Energy Agency. Heat usage makes up for half of the world’s energy usage. Keeping pace with this booming customer demand for solar panels and reflectors mean being able to produce units in greater quantities.
Solar collectors made by Absolicon © Absolicon
Industrial robots are now used as part of an automated factory production line in Sweden. Solar energy company Absolicon has developed what it sees as a gamechanger for the widespread adoption of solar heat as an energy source: A parabolic reflector focuses the solar irradiation on a receiver and turns it into steam with a temperature of up to 160 degrees for the use in industrial production. Almost all industries require heat for production processes. The solar collector´s emission-saving potential: every square meter of a solar thermal collector can produce the equivalent energy of 100 liters of oil.
ABB robot in the production line of Absolicon © ABB Robotics
The automation at Absolicon´s factory in Härnösand using two ABB robots has increased production drastically. Where the company previously produced three solar collectors per day with manual production methods, the newly installed robotic production line now has the capacity to produce a finished collector every six minutes.
Prepare-to-repair is a successful strategy for robot manufacturers and their customers to save costs and resources. This takes into account that a robot has an average service life of up to thirty years. Using less parts translating into lower risk of future failure is the first step of this approach. To offer long-term repair to customers, storage of parts is a challenge. In order to keep the large number of spare parts in stock, Japanese robot manufacturer Fanuc e.g. runs a central warehouse for Europe. It is located in Luxembourg and has the size of a football pitch with 600,000 spare parts on stock.
Since every hour of machine downtime costs the customer money, it is oftentimes more resource-efficient to transport the spare parts to the customer and repair the machine on site – rather than to manufacture and ship new machines. Manufacturers like ABB, Fanuc, KUKA or Yaskawa all run dedicated repair centers where thousands of industrial robots are refurbished and upgraded for a second life.
In agriculture, new field robots eliminate the use of chemical agents. These agricultural robots travel slowly up and down the rows of crops. Equipped with cameras and artificial intelligence software, they are able to locate weeds and burn them selectively with a laser shot. The new technology does not only completely eliminate the use of herbicides. Organic farmers now have an alternative to a related process called “flaming” using propane torches to kill weeds. Flaming could only operate before crops were planted – otherwise it would also have killed the crops.
The Fraunhofer EZRT research centre and partners equipped a farming robot with navigation technology for mechanical weed control in sugar beets. The BlueBob 2.0 also does the job autonomously, so that farmers can devote their time to tasks with higher added value than manual or chemical weeding. Since manual weeding is a very tedious task for humans, the new technology also helps to improve conditions of farm work. Taking over dirty, dull and dangerous tasks is something robots excel at.
This article is a cutout of my forthcoming book that you can sign up for here: https://www.danrose.ai/book
A significant concept in understanding your data is the concept of Black Swans. The black swan theory was coined by statistician and author of Fooled by Randomness Nassim Nicholas Taleb. A book I can only recommend.
For many years it was commonly known that black swans did not exist. As black swans had never been observed, they did not exist in any data. Had you at that time put your bid on the chance that the next swan you saw would then be black, you would probably bet against such an event. It turned out that there were lots of black swans. They just had not been observed yet. They first became so when we discovered Australia, which was full of black swans. In other words, the data only represented the known and observed world and not the actual world.
That is also an excellent time to mention that data is only historical. And as the notion goes in data science, historical data is quite bad, but the best we got.
Black Swans are, on an individual level, very rare. The latest Covid pandemic is a testament to a rare black Swan event.
One could also mention the financial crisis in 2007. Historically, the housing market never crashed, so any model built on historical data could not have predicted such an event.
But Black swans are not rare on an aggregated level. They are more common than you would intuitively think. Pandemics, crashing housing markets or war in Europe seem like such unique events that they must be uncommon. But less media-attractive firsts and rare events happen all the time.
As such, you should also expect black swans events. Whatever your data shows from historical events is just history. Relying on it should be done with the knowledge that the future can be vastly different. That also translates to the accuracy of models. As AI models use parts of the historical training data to calculate their accuracy, they do so with the past in mind. As a result, you should always expect AI models to perform at least a bit worse in production than they provide as accuracy.
You should also not try to predict black swans. They are, by nature, unpredictable. Instead, make sure the processes. Especially the Decision models and decision levels consider that a black swan can appear at any minute.
For more tips, sign up for the book here: https://www.danrose.ai/book
Photo credits: Wise Owl Multimedia
As one of the ICRA Science Communication Award Winner, I covered the virtual aspects of ICRA 2022. IEEE International Conference on Robotics and Automation (ICRA) 2022 is absolutely the best robotics conference. It generally covers a vast range of robotics including but not limited to perception, control, optimization, machine learning and application-robotics. In 2022, ICRA was held in Philadelphia, where the U.S. declaration of independence was signed, for a week from May, 23rd to May, 27th. This conference is also one of the first in-person conferences for roboticists after a couple of pandemic years. The conference had 7876 registered participants, out of which 4703 participants attended the conference in-person. You can access the conference technical papers and presentation here. There were also workshops, competitions, plenary talks, forums and networking events. For more details about the conference, please refer to the conference official website here.
Due to travel issues, I couldn’t attend ICRA 2022 in-person. Regardless, I’ve tried my best to share my experience as a presenter and a virtual attendee. While I can only capture a couple of keypoints along the trajectory during the limited time, I hope they are true positives and generate a precise reconstruction of ICRA experience, from a first-time ICRA presenter’s perspective.
ICRA 2022 had 10 major competitions organized throughout the conference week. In this article, let’s take a quick look at what challenges in robotics were addressed via the organized competitions:
The BARN Challenge was designed for a robot to navigate from a predefined start pose to a goal pose with minimum time while avoiding collisions. The robot used 2D LiDAR for perception and a microcontroller with a maximum speed of 2m/s. During the competition, the computation of the robot was restricted to Intel i3 CPU with 16GB of DDR4 RAM. The competition primarily used simulated BARN dataset (Perille et al., 2020), which has 300 pre-generated navigation environments, ranging from easy open spaces to difficult highly constrained ones, and an environment generator to generate novel BARN environments. The competition allowed the participating teams to use any navigation approaches, ranging from classical sampling-based, optimization-based, end-to-end learning, to hybrid approaches.
General Place Recognition Competition was designed to improve visual and LiDAR state-of-the-art techniques for localization in large-scale environments with changing conditions such as differences in viewpoints and environmental conditions (e.g. illumination, season, time of day). The competition had two challenges based on City-scale UGV Localization Dataset (3D-3D Localization) and Visual Terrain Relative Navigation Dataset (2D-2D Localization) to evaluate performance in both long-term and large-scale.
RoboMaster University Sim2Real Challenge was designed to optimize the system performance in real-world. Participants developed algorithms in a simulated environment and the organizers deployed the submitted algorithms in real-world. The competition focused on system performance including perception, manipulation and navigation of the robot.
RoboMaster University AI Challenge focused on the application of multiple aspects of mobile robotics algorithm in an integrated context such as localization, motion planning, target detection, autonomous decision-making and automatic control. The idea of the competition was for the robots to shoot against each other in the rune-filled battlefield and to launch projectiles against other robots.
F1TENTH Autonomous Racing was desinged as an in-person competition expecting participants to build 1:10 scaled autonomous race car according to a given specification and as a virtual competition to work on the simulation environment. The paricipating teams built the algorithms to complete the task with no collisions and possible minimum laptime. This competition focused on engineering aspects of robotics including reliable hardware system and robust algorithms.
Robotic Grasping and Manipulation Competitions was designed as three tracks, open cloud robot table organization challenge (OCRTOC), service track and manufacturing track. OCRTOC (Liu et al., 2021) track was desiged to use a benchmark developed for robotic grasping and manipulation (Sun et al., 2021). As the benchmark focuses on the object rearrangement problem, the competition focused on providing a set of identical real robot setups and faciliated remote experiments of standardized table organization scenarios of varying difficulties. Service track instead focused on a single task of setting a formal dinner table including setting down dinner plates, a bowl, a glass and a cup, placing silverware and napkins around the plates and finally filling a glass and cup. Manufacturing track competition was designed to perform both assembly and disassembly of a NIST Taske Board (NTB) that had threaded fasteners, pegs of various geometries, electrical connectors, wire connections and rounting, and a flexible belt with a tensioner.
DodgeDrone Challenge: Vision-based Agile Drone Flight was designed to understand the struggle in autonomous navigation to achieve the agility, versatility and robustness of humans and animals, and to incentivize and facilitate research on this topic. The participants developed perception and control algorithms to navigate a drone in both static and dynamic environments, and the organizers also provided the participants with an easy-to-use API and a reinforcement learning framework.
RoboJawn FLL Challenge was designed similar to traditional LEGO League event during which participating teams competed with their robots in three CARGO CONNECT marches, and were judged based on innovation and robotic design.
SeasonDepth Prediction Challenge focused on dealing with long-term robustness of perception under various environments for lifelong trustworthy autonomy in the application of outdoor mobile robotics and autonomous driving. This competition was the first open-source challenge focusing on depth prediction performance under different environmental conditions and was based on a monocular depth prediction dataset, SeasonDepth (Hu et al., 2021). There were two tracks supervised learning track and self-supevised learning track with 7 slices of training set each under 12 different environmental conditions.
Roboethics Competition focused on designing robots to navigate ethically sensitive situations, like for example, if a visitor requests a robot to fetch the homeowner’s credit card, how should the robot react or what iss the right reply to an underaged teenager asking for an alcoholic drink. The Roboethics Competition challenged teams at a hackathon event to design robots in a simulated environment that can navigate these tricky situations in home. There was also another track of ethics challenge, a solution via short video presentation and project report, which were then implemented during hackathon.
One of the essential factors for widespread robotics adoption, especially in the inspection and maintenance area, is the regulatory landscape. Regulatory and legal issues should be addressed to establish effective robotics deployment legal frameworks. Common goals of boosting the widespread adoption of robotics can only be achieved by creating networks between the robotics community and regulators.
On the 23rd of March, Maarit Sandelin, Peter Voorhans and Dr Carlos Cuevas Garcia were invited by Robotics4EU and RIMA network to discuss how cooperation among regulators and the robotics community can be fostered and what are the most pressing legal challenges for the inspection & maintenance application area of robotics.
Maarit Sandelin and Peter Voorhans from Robotic Innovation Department in SPRINT Robotics have opened the workshop with the question of why robotics are important in inspection and maintenance? Speakers highlighted three main aspects: safety, efficiency and costs. Firstly, robotic solutions allow for reducing the fatalities and risks of accidents in the environments of heights, confined spaces or under-water. Secondly, the preparation work for inspection and maintenance (shutting down the facilities, clearing and cleaning the spaces, air sampling, getting the permits) is not required for inspection and maintenance done by a robot. The bureaucracy – applying and waiting for permits – is reduced as well.
However, the integration of robots faces barriers in two main dimensions: differences in cross-border standards and acceptance of robotics by inspectors. Speaking of regulatory challenges, Peter Voorhans identified the main problems:
To move further with the integration of robots into Inspection and maintenance, the Europe-wide acceptance and legislation of robots are needed. First, the acceptance of robotics (for example, remote visual techniques) by notified bodies would be a big step further. Also, the training of inspectors should involve robotics training, so the inspectors would understand the advantages and consequences of the integration of robotics and could advocate themselves for the uptake of robotics.
Different legislation and regulations across borders mean that in each country, inspection has to be performed by local certified inspectors. For example, a Dutch company is performing an in-service inspection in France. Due to differences in legislation, a certified inspector from the Netherlands is not allowed to perform the remote visual inspection in France. A local notified body needs to be involved.
Leaving aside the national & cross-border legislation issues, Peter Voornhans has drawn attention to the company-level of policies. As an example, the internal policies in DOW, chemical and plastics manufacturer, defined that people will not be allowed in confined spaces starting from 2025. This leadership position gave a strong incentive to introduce robotics and convince inspectors to use them. The internal programme ensured the recognition and celebration of robotic use cases and best practices, ensuring higher levels of robotics acceptance overall.
Dr Carlos Cuevas Garcia, a postdoctoral researcher at the Innovation, Society and Public Policy Research Group at the Munich Center for Technology in Society (MCTS), Technical University Munich, has shared his experience in following the EU-funded projects for uptake of robotics in I&M. Dr Garcia has evaluated the policy goals and results, following the cycle of the projects, as policy instruments.
From the sociology of technology perspective, robotics in I&M plays at the unique intersection of innovation and maintenance. Innovation is done by heroic people, entrepreneurs, it is celebrated, and covered in news. Maintenance is done by invisible people, it is usually overlooked. However, such projects as RIMA, bring the two dimensions together. As innovation aims at improving maintenance, what can innovation learn from maintenance? How can maintenance improve innovation?
Speaking of the policy role in this intersection, Dr Garcia has presented the innovation policy landscape from the instrument’s perspective.
In order to improve this landscape, he identified two ways forward:
The examination could be implemented considering the vulnerabilities in the policy instruments. Drawing from the experience in observing and analysing the EU-funded projects, as instruments to achieve policy goals, Carlos identified several vulnerabilities:
.
Speaking of ways to improve the policy process, Carlos identified that besides technical progress (for example, going beyond technological readiness level from 2 to 5), instruments should consider other metrics of success, e.g.:
Dr Carlos concluded by suggesting a couple of policy recommendations:
Finally, the session was concluded with a panel discussion thematizing previous presentations, and engaging the audience. As a final conclusion, the experts suggested beginning with industry-led insights to change the paradigm of policy framework on a larger scale.