Collecting data is for many AI projects without a doubt the most expensive part of the project. Labelling data like images and text pieces is hard and tedious work without much possibility of scaling. If an AI project requires continuously updated or fresh data then this can be a high cost that can challenge the whole business case of an otherwise great project.
There are a few strategies though to lower the costs of labelling data. I have previously written about Active Learning; a data collection strategy that focuses on prioritizing the labelling of the most crucial data first given the models weakest confidence. This is a great strategy but in most cases you still need to label a lot of data.
To speed up the labelling process the strategy of model-assisted labelling has come up. The idea is simply that you train an AI in parallel with labelling and as the AI starts to see a pattern in the data, the AI will suggest labels to the labeller. In that way the labeller in many cases can simply approve the pre suggested label.
Model-assisted labelling can be done both by training a model solely for the purpose of labelling but can also be done by putting the actual production model in the labelling loop and letting that suggest labels.
But is modelassisted labelling just a sure way to get data labelled quicker? Or are there downsides to the strategy? I have worked intensively with model-assisted labelling and I know for sure that there are both pros and cons and if you’re not careful you can end up doing more harm than good with this strategy. If you manage it correctly it can work wonders and save you a ton of resources.
So let’s have a look at the pros and cons.
The first and foremost advantage is that it’s faster for the person working with labelling to work with pre-labelled data. Approving the label with a single click for most cases and only having to manually select a label once in a while is just way faster. Especially when working with large documents or models with many potential labels the speed can increase significantly.
Another really useful benefit with model-assisted labelling is that you very early on get an idea about the models weak points. You will get a hands-on understanding of what instances are difficult for the model to understand and usually mislabels. This reflects on the results you should expect in production and as a result youtube the chance early to improve or work around these weak points. When seeing weak points in the model that also often suggests a lack of data volume or quality in these areas. So it also provides an insight to what kind of data you should go look for to be labelled more of.
Now for the cons. As I mentioned the cons can be pretty bad. The biggest issue with model-assisted labelling is that you are running the risk of lowering the quality of your data. So even though you get more data labelled faster with less quality you can end up with a model performing worse than it would had you not used model-assisted labelling.
So how can model-assisted labelling lower the data quality? It’s actually very simple. Humans tend to prefer defaults. The second you slip into autopilot you will start making mistakes by being more likely to choose the default or suggested label. I have seen this time and time again. The biggest source of mistakes in labelling tend to be accepting wrong suggestions. So you have to be very careful when suggesting labels.
Another downside can be if the pre-labelling quality is simply so low that it takes the labeller more time to correct than it would have to start with a blank answer. So you will have to be careful to not enable the pre-labelling too early.
I have a few tips for being more successful with model-assisted labelling.
First tip is to set a target for data quality. You will never get 100% correct data anyway so you will have to accept some number of wrong labels. If you can set a target that is acceptable to train the model from, you can monitor if the model-assisted labelling is begging to do more harm than good. That also works great as an expectations alignment on your team in general.
I’d also suggest doing samples without pre-labelling to measure if there’s a difference between the results you get with and without pre-labelling. You simply do this by turning off the assist model for an example one out of every ten cases. It’s easy and will show a lot of truth.
Lastly I will suggest one of my favorites. Probabilistic programming models are very beneficial for model-assisted labelling. Probabilistic models are Bayesian and as a result offer uncertainty in distributions instead of scalars(a number) and make it much easier to know if the pre-label is likely to be correct or not.
When I write about AI I very often refer to data operations and how important a foundation it is for most AI solutions. Without proper data operations you can easily get to a point where handling the necessary data will be too difficult and costly for the AI business case to make sense. So to clarify a little I wanted to give you some insight on what it really means.
Data operations is the process of obtaining, cleaning, storing and delivering data in a secure and cost effective manner. It’s a mix of business strategy, DevOps and data science and is the underlying supply chain for many big data and AI solutions.
Data operations was originally coined in Big Data regi but has become a more broadly used term in the later years.
As I have mentioned in a lot of previous posts, I see the data operations as a higher priority than algorithm development when it comes to trying to beat the competition. In most AI cases the algorithms used are standard AI algorithms from standard frameworks that are fed data, trained and tuned a little before being deployed. So since the underlying algorithms are largely the same the real difference is in the data. The work that goes into to get good results from high quality data is almost nothing compared to the amount of work it takes when using mediocre data. Getting data at a lower cost than the competition is also a really important factor. Especially in AI cases that require a continuous flow of new data. In these cases getting new data all the time can become an economic burden that will weigh down the business.
To make it more concrete I wanted to use the AI company I co-founded Paperflow as an example. Paperflow is an AI company that receives invoices and other financial documents and captures data such as invoice date, amounts and invoice lines. Since invoices can look very different and the layout of invoices changes over time, getting a lot and getting more data all the time is necessary. So to make Paperflow a good business we needed good data operations.
To be honest we weren't that aware of the importance when we made these initial decisions but luckily we got it right. Our first major decision in the data operations was that we wanted to collect all data in-house and make our own system for collecting data. That’s a costly investment with both a high investment into the initial system development but also a high recurring cost to our employees with the job of entering data from invoices into the system. The competition had chosen another strategy. They instead had the customers enter the invoice data to their system when their AI failed to make the right prediction on the captured data. That’s a much cheaper strategy that can provide you with a lot of data. The only problem is that customers only have one thing in mind and that is to solve their own problems disregarding if it is correct or not in terms of what you need for training data.
So in Paperflow we found a way to get better data. But how do you get the costs down then?
A part of the solution was heavily investing in the system that was used for entering data and trying to make it as fast to use as possible. It was really trial and error and it took a lot of work. Without having the actual numbers I guess we invested more in the actual data operating systems than the AI.
Another part of the solution was to make sure we only collected the data we actually needed. This is a common challenge in data operations since it’s very difficult to know what data you are going to need in the future. Our solution was to first go for collecting a lot of data (and too much) and then slowly narrowing down the amount of data collected. Going the other way around can be difficult. If we had suddenly started to collect more data on each invoice we would basically have needed to start over and discard all previously validated invoices.
We also started to work a lot on understanding a very important metric. When were our AI guesses so correct that we trust it and avoid to validate a part of the data. That was achieved with a variety of different tricks and technologies one of them being probabilistic programming. Probabilistic programming has the advantage of delivering a uncertainty distribution instead of a percentage that most machine learning algorithms will do. By knowing how sure you are that you are such significantly lowers the risks of making mistakes.
The strategy of only collecting data that you need the most by choosing cases where you AI is the most uncertain is also known as active learning. If you are working on your data operations for AI, you should definitely look into that.
On the more tech-heavy part of storing data in an effective way you will also see challenges. I’m not a DevOps expert but I have seen the problem of suddenly having too much data that grows faster than expected in real life. That can be crucial since the scaling ability quickly is coming under pressure. If I could provide one advice here it would be to involve a DevOps early on in the architecture work. Building on a scalable foundation is much more fun than trying to find short term solutions all the time.
The debate about man vs robots is an evergreen and common thing now. While robots are viewed as an enabler of a dystopian future brought by digital disruption, the main question that has baffled minds is how smart are they. When it comes to human intelligence, there isn’t any other living being or ‘mechanical or AI mind’ that can draw parallel with us. Yet, robots powered by AI have been able to perform trivial, monotonous tasks with accuracy far better than us. It is important to note that this does not imply robots have acquired cognitive intelligence nor common sense which are intrinsic to humans, despite de facto of the recent marvels of robotics.
The main problem is that most of the algorithms that are written for robots are based on machine learning coding. These codes are collected from a particular type of data, and models are trained based on individual test conditions. Hence, when put in a situation that is not in their code nor algorithm, robots can fail terribly or draw a conclusion that can be catastrophic. This has highlighted in Stanley Kubrick’s landmark film 2001: A Space Odyssey. The movie features a supercomputer, HAL-9000, who is informed by its creators of the purpose of the mission: to reach Jupiter and search for signs of extra-terrestrial intelligence. When HAL makes an error, it refuses to admit this and alleges that it was caused due to human error. Therefore, astronauts decide to shut HAL down, but unfortunately, the AI discovers their plot by lip-reading. Conclusively, HAL arrives at a new conclusion that wasn’t part of its original programming, deciding to save itself by systematically killing off the people onboard.
Another illustration which experts often mention it that, while we can teach a robot on how to open a door by training it and feeding data on 500 different types of door, the robots will still fail when asked to open the 501st door. Also, this example is the best way to explain why robots don’t share the typical thought process and intelligence of humans. Humans don’t need to be ‘trained’ they observe and learn, or they experiment thanks to curiosity. Further, every time someone knocks the door, we don’t tend to open it, there is always an unfriendly neighbor we dislike. Again we don’t need to be reminded to lock the door either, but robots need a clear set of instruction. Let us consider other aspects of our life, robots and AI are trained on a particular set of data; hence they will function effectively when the input is something they have been trained or programmed for, beyond it the observation is different. For instance, if one uses the expression “Hit the road” while driving a car, she means to say to herself or the driver to begin the journey emphatically. If a robot does not know the phrasal meaning of the same expression, it may believe that the person is asking to ‘hit’ the road. This misunderstanding can lead to accidents. While researchers are working hard, devising algorithms, running codes, we are yet to see a robot that understands the way humans converse, all with accents, dialects, colloquy and jargons.
Michio Kaku, a futurist and theoretical physicist, once said that “Our robots today, have the collective intelligence and wisdom of a cockroach.” While robots of today can make salads on our command, or robots like Deep Blue or AlphaGo Zero can defeat humans in chess, it does not necessarily qualify as ‘common sense’ nor smartness. And let us not forget that Deep Blue and AlphaGo Zero were following instructions given by a team of ‘smart’ human scientists. These robots were designed by people who were smart enough to solve a seemingly impossible task. So to sum up, while robots are becoming smarter that, they are now able to fold laundry, impersonate as a person looking for dating online, they still lag when it comes to cognitive intelligence and common sense. It is a long wait till we find a robot we see in sci-fi movies, i.e. C3P0, R2D2 or WALL-E.
The journal Perception published new insights into the cognitive mechanisms underlying this phenomenon made by psychologists at Emory University.
Since the uncanny valley was first described, a common hypothesis developed to explain it. Known as the mind-perception theory, it proposes that when people see a robot with human-like features, they automatically add a mind to it. A growing sense that a machine appears to have a mind leads to the creepy feeling, according to this theory.
"We found that the opposite is true," says Wang Shensheng, first author of the new study, who did the work as a graduate student at Emory and recently received his Ph.D. in psychology. "It's not the first step of attributing a mind to an android but the next step of 'dehumanizing' it by subtracting the idea of it having a mind that leads to the uncanny valley. Instead of just a one-shot process, it's a dynamic one."
The findings have implications for both the design of robots and for understanding how we perceive one another as humans.
"Robots are increasingly entering the social domain for everything from education to healthcare," Wang says. "How we perceive them and relate to them is important both from the standpoint of engineers and psychologists."
"At the core of this research is the question of what we perceive when we look at a face," adds Philippe Rochat, Emory professor of psychology and senior author of the study. "It's probably one of the most important questions in psychology. The ability to perceive the minds of others is the foundation of human relationships. "
The research may help in unraveling the mechanisms involved in mind-blindness—the inability to distinguish between humans and machines—such as in cases of extreme autism or some psychotic disorders, Rochat says.
Co-authors of the study include Yuk Fai Cheong and Daniel Dilks, both associate professors of psychology at Emory.
Anthropomorphizing, or projecting human qualities onto objects, is common. "We often see faces in a cloud for instance," Wang says. "We also sometimes anthropomorphize machines that we're trying to understand, like our cars or a computer."
Naming one's car or imagining that a cloud is an animated being, however, is not normally associated with an uncanny feeling, Wang notes. That led him to hypothesize that something other than just anthropomorphizing may occur when viewing an android.
To tease apart the potential roles of mind-perception and dehumanization in the uncanny valley phenomenon the researchers conducted experiments focused on the temporal dynamics of the process. Participants were shown three types of images—human faces, mechanical-looking robot faces and android faces that closely resembled humans—and asked to rate each for perceived animacy or "aliveness." The exposure times of the images were systematically manipulated, within milliseconds, as the participants rated their animacy.
The results showed that perceived animacy decreased significantly as a function of exposure time for android faces but not for mechanical-looking robot or human faces. And in android faces, the perceived animacy drops at between 100 and 500 milliseconds of viewing time. That timing is consistent with previous research showing that people begin to distinguish between human and artificial faces around 400 milliseconds after stimulus onset.
A second set of experiments manipulated both the exposure time and the amount of detail in the images, ranging from a minimal sketch of the features to a fully blurred image. The results showed that removing details from the images of the android faces decreased the perceived animacy along with the perceived uncanniness.
"The whole process is complicated but it happens within the blink of an eye," Wang says. "Our results suggest that at first sight we anthropomorphize an android, but within milliseconds we detect deviations and dehumanize it. And that drop in perceived animacy likely contributes to the uncanny feeling."
I hear a lot about how the 5G mobile network technology will change the world and especially be a big enabler for applied AI. But is that really true? Will 5G be an AI gamechanger?
The short answer is no. 5G is nice. It's a convenient technology that will be nice for AI but in no way a big deal. The best comparison I can make is that 5G is like new roads. Asking people behind AI solutions if 5G is a gamechanger is like asking a shopkeeper the same about new roads. And that being in a location with already decent roads. The shopkeeper would probably answer something along the lines of: "It'll be nice. It'll be a little easier for my suppliers to bring goods and a little easier for my customers to reach me. And that might be especially for shops in the city, where the new roads are prioritized due to being more used. But it won't be something that can immediately be seen on my stores profits.".
So what does that mean? It means that better infrastructure is always nice, but if it's already decent like 4G, it’s not a huge difference. 5G is an incremental improvement to the infrastructure surrounding AI. As the infrastructure is just one of many parameters that affects AI solutions and grocery stores alike, it’s nice but no biggie.
So why is it just an incremental improvement? Isn't 5G ground breaking technology ?
Well no. 5G is like 4G radio waves transmitting data. 5G radio waves are just different from 4G waves by being shorter wavelength that can carry way more data as a result. And shorter wavelengths are not groundbreaking. We have actually been able to make short waves for a very long time.
So why haven't we done this before? Well there’s a lot of practical problems with shorter wavelengths. They reach a shorter distance and they have trouble going through walls and other material. To counter that problem you have to put more towers closer together. More towers means more costs associated with setting up and maintaining the towers. Furthermore having towers closer together brings another problem. Radio waves interfering and disturbing each other and as a result making the data transfer unstable.
Finally 5G is here. And that's for two reasons. I might sound like I’ve been putting 5G down as an unimpressive technology that deserves no credit. But 5G is impressive. Just not on the immediate effect on society. 5G is impressive like building a big bridge. When you build a big bridge you have to go through way more trouble figuring out how massive concrete can hold up in the conditions it’s given. That’s impressive engineering compared to a small bridge.
And that is one of the reasons we are getting 5G now. Some really impressive engineering have gone into getting around the problems with shorter wavelengths. Making the short waves reach enough ground to make 5G practically viable and making it not interfere too much is really impressive.
The other reason we get 5G now is simply that we are so many people using so much mobile network that we can share the costs of all those mobile network towers and make a good business case.
That’s it plain and simply. 5G is nice and an improvement but it’s no crazy invention that will change the world tomorrow.
Sometimes a very impressive algorithmic achievement is done and it should be celebrated. GPT-3 is a great example. GPT-3 is amazing engineering and data science and very well deserved it gets a lot of media attention. But for every GPT-3 there are hundreds of thousands of AI solutions, that are based on standard algorithms and not necessarily a genius achievement but a school book approach.
It might sound like I’m having a go at many AI solutions out there but in fact it’s the other way around. Going for a groundbreaking genius solution is for the vast majority of AI-cases not the right way to go. The standard algorithms can in most cases easily be sufficient for the task at hand and everything beyond that effort is usually bad business.
Given all this I still hear a lot about the “unique”, “genius” or “special” algorithm that some company has developed to solve a problem. I often hear terms like this in the media and the fact that this is so popular makes a lot of sense. When you have a business that you want to market and sell your product at a high price. It also helps to scare competition away when they think that the entry barrier to make a certain product or solution is very high. But that is what the genius algorithm is 99 out of a 100 times. A marketing message.
In reality much of the AI out there is standard algorithms such as CNN’s, Random Forest or even logistics regression that some would claim isn’t even AI. These algorithms can be used by most novice developers by using freely available frameworks such as Tensorflow or Scikit learn.
My primary reason to write this post is the same as for a lot of our posts I’m writing. I want to demystify AI and by killing the narrative about the genius algorithm I hope more people will have the chance to utilize AI.
So when you hear these claims, be critical and don’t let it be the reason not to get started with AI.
I’m not usually one to call “the fake media”, but in this case of AI I fell that the media has not lived up to it’s responsibility and a a naive way followed the hype and corporate press releases without taking the critical look that is in many ways what separates the news outlets from other information sources.
I often wonder how the danish(Where I live) news stations can have an Egypt correspondent but not an AI or even deep tech correspondent. The events in Egypt might not be as important to the everyday life in my and many other countries than AI is starting to have.
I really hope the media will improve here and not keep AI in a mystified aura.
I’m pretty sure the future for AI-algorithms are a given. The big tech companies like OpenAI, Deepmind, Google, Facebook and Apple will be the ones to develop the genius algorithms and very often release them into the wild for everyone to use. It’s already happening and we will only see more of this. So claiming to have a genius algorithm is just not very likely a true claim in the future.