A new report tries to bring order to the messy business of measuring AI progress
How clever is artificial intelligence, really? And how fast is it progressing? These are questions that keep politicians, economists, and AI researchers up at night. And answering them is crucial — not just to improve public understanding, but to help societies and governments figure out how to react to this technology in coming years.
A new report from experts at MIT, Stanford University, OpenAI, and other institutions seeks to bring some clarity to the debate — clarity, and a ton of graphs. The AI Index, as it’s called, was published this week, and begins by telling readers we’re essentially “flying blind” in our estimations of AI’s capacity. It goes on to make two main points: first, that the field of AI is more active than ever before, with minds and money pouring in at an incredible rate; and second, that although AI has overtaken humanity when it comes to performing a few very specific tasks, it’s still extremely limited in terms of general intelligence.
As Raymond Perrault, a researcher at SRI International who helped compile the report, told The New York Times: “The public thinks we know how to do far more than we do now.”
To come to these conclusions, the AI Index looked at a number of measures of progress, including “volume of activity” and “technical performance.” The former stat examines how much everything happens in the field, from conference attendance to class enrollment, to VC investment and startups started. The short answer here is that everything’s happening a lot. In graph terms, it’s all “up and to the right.”
The other factor, “technical performance,” attempts to measure AI’s capabilities to outcompete humans at specific tasks, like recognizing objects in images and decoding speech. Here, the picture is more nuanced.
There are definitely tasks where AI has already matched or eclipsed human performance. These include identifying common objects in images (on a test database, ImageNet, humans get a 5 percent error rate; machines, 3 percent), and transcribing speech (as of 2017, a number of AI systems can transcribe audio with the same word error rate as a human). A number of games have also been definitively conquered, including Jeopardy, Atari titles like Pac-Man, and, most famously, Go.
But as the report says, these metrics only give us a partial view of machine intelligence. For example, the clear-cut world of video games is not only easier to train AI in, because well-defined scoring systems help scientists assess and compare different approaches. It also limits what we can ask of these agents. In the games AI has “solved,” the computer can always see everything that’s happening — a quality known to scientists as “perfect information.” The same can’t be said of other tasks we might set AI on, like managing a city’s transport infrastructure. (Although researchers have begun to tackle video games that reflect these challenges, like Dota.)
Caveats of a similar nature are needed for tasks like audio transcription. AI may be just as accurate as humans when it comes to writing down recorded dialogue, but it can’t gauge sarcasm, identify jokes, or account for a million other pieces of cultural context that are crucial to understanding even the most casual conversation. The AI Index acknowledges this, and adds that a bigger problem here is that we don’t even have a good way to measure this sort of commonsense understanding. There’s no IQ test for computers, despite what some PR people claim.
Does that mean we need to worry less about AI’s effects on society? Unfortunately not. Even though our most advanced AI systems are dumber than a rat (so says Facebook’s head of AI, Yann LeCun), it won’t stop them from having a huge effect on our lives — especially in the world of work.
Earlier this week, a study published by consultancy firm McKinsey suggested that as many as 800 million jobs around the world could be under threat from automation in the next 12 years. But, the study’s authors clarify that only 6 percent of the most rote and repetitive jobs are at danger of being automated entirely. For the rest, only parts of the job can be done by machines. This is where the narrow intelligence of AI will really have an impact, and here, it’s tricky to say what the effect will be.
If a computer can do one-third of your job, what happens next? Do you get trained to take on new tasks, or does your boss fire you, or some of your colleagues? What if you just get a pay cut instead? Do you have the money to retrain, or will you be forced to take the hit in living standards?
It’s easy to see that finding answers to these questions is incredibly challenging. And it mirrors the difficulties we have understanding other complex threats from artificial intelligence. For example, while we don’t need to worry about super-intelligent AI running amok any time soon, we do need to think about how machine learning algorithms used today in healthcare, education, and criminal justice, are making biased judgements. The conclusion of both the AI Index and McKinsey’s study is that these questions, and others, need deep consideration in order to stay ahead of what’s coming. As machines get clever, we can’t afford to be dumb.