Disjunctive AI scenarios: Individual or collective takeoff?

Posted on Jan 11, 2017 in English Posts | 4 comments

In this post, I examine Magnus Vinding’s argument against traditional “single AI fooms off” scenarios, as outlined in his book “Reflections on Intelligence”. While the argument itself is not novel – similar ones have been made before by Robin Hanson and J Storrs Hall, among others – I found Vinding’s case to be the most eloquently and compellingly put so far.

Vinding’s argument goes basically as follows: when we talk about intelligence, what we actually care about is the ability to achieve goals. For instance, Legg & Hutter collected 70 different definitions for intelligence, and concluded that a summary which captured the essential spirit of most of them was “Intelligence measures an agent’s ability to achieve goals in a wide range of environments”.

But once we substitute “intelligence” with “the ability to achieve goals”, we notice that we are actually talking about having tools, in several senses of the word:

• Cognitive tools: our brains develop to have specialized processes for performing various kinds of tasks, such as recognizing faces, recognizing emotions, processing language, etc. Humans have some cognitive tools that are unique to us (such as sophisticated language) while lacking some that other animals have (such as the sophisticated smell processing of a dog).
• Anatomical tools: not only do our brains carry out specific tasks, we also have an anatomy that supports it. For instance, our vocal cords allow us to produce a considerable variety of sounds to be used together with our language-processing capabilities. On the other hand, we also lack some other anatomical tools, such as the impressive noses of dogs. It is the combination of cognitive and anatomical tools that allows us to achieve a variety of different goals.
• Physical tools: tools in the most conventional sense of the word, we would not be capable of achieving much unless we had various physical devices that can be used for manipulating the world.
• Cultural tools: nobody would get very far if they had to derive all of their ideas from scratch. Rather, we acquire most of our language, ideas, and ways of thought that we use from the people around us.
• Societal tools: an individual’s ability to achieve things has grown enormously as our economy has grown increasingly specialized. No single person could build a laptop, or even a pencil, all by themselves. Yet we have at our disposal tools – computers, web browsers, Internet service providers, online stores, manufacturers, delivery companies – which allow us to almost effortlessly acquire laptops and pencils and then put them into use.

This paragraph from Vinding’s book summarizes much of his argument:

“Human intelligence” is often compared to “chimpanzee intelligence” in a manner that presents the former as being so much more awesome than, and different from, the latter. Yet this is not the case. If we look at individuals in isolation, a human is hardly that much more capable than a chimpanzee. They are both equally unable to read and write on their own, not to mention building computers or flying to the moon. And this is also true if we compare a tribe of, say, thirty humans with a tribe of thirty chimpanzees. Such two tribes rule the Earth about equally little. What really separates humans from chimpanzees, however, is that humans have a much greater capacity for accumulating information, especially through language. And it is this – more precisely, millions of individuals cooperating with this, in itself humble and almost useless, ability – that enables humans to accomplish the things we erroneously identify with individual abilities: communicating with language, doing mathematics, uncovering physical laws, building things, etc. It is essentially this you can do with a human that you cannot do with a chimpanzee: train them to contribute modestly to society. To become a well-connected neuron in the collective human brain. Without the knowledge and tools of previous generations, humans are largely indistinguishable from chimpanzees.

So what are the implications for AI risk?

One of Vinding’s arguments is that “intelligence” has gotten increasingly distributed. Whereas a hunter-gatherer might only have drawn upon the resources of their own tribe, a modern human will enhance their capabilities by tapping into a network of resources that literally spans the entire globe. Thus, it may be misguided to focus on the point when AIs achieve human-level intelligence, for a single individual’s intelligence alone isn’t sufficient for achieving much. Instead, if AIs were to wipe out humanity, they would need to first achieve the level of capability that human society has… but the easiest way of achieving that would be to collaborate with human society and use its resources peacefully, rather than cause damage to it.

A similar argument was previously put forward by J Storrs Hall in his paper Engineering Utopia, which uses a more economic argument. Hall notes that even when a single AI is doing self-improvement (such as by developing better cognitive science models to improve its software), the rest of the economy is also developing better such models. Thus it’s better for the AI to focus on improving at whatever thing it is best at, and keep trading with the rest of the economy to buy the things that the rest of the economy is better at improving.

However, Hall notes that there could still be a hard takeoff, once enough AIs were networked together: AIs that think faster than humans are likely to be able to communicate with each other, and share insights, much faster than they can communicate with humans. The size of the AI economy could grow quite quickly, with Hall suggesting a scenario that goes “from […] 30,000 human equivalents at the start, to approximately 5 billion human equivalents a decade later”.

Any individual AI, then, will be most effective as a cooperating element of a community (as is any individual human […]). AI communities, on the other hand, will have the potential to grow into powers rivalling or exceeding the capability of the human race in relatively short order. The actions of communities are effects of the set of ideas they hold, the result of an extremely rapid memetic evolution […]

Real-time human oversight of such AI communities is infeasible. Once a networked AI community was established, a “cultural revolution” could overtake it in minutes on a worldwide scale, even at today’s communication rates. The essence of our quest for a desirable future world, then, both for ourselves and for the AIs, lies in understanding the dynamics of memetic evolution and working out ways to curb its excesses.

Hall suggests that an community could rapidly grow to the point where they were exclusively communicating and trading with each other, humans being too slow to bother with. Suppose that you were a digital mind that thought a thousand times as fast as biological humans. If you wanted a task done, would you rather hire another digital mind to do it, taking what felt to you like an hour – or would you hire a biological human, and have to wait what felt like a month and a half? You’d probably go with your digital friend.

One obvious limitation is that this speed advantage would only apply for purely mental tasks. If you needed something manufactured, you might as well order something from the humans.

Vinding’s book could also be read as a general argument suggesting that the amount of distributed intelligence in human society was so large that AIs would still benefit from trade, and would need a large amount of time to learn to do everything themselves. Vinding writes:

… the majority of what humans do in the economy is not written down anywhere and thus not easily copyable. Customs and know-how run the world to an extent that is hard to appreciate – tacit knowledge and routines concerning everything from how to turn the right knobs and handles on an oil rig to how to read the faces of other humans, none of which is written down anywhere. For even on subjects where a lot is written down – such as how to read faces – there are many more things that are not. In much of what we do, we only know how we do, not exactly “what”, and this knowledge is found in the nooks and crannies of our brains and muscles, and in our collective organization as a whole. Most of this unique knowledge cannot possibly be deduced from a few simple principles – it can only be learned through repeated trial and error – which means that any system that wants to expand the economy must work with this enormous set of undocumented, not readily replaceable know-how and customs.

This is a compelling argument, but with recent progress in AI, it feels less compelling than it might have felt a few years back. Vinding mentions reading faces as an example of a domain involving much tacit knowledge, but computers are already outperforming humans at facial recognition and are starting to match humans at recognizing and interpreting emotional expressions, as well as in recognizing rare syndromes from facial patterns. As a more industrial example, DeepMind’s AI technology was recently deployed to optimize power usage at Google’s data centers, for a 15 percent improvement in power usage efficiency. Since relatively small reductions in power use translate to large savings – this change is estimated to save Google hundreds of millions of dollars – these were already highly-optimized centers.

Tacit knowledge is essentially knowledge that is based on pattern recognition, and pattern recognition is rapidly becoming one of AI’s strengths. Currently this still requires massive datasets – Goodfellow et al. (2016, chap 1) note that as a rule of thumb, a deep learning algorithm requires a dataset of at least 10 million labeled examples in order to achieve human-level or better performance. On the other hand, they also note that a large part of the success of deep learning has been because the digitization of society has made such large datasets increasingly available.

It seems likely that as the development of better and better AI pattern recognition will drive further investment into collecting larger datasets, which will in turn make it even more profitable to continue investing in better pattern recognition. After DeepMind’s success with improving power efficiency at Google’s data centers, DeepMind’s Demis Hassabis told Bloomberg that “[DeepMind] knows where its AI system lacks information, so it may ask Google to put additional sensors into its data centers to let its software eke out even more efficiency”.

If AI allows efficiency to be increased, then businesses will be rebuilt in such a way as to give AI all the necessary information it needs to run them maximally efficiently – making tacit human knowledge of how things were previously done both unnecessary and obsolete. The items in Amazon’s warehouses are algorithmically organized according to a logic that makes little intuitive sense to humans, with an AI system telling the workers where to go; Foxconn is in the process of fully automating its factories; Uber is seeking to replace human drivers with self-driving cars. We are bound to see this kind of automation penetrate into ever larger parts of the economy over time, which will drive the further deployment of sensors and collection of better datasets in order to enable it. By the time AGI manifests, after several decades of this development, there’s no obvious reason to assume that very much of the tacit knowledge needed for running an economy would necessarily remain locked up in human heads anymore.

To sum things up, this suggests that beyond the classical “one AI fooms to a superintelligence and takes over the world” scenario, there may plausibly exist a scenario where the superintelligences are initially best off trading with humans. As time goes on and the size of the AI community grows, this community may collectively foom off as they come to only trade with each other and have little use for humans. Depending on how long it takes for the community grow, this may or may not look any different from traditional foom.

This blog post was written as part of research funded by the Foundational Research Institute.