Disjunctive AI scenarios: Individual or collective takeoff?
In this post, I examine Magnus Vinding’s argument against traditional “single AI fooms off” scenarios, as outlined in his book “Reflections on Intelligence”. While the argument itself is not novel – similar ones have been made before by Robin Hanson and J Storrs Hall, among others – I found Vinding’s case to be the most eloquently and compellingly put so far.
Vinding’s argument goes basically as follows: when we talk about intelligence, what we actually care about is the ability to achieve goals. For instance, Legg & Hutter collected 70 different definitions for intelligence, and concluded that a summary which captured the essential spirit of most of them was “Intelligence measures an agent’s ability to achieve goals in a wide range of environments”.
But once we substitute “intelligence” with “the ability to achieve goals”, we notice that we are actually talking about having tools, in several senses of the word:
- Cognitive tools: our brains develop to have specialized processes for performing various kinds of tasks, such as recognizing faces, recognizing emotions, processing language, etc. Humans have some cognitive tools that are unique to us (such as sophisticated language) while lacking some that other animals have (such as the sophisticated smell processing of a dog).
- Anatomical tools: not only do our brains carry out specific tasks, we also have an anatomy that supports it. For instance, our vocal cords allow us to produce a considerable variety of sounds to be used together with our language-processing capabilities. On the other hand, we also lack some other anatomical tools, such as the impressive noses of dogs. It is the combination of cognitive and anatomical tools that allows us to achieve a variety of different goals.
- Physical tools: tools in the most conventional sense of the word, we would not be capable of achieving much unless we had various physical devices that can be used for manipulating the world.
- Cultural tools: nobody would get very far if they had to derive all of their ideas from scratch. Rather, we acquire most of our language, ideas, and ways of thought that we use from the people around us.
- Societal tools: an individual’s ability to achieve things has grown enormously as our economy has grown increasingly specialized. No single person could build a laptop, or even a pencil, all by themselves. Yet we have at our disposal tools – computers, web browsers, Internet service providers, online stores, manufacturers, delivery companies – which allow us to almost effortlessly acquire laptops and pencils and then put them into use.
This paragraph from Vinding’s book summarizes much of his argument:
“Human intelligence” is often compared to “chimpanzee intelligence” in a manner that presents the former as being so much more awesome than, and different from, the latter. Yet this is not the case. If we look at individuals in isolation, a human is hardly that much more capable than a chimpanzee. They are both equally unable to read and write on their own, not to mention building computers or flying to the moon. And this is also true if we compare a tribe of, say, thirty humans with a tribe of thirty chimpanzees. Such two tribes rule the Earth about equally little. What really separates humans from chimpanzees, however, is that humans have a much greater capacity for accumulating information, especially through language. And it is this – more precisely, millions of individuals cooperating with this, in itself humble and almost useless, ability – that enables humans to accomplish the things we erroneously identify with individual abilities: communicating with language, doing mathematics, uncovering physical laws, building things, etc. It is essentially this you can do with a human that you cannot do with a chimpanzee: train them to contribute modestly to society. To become a well-connected neuron in the collective human brain. Without the knowledge and tools of previous generations, humans are largely indistinguishable from chimpanzees.
So what are the implications for AI risk?
One of Vinding’s arguments is that “intelligence” has gotten increasingly distributed. Whereas a hunter-gatherer might only have drawn upon the resources of their own tribe, a modern human will enhance their capabilities by tapping into a network of resources that literally spans the entire globe. Thus, it may be misguided to focus on the point when AIs achieve human-level intelligence, for a single individual’s intelligence alone isn’t sufficient for achieving much. Instead, if AIs were to wipe out humanity, they would need to first achieve the level of capability that human society has… but the easiest way of achieving that would be to collaborate with human society and use its resources peacefully, rather than cause damage to it.
A similar argument was previously put forward by J Storrs Hall in his paper Engineering Utopia, which uses a more economic argument. Hall notes that even when a single AI is doing self-improvement (such as by developing better cognitive science models to improve its software), the rest of the economy is also developing better such models. Thus it’s better for the AI to focus on improving at whatever thing it is best at, and keep trading with the rest of the economy to buy the things that the rest of the economy is better at improving.
However, Hall notes that there could still be a hard takeoff, once enough AIs were networked together: AIs that think faster than humans are likely to be able to communicate with each other, and share insights, much faster than they can communicate with humans. The size of the AI economy could grow quite quickly, with Hall suggesting a scenario that goes “from […] 30,000 human equivalents at the start, to approximately 5 billion human equivalents a decade later”.
Any individual AI, then, will be most effective as a cooperating element of a community (as is any individual human […]). AI communities, on the other hand, will have the potential to grow into powers rivalling or exceeding the capability of the human race in relatively short order. The actions of communities are effects of the set of ideas they hold, the result of an extremely rapid memetic evolution […]
Real-time human oversight of such AI communities is infeasible. Once a networked AI community was established, a “cultural revolution” could overtake it in minutes on a worldwide scale, even at today’s communication rates. The essence of our quest for a desirable future world, then, both for ourselves and for the AIs, lies in understanding the dynamics of memetic evolution and working out ways to curb its excesses.
Hall suggests that an community could rapidly grow to the point where they were exclusively communicating and trading with each other, humans being too slow to bother with. Suppose that you were a digital mind that thought a thousand times as fast as biological humans. If you wanted a task done, would you rather hire another digital mind to do it, taking what felt to you like an hour – or would you hire a biological human, and have to wait what felt like a month and a half? You’d probably go with your digital friend.
One obvious limitation is that this speed advantage would only apply for purely mental tasks. If you needed something manufactured, you might as well order something from the humans.
Vinding’s book could also be read as a general argument suggesting that the amount of distributed intelligence in human society was so large that AIs would still benefit from trade, and would need a large amount of time to learn to do everything themselves. Vinding writes:
… the majority of what humans do in the economy is not written down anywhere and thus not easily copyable. Customs and know-how run the world to an extent that is hard to appreciate – tacit knowledge and routines concerning everything from how to turn the right knobs and handles on an oil rig to how to read the faces of other humans, none of which is written down anywhere. For even on subjects where a lot is written down – such as how to read faces – there are many more things that are not. In much of what we do, we only know how we do, not exactly “what”, and this knowledge is found in the nooks and crannies of our brains and muscles, and in our collective organization as a whole. Most of this unique knowledge cannot possibly be deduced from a few simple principles – it can only be learned through repeated trial and error – which means that any system that wants to expand the economy must work with this enormous set of undocumented, not readily replaceable know-how and customs.
This is a compelling argument, but with recent progress in AI, it feels less compelling than it might have felt a few years back. Vinding mentions reading faces as an example of a domain involving much tacit knowledge, but computers are already outperforming humans at facial recognition and are starting to match humans at recognizing and interpreting emotional expressions, as well as in recognizing rare syndromes from facial patterns. As a more industrial example, DeepMind’s AI technology was recently deployed to optimize power usage at Google’s data centers, for a 15 percent improvement in power usage efficiency. Since relatively small reductions in power use translate to large savings – this change is estimated to save Google hundreds of millions of dollars – these were already highly-optimized centers.
Tacit knowledge is essentially knowledge that is based on pattern recognition, and pattern recognition is rapidly becoming one of AI’s strengths. Currently this still requires massive datasets – Goodfellow et al. (2016, chap 1) note that as a rule of thumb, a deep learning algorithm requires a dataset of at least 10 million labeled examples in order to achieve human-level or better performance. On the other hand, they also note that a large part of the success of deep learning has been because the digitization of society has made such large datasets increasingly available.
It seems likely that as the development of better and better AI pattern recognition will drive further investment into collecting larger datasets, which will in turn make it even more profitable to continue investing in better pattern recognition. After DeepMind’s success with improving power efficiency at Google’s data centers, DeepMind’s Demis Hassabis told Bloomberg that “[DeepMind] knows where its AI system lacks information, so it may ask Google to put additional sensors into its data centers to let its software eke out even more efficiency”.
If AI allows efficiency to be increased, then businesses will be rebuilt in such a way as to give AI all the necessary information it needs to run them maximally efficiently – making tacit human knowledge of how things were previously done both unnecessary and obsolete. The items in Amazon’s warehouses are algorithmically organized according to a logic that makes little intuitive sense to humans, with an AI system telling the workers where to go; Foxconn is in the process of fully automating its factories; Uber is seeking to replace human drivers with self-driving cars. We are bound to see this kind of automation penetrate into ever larger parts of the economy over time, which will drive the further deployment of sensors and collection of better datasets in order to enable it. By the time AGI manifests, after several decades of this development, there’s no obvious reason to assume that very much of the tacit knowledge needed for running an economy would necessarily remain locked up in human heads anymore.
To sum things up, this suggests that beyond the classical “one AI fooms to a superintelligence and takes over the world” scenario, there may plausibly exist a scenario where the superintelligences are initially best off trading with humans. As time goes on and the size of the AI community grows, this community may collectively foom off as they come to only trade with each other and have little use for humans. Depending on how long it takes for the community grow, this may or may not look any different from traditional foom.
This blog post was written as part of research funded by the Foundational Research Institute.
Working on self-compassion: befriending my childhood self
For some reason, I’ve always felt an instinctive dislike towards my childhood self. I generally like kids, but if somebody had magically produced a copy of the person that I was at 5 or 10 and asked me to look after that kid for a while, my automatic reaction would have been “no, I don’t like that kid”.
I’ve also had somewhat of a bad self-esteem for a long, long time. For my tenth birthday, I decided that I didn’t want to get any presents, because I felt like I had done nothing to deserve them. And I didn’t want to get any presents on future birthdays, or on any Christmas, either. (This caused what’s probably one of the oddest child-parent fights that I know of, with my dad being angry about wanting to give me presents and me steadfastly refusing them.)
These two things seemed obviously related.
So today I started exploring that feeling of dislike. Where was it coming from? Why did I have such an aversion regarding my younger self?
Now here’s the thing. I was an only child who frequently spent more time by himself or around adults than he did around other kids. Like all kids, I had a fair share of fights with my parents about stuff like bedtimes and such.
But I never realized that other kids had those same kinds of fights and tantrums too.
I remember having been distinctly shocked when a teacher we had when I was 13-15 made an off-handed comment about this happening with younger kids.
I hadn’t known that this was a Kid Thing: I had thought it was a Kaj Thing.
And as a result, I’d felt guilty and bad over each time that I’d been self-centered and emotional in the way kids are. By the time I heard my teacher make that comment, it started to dawn on me on an intellectual level that this was nothing special: but on an emotional level I had already internalized a belief that I was exceptionally ungrateful and undeserving for everything my parents did for me.
Today I went back to those experiences. A few memories in particular stuck out: one of the countless bedtime struggles, as well as an occasion when I’d told my dad over the phone that I didn’t like him. And now, instead of just recalling my behavior in those memories – like on every previous occasion when I had recalled them – I tried to remember my emotional state, and to sympathize with it, and to recall other kids that I’ve seen acting up and who I’ve felt sympathetic towards.
And then there was a shift, and those memories started feeling like instances of a Kid Thing, rather than a uniquely Kaj Thing.
And now if you’d bring me a copy of me as I was at 5 or 10, I’d just like to hug that poor kid and tell him that it’s okay.
On being a triad and a team
For a few months this fall, I was part of a poly triad which ultimately didn’t work out… but the moments when it did work, worked. So well in fact, that I suspect that any relationship with only two people involved will from now on feel somehow lacking to me, no matter how good otherwise.
There were two of us guys involved with one gal, with the guys starting out as strangers to each other. Still, from the start it was clear that everyone wanted everybody to be happy, and was going to act accordingly.
To me, at the best moments, we felt like family. Not just two men who happened to both have a relationship with the same woman, but a cohesive unit doing its best that everyone in it (as well as the kid from a previous relationship) would be as well off as possible. Thinking back to it, I recall moments like:
- all three brainstorming and looking up stuff about how to make the kid sleep better at night, or to be more willing to sit still while riding a bus
- one of us reading a book aloud to the two others, all three cuddling together
- everyone spending several hours carrying some fresh wood together
- all three sitting together and discussing some conflicts that had come up between two of them, with the third one offering a more neutral outside perspective and acting as a general calming force
It’s hard to describe, but I feel like there was a very strong sense in which there being three of us brought a sense of extra stability to the relationship. If someone was upset or doing badly, nobody needed to feel like they alone had the primary burden of helping that person out. Whoever needed support, there were two other people to shoulder the effort of providing it. And nobody would hesitate to provide it, if only they were in a shape where they could.
While it ultimately didn’t work out, that feeling of being a tight-knit family, with a sense of “one for all, all for one”… I’m going to miss that, in any relationship that doesn’t have it. You can get the sense of mutual support with just a single couple, of course; but things like that sense of “we’re both in love with the same person so we’re going to work together to make her happy; and we know that she cares about us both and will be the happiest if both of us are happy, so we’ll also do our best to help each other out whenever we can”… that I don’t think you can really get without having a triad.
Suddenly, a taste of freedom
So a few days back, I mentioned that after getting rid of my subconscious idealized assumptions of what a relationship “should” be like, I stopped being so desperate to be in a relationship.
And some time before that, I mentioned that I’d decided to put the whole “saving the world” thing on hold for a few years and focus on taking care of myself first.
As a result, I’ve suddenly found myself having *no* pressing goals that would direct my life. No stress about needing to do something big-impact. No constant loneliness and thinking about how to best impress people.
Just a sudden freedom to do basically anything.
I’m still in the process of disassembling various mental habits that were focused on making me more single-mindedly focused on the twin goals of saving the world and getting into a relationship. But starting to suspect that even more things were defined by those goals than I suspected.
For instance, my self-esteem has usually been pretty bad, probably because I was judging myself and my worth pretty much entirely by how well I did at those two goals. And I didn’t feel like I was doing particularly well at either.
Now I can just… Live a day at a time and not sweat it.
It’s going to take a while to get used to this.
This work by Kaj Sotala is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
In addition to making a public comment, you may also send me anonymous feedback.