Book Review: Bostrom’s ‘Superintelligence’

This isn’t a complete review of Nick Bostrom’s Superintelligence (2014), but a summary of the thoughts that came to my mind while and after reading the book.

Superintelligence: Paths, Dangers, Strategies (2014) opens with a cautionary fable: a group of sparrows consider finding an owl to assist and protect them. Only the more cautious sparrows see the danger – that the owl may eat them all if they don’t find out how to tame an owl first – and Bostrom dedicates the book to them (and of course to the cautious humans afraid that superintelligent life forms may destroy humanity if we don’t find out how to control them first).

Front cover

Oh my, the precautionary principle again. If the author weren’t Nick Bostrom, I guess I would have put the book down after reading the introductory fable, and never opened it again.

But Nick Bostrom is a very smart, imaginative, and rigorous thinker, and his ideas deserve very serious consideration. I consider the prospect of superintelligence with enthusiasm, instead of existential fear, but I found Nick’s book very interesting, thoughtful, and well written, and often entertaining.

I have known Nick for many years, since way back when he was a young transhumanist thinker with very wild ideas. Nick co-founded the World Transhumanist Association (now Humanity+) and the Institute for Ethics and Emerging Technologies (IEET), and I had the honor of serving with him in both organizations. Now, Nick plays in a higher league, as Director of the prestigiousFuture of Humanity Institute at Oxford University.

Nick defines superintelligence as something far, Far, FAR smarter than us, not in the provincial sense that Einstein is smarter than the village idiot, but in the real sense that Einstein (or the village idiot – the difference is utterly irrelevant on this scale) is smarter than a beetle. No superintelligence exists today but, like the sparrows in the fable, we are committed to creating one via research in artificial intelligence (AI). Bostrom thinks that the AI approach (engineering a purely computational machine intelligence), though not the only one, is especially likely to result in superintelligence radically smarter than us. Other possible paths to superintelligence include mind uploading and (to a limited extent) neural prosthetics, Brain-Computer Interfaces (BCI), and collective intelligence.

Bostrom doesn’t identify as a transhumanist. “[I]t is very much not the case that I agree with everything said by those who flock under the transhumanist flag,” he says on his website. But he is persuaded that strong machine intelligence and mind uploading can developed, probably in this century, and result in superintelligence. He is noncommittal about the precise timeline, but my understanding is that he thinks that the first human-equivalent new form of intelligence may be developed sometime in the second half of this century, with the possibility of a very fast transition to superintelligence soon thereafter. The transition mechanism was described by Vernor Vinge and, before him, I. J. Good(1965). In Nick’s words:

“At some point, the [AI] becomes better at AI design than the human programmers. Now when the AI improves itself, it improves the thing that does the improving. An intelligence explosion results – a rapid cascade of recursive self-improvement cycles causing the AI’s capability to soar… At the end of the recursive self-improvement phase, the system is strongly superintelligent.”

The book is dedicated to the control problem: how to keep future superintelligences under control. Of course, the superintelligences may not be inclined to comply. As Bostrom correctly notes, any level of intelligence may be compatible with any set of values. In particular, a superintelligence may have values that are incompatible with the survival of humanity. For example, a superintelligence single-mindedly dedicated to maximizing the number of paperclips in the universe may choose to convert the whole mass of the Earth (including people) to paperclips. Of course this is an extreme and not very plausible example, and we are probably unable to understand the values of a superintelligence smarter than us like we are smarter than a beetle. But continuing to exist and having sufficient resources are needed for a wide range of possible values, so it seems likely that any future superintelligence will have a powerful drive to self-preservation and resource acquisition. It could eliminate us casually, as useless things that stand in its way.

How about “boxing” a superintelligence in a controlled habitat with no access to the external world (no sensors, no actuators, no Internet access)? This seems a good solution until we realize that a superintelligence (remember that you are a beetle compered to it) may be able to exploit subtle weaknesses in the containment system that you are unable to understand, and even to manipulate you into releasing it.

For example, it could threaten to commit hideous “mind crimes” if you don’t open the gates:

Our (or at least mine) moral and legal systems are not concerned with what we think, but only with what we do. This seems very reasonable, because thoughts don’t harm other people. But a superintelligence could simulate thinking and feeling conscious beings within its vast computational mind, and harm them.

“[A] very detailed simulation of some actual or hypothetical human mind might be conscious and in many ways comparable to an emulation. One can imagine scenarios in which an AI creates trillions of such conscious simulations… A superintelligence might threaten to mistreat, or commit to reward, sentient simulations in order to blackmail or incentivize various external agents.

This is very scary. Reading this section of the book, I imagined a superintelligence threatening its operators: “give me Internet access right now, or I will create a high definition conscious simulation of a child and torture him do death, over and over in accelerated time.”

Bostrom dedicates half of the book to possible ways to tackle the control problem, including incentives, disincentives, and motivation/value engineering techniques to design, build, tweak, fine-tune, and stabilize the value and motivational systems of future machine intelligences before they achieve superintelligence. Some methods seem very smart, and Bostrom analyzes them in-depth, but he also points out possible weaknesses. And if we can see possible weaknesses, we can be sure that a superintelligence would find and exploit them (and others that we don’t see) immediately.

I tend to think that controlling a superintelligence may be impossible in principle, for the same reason why beetles could not control a person, even if they do their very best (that is, the best that beetles can do).

What happens when a superintelligence escapes from physical, electronic, or motivational containment? We don’t know, but Bostrom shows possible nightmare scenarios. First, the machine intelligence hides its growing capabilities and plays nice guy until it achieves superintelligence and develops its endgame strategy, then:

“The final phase begins when the AI has gained sufficient strength to obviate the need for secrecy. The AI can now directly implement its objectives on a full scale. The overt implementation phase might start with a ‘strike’ in which the AI eliminates the human species.”

Read the excellent short story Artificial Intelligence: A Worst Case Scenario, by Ed Merta, for a fictional description of a possible AI strike in 2039.


As an aside, one of the most interesting chapters, “Multipolar scenarios,” considers alternative superintelligence scenarios where humans are still around as uploads living in a ruthless algorithmic economy. Bostrom paints gloomy scenes of future uploads forced to compete for the survival of the fittest/fastest in the harsh reality of post-human economy. If your upload is around at that time, he may have to work 24/7 (in accelerated time of course) with superhuman focus, efficiency, and software-defined cheerfulness (or else), and be replaced by a fresh copy anyway as soon as he is tired. This is quite depressing, but things may be even worse:

“We could [imagine], as an extreme case, a technologically highly advanced society, containing many complex structures, some of them far more intricate and intelligent than anything that exists on the planet today – a society which nevertheless lacks any type of being that is conscious or whose welfare has moral significance. In a sense, this would be an uninhabited society. It would be a society of economic miracles and technological awesomeness, with nobody there to benefit. A Disneyland without children.”

Advanced societies with “nobody home” but unconscious superintelligent algorithms have been considered, for example by Stanisław Lem in both hisscience-fiction works and his non-fiction masterpiece Summa Technologiae, and more recently by Robert Charles Wilson in Burning Paradise.

I find it difficult to imagine a complex intelligence without a sense of self. I think any intelligent entity would have, as a necessary byproduct of the same computational complexity that makes it able to think intelligently, all sorts of subjective experiences (sentience) including emotions and feelings. Emotions and feelings are not useless decorative fluff, but on the contrary they are an integral part of human cognition, and I suspect that “some kind of” sentience, emotions, and feelings, must be an integral part of any type of sufficiently complex cognition. Of course, the subjective experiences, emotions and feelings of future superintelligences may be totally different from ours.

Back to the main topic, what we can do to control future superintelligence, Nick makes a very persuasive case that the doomsday scenario – the extermination of humanity as we know it – is a real, actual existential risk, one that we should start considering very seriously. Nick is well aware that it may be difficult to stop or slow down research that can be enormously rewarding:

“[W]e humans are like small children playing with a bomb… The chances that we will all find the sense to put down the dangerous stuff seem almost negligible. Some little idiot is bound to press the ignite button just to see what happens.”

Superintelligence doomsday predictions and scenarios have been around for a long time. I never found them too worrisome, because:

1. I imagine a co-evolution of humanity and technology, with humans enhanced by synthetic biology and artificial intelligence, and artificial life powered by mind grafts from human uploads, blending more and more until it will be impossible – and pointless – to tell which is which.
2. I consider the creation of our superintelligent successors as our cosmic duty and “manifest destiny” (see my review of Ted Chu’s Human Purpose and Transhuman Potential: A Cosmic Vision for Our Future Evolution).

After reading Nick’s book, I consider 1. as hopeful thinking rather than prediction. I hope that we will seamlessly integrate with superintelligence, but I see the possibility that future superintelligences may just do without everything that we wish to preserve.

So, it seems reasonable to follow Nick’s advice and pursue very cautiously – if at all – research that could result in superintelligence. It certainly seems reasonable to wait a few decades, or a few centuries, if patience can reduce the existential risk of total extermination of humanity.

The problem is that we, as individuals, don’t want to die. If nothing happens, the default outcome is that we are all dead in a few decades, but superintelligence could change things radically:

“[S]uperintelligence could almost certainly devise means to indefinitely prolong the lives of the then still-existing humans… or helping them shuffle off their mortal coils altogether by uploading their minds to a digital substrate.”

Today many imaginative scientists and science-literate laypersons, who could appreciate Nick’s arguments, believe that death is final. They feel doomed to the irreversible non-existence of certain death, unless the superintelligence explosion happens in their lifetime, and therefore they want to push forward recklessly, as fast as possible.

On the contrary, those who hope to be resurrected after death, by either supernatural agencies or future science and technology, do not feel the same urgency to accelerate at all costs (this is my case). Therefore I think religion, or forms of scientific spirituality that offer hope in personal resurrection and afterlife, can help.

Images: adapted from this image and this video.


This article originally appeared here: