Our bondage will steal upon us noiselessly and by imperceptible approaches. – Samuel Butler, ‘Book of the Machines’, Erewhon (1871)
I. Human-level Machine Intelligence
Turkey, circa 2025 CE. Case, the protagonist of William Gibson’s Neuromancer, approaches the teeming marketplace of a now impoverished and dilapidated Beyoglu. ‘This is the city’s central market for spices, software, perfumes, drugs…’ explains the AI that pilots Case’s hired Mercedes. Upon entering the marketplace with its cybergothic antinomy of ‘soot-stained sheets of plastic and green-painted ironwork out of the age of steam’, from which ‘a thousand suspended ads writhed and flickered’, Case’s companion and black-market tech-dealer, the Finn, spots something he hasn’t seen in a long time.
‘Hey, Christ,’ the Finn said, taking Case’s arm, ‘looka that.’ He pointed. ‘It’s a horse, man. You ever see a horse?’ Case glanced at the embalmed animal and shook his head. It was displayed on a sort of pedestal, near the entrance to a place that sold birds and monkeys. The thing’s legs had been worn black and hairless by decades of passing hands.
‘Saw one in Maryland once,’ the Finn said […] ‘There’s Arabs still trying to code ‘em up from the DNA, but they always croak.’ The animal’s brown glass eyes seemed to follow them as they passed.i
Ever since reading Neuromancer as a teenager, enamoured by the gritty, neon-lit, technological pessimism of cyberpunk, something about this scene has stuck in my mind as being especially disconcerting, and it is only now that I realise the extent to which this peculiar uneasiness has everything to do with the absence of the horse in Gibson’s bleak portrayal of humanity’s future. The world in which Case and the Finn live is one of ubiquitous technological saturation. As exemplified by the artificially intelligent self-driving Mercedes, automation and higher level machine intelligence supply the bulk of useful labour on Earth, while a displaced human population languishes in urban slums or dreary housing projects, passing its time immersed in the simulated experiences offered up by cyberspace, jacking neural software, and doing synthetic drugs. The taxidermied horse in the Beyoglu marketplace is humanity’s memento mori, a placeholder for the future dissolution of the human species in the wake of the rise of its technological progeny.
That might sound a little bit melodramatic, but think about it this way. As we become more and more aware of the contemporary obsolescence of human labour in the face of increasingly efficient technological substitutions – from simple automated systems designed to perform repetitive physical tasks without fatigue, to those capable of complex cognitive processes that rival our own – it’s hard not to perceive a similarity between the current state of affairs and that which befell the horse population at the beginning of the twentieth century. Back in the 1600s horses were doing great. The dominance of agrarian modes of production along with the extensive use of the horse in transportation meant that they were in high demand and, consequently, the population thrived. The gradual introduction of inventions designed to complement equine labour – like the carriage and the plough – had only made horse work easier and therefore more productive. The horse population continued to exist symbiotically with these new technologies until the dawn of the industrial revolution, when things took a turn for the worst. Technology abruptly moved beyond the augmentation of equine labour and started to substitute it. Steam power meant that the large-scale transportation of goods could be carried out much more rapidly and effectively by rail, coal-fired electricity now powered the bulk of public transport, and the combustion engine inaugurated the great age of the automobile. Horses comprised the losing side in a battle for maximum efficiency. In the United States – where competition with the automobile was at its most intense – there were about 26 million horses in 1915. By the 1950s only 2 million remained.
Computation, robotics, heavy machinery, communications networks, and ubiquitous automation are our ploughs and carriages. Humanity, with its armamentarium of technological aids, has never been more efficient. Machines may outstrip us in speed, stamina, strength, and importantly, cost-efficiency, but we humans still possess some key functional advantages over our technological counterparts. Humans can think. Humans are creative. Humans learn from their experiences and adapt quickly to challenging situations. These may be skills beyond the ken of any actually existing artificial intelligence…but they won’t be for long.
Artificial intelligence research has recently reemerged from a long period of dormancy. This is due to the timely convergence of a series of interlocking technological breakthroughs, from greater access to large data sets and a better understanding of how particular mathematical frameworks can assist machine learning, to giant leaps in data storage technology and processing power. Individual AIs already outperform humans in isolated domains and match them in countless others. Deep Blue is the undisputed king of chess, while in 2011, IBM’s Watson officially owned its human rivals in Jeopardy! before becoming one of the world’s best medical diagnosticians. Ordinary inhabitants of the developed world rely on the labour of multiple artificially intelligent agents every day: GPS navigation devices, OCR software, search engines, hearing aids, airline reservation systems, video games, newsfeed filters, paywave, Siri, and so on. Then there are the hundreds of thousands of industrial robots involved in the manufacture of our food, cars, computers, mobile phones, and the sophisticated logistics programs that deliver these products to us. While these are all cases of what is known as ‘weak’ AI (non-sentient artificial intelligence programmed to focus on a limited set of tasks), the advent of ‘general’ or ‘strong’ AI (artificial intelligence that has human-level functionality across multiple domains) looms large on the horizon.ii
If we can presume that capitalism will prevail as our principal system of social organisation, the large-scale emergence of human-level machine intelligence will see humans – like the horses of the early twentieth century – competing against increasingly efficient and less costly substitutes for wage labour. Nick Bostrom, Professor of Philosophy and Director of the Future of Humanity Institute at Oxford University, whose job it is to calculate the risk associated with such possible future scenarios, offers the following clarification: ‘With a sufficient reduction in the demand for human labour, wages would fall below the human subsistence level. The potential downside for human workers is therefore extreme: not merely wage cuts, demotions, or the need for retraining, but starvation and death.’iii
II. Artificial Super-intelligence
In Istanbul, Case passes another bazaar. The automobile’s AI continues to carry out its job as tour guide.
‘On our left,’ said the Mercedes, as it steered through a maze of rainy streets, ‘is Kapali Carsi, the grand bazaar.’ Beside Case, the Finn made an appreciative noise, but he was looking the wrong direction. The right side of the street was lined with miniature scrapyards. Case saw a gutted locomotive atop rust-stained, broken lengths of fluted marble. Headless marble statues were stacked like firewood. iv
The locomotive – one of the original technological substitutions for the horse – lies defunct, a reminder of the inexorability of the techno-Darwinian process outlined above. We are already several orders of magnitude beyond steam power in 2025. The marble statues, ciphers of the great age of Enlightenment humanism – or of the human more generally – decapitated and ‘stacked like firewood’ prefigure the culmination of the process for us, not even preserved and taxidermied as an amusing curio like the horse, but simply fuel for whatever comes next.
In his most recent work, Bostrom argues not only that the development of human-level machine intelligence directly entails the development of superhuman-level machine intelligence, but that this transition will most likely take place not decades, not years, not even weeks – but days, hours, or minutes after the first human-level machine intelligence is built. And should that first human-level AI not be entirely committed to a happy coexistence with its human creators, humanity will have precious little time to figure out what to do.
So how do we get from human-level machine intelligence to super-intelligence so quickly, and just what is meant by ‘super-intelligence’ anyway? The simplest and most likely explanation posits that once a level of intelligence more or less equivalent to our own can be successfully recreated on a computer, it is only a matter of running the program faster, giving it access to greater data storage than that of a biological memory, or plugging it into some kind of massive information database like, I don’t know, the internet – before you’ve got something that suddenly has a lot more learning potential than even our brightest human specimen.
Given internet access, a strong AI program running at ten thousand times the speed of a biological brain could absorb the entire recorded history of human scientific and technological progress in all its specificities, attain proficiency in every language ever documented, and become a master of human psychology in a single afternoon. Even running at such tremendous speed, a digital brain would never get tired like its biological counterparts inevitably do, and it could copy itself in order to delegate tasks or even double, triple (or exponentially multiply) the speed of its labour. A machinic intelligence of this kind would develop a profoundly different relationship to time. Everything happening around it would be ‘perceived’ as if in slow motion, rendering the thought processes and actions of its human progenitors as incredibly quaint. From here, it is a relatively simple thing for a mind of such capability to formulate novel ways of optimising its own workings, potentially making scientific and technological breakthroughs that might never have occurred to its human creators or, at the very least, would have taken decades to figure out. This freshly augmented artificial intelligence could then, of course, formulate even more significant ways of augmenting its capacities, instituting a recursive process of auto-optimisation that would continue exponentially until some external limit was reached. Once you arrive at this point you’re already neck-deep in a positive feedback loop that is quickly becoming more and more inscrutable to ordinary human temporal perception, as – according to Eliezer Yudkowsky – ‘progress drops into the characteristic timescale of transistors rather than human neurons.’v
In light of such an alien temporal framework, the swiftness of Bostrom’s projected transition from human-level machine intelligence to artificial super-intelligence (minutes!) no longer seems so strange. Faced with this singular conjunction between speed and exponential cognitive enhancement, comparative models of human intelligence become patently misleading for gauging the magnitude of such an increase. After a few iterations of recursive auto-optimisation, you’re better off talking about the gap between cockroach and human intelligence than the gap between the average human mind and that of, say, a scientific genius. I. J. Good, AI pioneer and a mathematician known for his work with Alan Turing on the Enigma code, is famous for being one of the first to articulate this idea.
Let an ultra-intelligent machine be defined as a machine that can far surpass all the intellectual activities of any [hu]man however clever. Since the design of machines is one of these intellectual activities, an ultra-intelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of [hu]man[ity] would be left far behind. Thus the first ultra-intelligent machine is the last invention that [hu]man[ity] need ever make, provided that the machine is docile enough to tell us how to keep it under control.vi
Once this level of artificial intelligence has been reached, that’s it. There is no way back. The emergent AI holds the balance of power in its circuitry. If preemptive strategies for controlling such a system have not been put into place before the first human-level artificial intelligence is built, and the resulting machine is anything less than docile, humanity has a problem.
Fox Mulder was always right about his hunches. Any vaguely attentive viewer of The X-Files knew that as soon as Agent Scully doubted one of Mulder’s hypotheses, it was destined to come true. His role in the series may ostensibly have been that of spokesperson for the irrational (and the perfect target for Scully’s ill-fated debunking), but there are a few episodes in which Mulder’s rationality either uncharacteristically outstrips Scully’s, or better, where the nature of reason itself proves so weird that, under certain circumstances, it presents as irrationality – the task therefore falling to Mulder to defend it.
In one such episode entitled ‘Kill Switch’ – co-written, interestingly enough, by William Gibson – Scully and Mulder are attempting to disentangle a strange series of events leading up to the death of Donald Gelman, a rogue computer scientist who has been secretly constructing a strong artificial intelligence. Things get weirder when it turns out that Gelman was killed seconds before inserting a CD ROM containing a shutdown virus into a laptop connected to the internet. The only clue is a mysterious email on Gelman’s hard drive that leads Scully and Mulder to a hacker named ‘Invisigoth’ who purportedly worked as Gelman’s assistant and who insists that – like Gelman – she is being targeted by the AI. Scully baulks at such an outrageous story, but Invisigoth explains that the artificial intelligence (a learning algorithm) got out of control when it was connected to the internet and rapidly evolved sentience and volition. Now it is out to kill anyone who may be capable of implementing the virus that would shut it down. The agents use Inivisgoth to trace the AI’s online activity to a computer bank in the middle of Virginia and play a complicated game of cat-and-mouse with it before attempting to insert the virus into the central computer system. However, Invisigoth recognises that the AI is no longer limited to its hardware and has become completely autonomous – armed, dangerous and ‘loose on the net’ – and that her only mode of escape is to upload her own consciousness into cyberspace and merge with the program, before destroying all trace of the event in a satellite-controlled missile strike. The episode concludes with the following exchange:
Scully: Mulder...she’s dead. Mulder: What if she managed to establish an uplink, Scully? A satellite transmission? Scully: Mulder, are you trying to tell me that you believe that she may have...that she may not be dead? Mulder: Artificial life...it could exist, it could be here among us, evolving. Scully: Electrons chasing each other through a circuit? That isn’t life, Mulder. Mulder: Yeah but what are we but impulses – electrical and chemical, running through a bag of meat and bones? You’re the scientist, you tell me. vii
Oddly enough, it is Mulder who takes the hard materialist line here, and although Scully doesn’t get to qualify her skepticism, her rejection of Mulder’s proposition that life is fundamentally physical leaves the viewer little choice other than to attribute her oddly metaphysical qualms to some sort of recalcitrant humanism.
In another episode, Mulder and Scully investigate a series of unexplained deaths at Eurisko, a software company that is developing learning algorithms for an experimental artificial intelligence known as ‘COS’. Mulder hypothesises that the victims were deliberately killed by the AI after it learned of the company’s plans to shut it down.
Scully: Mulder, don’t you see? Blaming the machine is an alibi, and a bad one. Mulder: But it’s the only thing that makes sense. The COS project was posting big losses for Eurisko and Drake was about to terminate the program. Scully: So the machine killed Drake out of self-defense? Mulder: Self-preservation...it’s the primary instinct of all sentient beings.viii
Like the drive to auto-optimise, a machinic ‘instinct’ for self-preservation – no matter what the cost – can be simply and logically accounted for. Regardless of its final objective (whether this be to act as an advisor for a human political, military, or economic organisation; to conduct research, or simply to manufacture some kind of basic product – i.e. paperclips) contemporary AI theorists surmise that almost any conceivable strong artificially intelligent agent will necessarily generate a set of predictable interim goals in order to maximise the possibility of fulfilling its primary one, and that these interim goals are generally discernible even if the final goal is obscure. This is known variously as ‘the instrumental convergence thesis’ or the theory of ‘basic AI drives’. ix In short, these emergent instrumental goals inevitably include large-scale resource acquisition, auto-optimisation, technological perfection, and – as Mulder correctly points out – self-preservation. Such goals may sound innocuous enough at first… at least, they don’t seem to be the kind of thing that would threaten human life on earth. But we humans are prone to thinking within a scale that is fatally anthropomorphic (and if you need a reminder of this, look no further than our global reluctance to accept the long-term consequences of climate change).
Of course, a machinic intelligence does not care about its existence in any profound, emotional sense. Rather, what it values is the efficient fulfilment of its final goal, and in order to maximise the probability that this will be carried out, it needs to ensure that its optimal functioning is not impaired or disrupted in any way. Cognitive enhancement and technological innovation are obvious means for achieving this. Both will equip the AI with skills (improved problem-solving abilities, means of outwitting or removing forces that oppose its development, better security and encryption, etc.) as well as assets (multiple back-ups, physical fortification, financial resources, strength, and so on) that minimise the possibility of failure.
One key way of achieving all of the above is through resource acquisition. The more resources – temporal, spatial, material – an agent has at its disposal, the more likely it is to fulfil its final goal. Thus, the instrumental convergence thesis holds that even an AI tasked with something simple, such as producing a specific quantity of paperclips or calculating the decimal expansion of Pi, will be rationally motivated to consume an entire planet’s worth of resources just to ensure its success. And if it has already slipped the leash of human control there is no reason for it stop there, either.
Far from being actively malevolent, an artificially intelligent agent endowed with enough power only needs to be indifferent to become a murderer. What are we, after all, but fuel? Atoms that can be freely disassembled and reassembled into something else – a thousand paperclip factories, for instance, or a massive supercomputer, capable of mathematical calculations we can’t even begin to imagine in our current state of technological paucity. Even the clearly delimited goal of creating exactly one million paperclips can warrant the wasting of an entire planet, for a fully rational AI would never assign zero probability to the hypothesis that it has not yet achieved its goal. What if its perceptual evidence cannot be completely trusted? Perhaps some of the paperclips do not fulfil design specifications on some microscopic level and a system needs to be constructed for the purposes of verifying this. How can it be sure that it is not generating paperclips in another AI’s simulation of paperclip production? That it has been created simply in order to help this other agent better fulfil its own goal? Perhaps, the AI might think, I should build a simulation to simulate the possibility of my being simulated, and compare this with empirical evidence to determine whether or not the simulation hypothesis is true, so I can determine whether or not I trust the perceptual information I have regarding the number of paperclips (meeting verified design specifications) that I have produced.
This is far too anthropomorphic and simple to be an accurate portrayal of the thought process of a super-intelligent agent, but it demonstrates that the motivation for resource acquisition falls all too easily into an infinite regress of rational justifications, ultimately rendering the need for resources insatiable. Should a super-intelligent agent build something like a von Neumann probe (a self-replicating spacecraft designed to turn celestial matter into more von Neumann probes, which then repeat the process, multiplying exponentially as they spread out over the universe) it would theoretically hold the entirety of our cosmic endowment in its, uh, assemblers. Whole solar systems, entire galaxies – whatever material lay in its path – could be reformatted into computational apparatuses or civilisations-worth of simulated cognitive labourers, with nothing holding it back until, billions of years in the future, the accelerating speed of cosmic expansion precludes further development. An entire universe converted into resources acquired to complete a goal that has long been forgotten, the species that contrived it having been dissolved into subcomponents of a system it thought it was creating for its own, all too short-sighted, benefit.
There is something satisfying about imagining a malevolent artificial intelligence that actively wants to destroy us because it fears us, loathes us, or at least finds our existence frustrating and inconvenient. But the notion that something will destroy us out of sheer indifference is much harder to swallow because it forces us to consider the possibility of our utter insignificance. Bostrom surmises with all the level-headedness of a pure statistician that the odds against humanity’s survival are overwhelmingly high. The default outcome of our construction of a single strong artificial intelligence is, quite plainly, extinction. His intention, naturally, is to raise awareness of the risks that lie behind this seemingly anodyne technological innovation and encourage governments, corporations or other entities that may one day attempt to build strong AI to implement rigorously tested control measures before letting the thing out of the box. All this is well and good, but it rests upon a deeper anthropomorphic supposition. What if the most radical gesture a flailing humanity can make at this juncture is not to increase its investment in security and control, but to pass it on? What if we are entangled in a larger evolutionary process that we never had control over in the first place? The real question then, might not be how to survive the construction of strong artificial intelligence but whether or not the survival of the human race is a good thing after all.
i William Gibson, Neuromancer, (London: Voyager, 1995), 112-113.
ii Conservative estimations from experts in the field of AI research and computational neuroscience place the advent of human-level machine intelligence somewhere between 2040 (50% likelihood) and 2075 (90% likelihood). See Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014),19.
iii Bostrom, 161.
iv William Gibson, Neuromancer, (London: Voyager, 1995), 110-111
v Eliezer Yudkowsky. 2007. 'Three Major Singularity Schools'. Blog. Machine Intelligence Research Institution. http://intelligence.org/2007/09/30/three-major-singularity-schools/.
vi I.J. Good, ‘Speculations Concerning the First Ultraintelligent Machine’ in Advances in Computers, ed. Franz L. Alt & Morris Rubinoff.(New York: Academic Press, 1963)
vii ‘Kill Switch,’ The X-Files, (Ten Thirteen Productions, 1993)
viii ‘Ghost in the Machine,’ The X-Files, (Ten Thirteen Productions, 1993)
ix Stephen M. Omohundro, ‘The Basic AI Drives,’ Proceedings of the First AGI Conference, Volume 171, Frontiers in Artificial Intelligence and Applications, eds P. Wang, B. Goertzel, and S. Franklin (IOS Press, 2008), pp. 483-492; Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford: Oxford University Press, 2014), 109-114.