If, many decades from now, some form of rogue artificial intelligence does manage to follow the playbook of a thousand science-fiction narratives and enslave the human race, I suspect the last remaining historians will look back to an obscure computer science experiment conducted at the turn of this century as an augur of the revolution to come. The experiment was the brainchild of two researchers at the University of Sussex named Jon Bird and Paul Layzell, and it involved a programming technique known as “evolutionary” software that uses a kind of simulated version of natural selection to engineer and optimize solutions to a design problem.
Bird and Layzell wanted to see if they could harness the creativity of natural selection to design an oscillator, one of the key building blocks of modern electronics. (Oscillators output a repetitive signal that is used in everything from the clocks that coordinate activity inside a computer’s CPU to the sounds of synthesized music.) The Sussex researchers created a kind of virtual genome made up of transistors, and used the software to randomly explore possible connections between those transistors, simulating the random explorations of genetic mutation that drive so much of evolutionary change. Configurations that produced outputs that were closer to the rhythmic pulse of a oscillator were selected and then used as the basis for the next round of mutations.
After many generational cycles, Bird and Layzell’s creation settled on a reliable sine wave, creating a signal that was more or less indistinguishable from that of an oscillator designed by human engineers. But when the two scientists examined their contraption, they discovered something very puzzling: There was no evidence of an actual oscillating circuit in the transistors themselves. The system was outputting an oscillating signal, but there wasn’t anything on the circuit board that suggested how the system was creating that signal.
Eventually, Bird and Layzell discovered, to their astonishment, that the mysterious oscillation was not being generated by the transistors, but rather by other computers in the room whose clock cycles were emitting periodic radio waves timed to their own oscillators. Instead of building an oscillator of its own, Bird and Layzell’s creation had hijacked the rhythmic signals being generated by its neighbors.
This was startling enough, but there was a further, even more baffling twist: the “genome” of transistors that they began with did not include a radio antenna. There didn’t seem to be any way for the evolved system to “hear” the signals emanating from the other machines. This required even more detective work. In the end, they discovered that the software had stumbled across the strategy of employing the printed circuit board tracks as an antenna, picking up an extremely faint signal from the surrounding machines, which was then amplified by the transistors.
Bird and Layzell had designed their evolutionary software to select for the production of an oscillating signal, which naturally led them to assume that their creation would set about to build an oscillator. Instead, it took another route. It built a radio.
Nick Bostrom alludes to the story of Bird and Layzell’s evolutionary surprise in the opening chapters of his 2014 book Superintelligence. It’s a passing reference in Bostrom’s account, but to me it seems like a particularly useful case study for helping non-AI experts understand the issues at play in the current debate.
For starters, some sort of evolutionary algorithm is probably the approach humans will need if they are going to engineer a machine capable of thinking at a level that exceeds human thought. The “designer” of the first truly superintelligent machine will likely be natural selection’s “blind watchmaker” and not some 21st-century Steve Wozniak.
That combination of evolutionary engineering and superintelligence, if it does come to pass, will mean that humans will risk being fundamentally incapable of understanding how the thinking machine works, or what materials it draws upon to achieve its objectives. We will all be like Bird and Layzall, scratching our heads, trying to figure out where the sine wave is coming from. In a way, the scariest twist in the oscillator story is how the algorithm was able to harness other technology in the room to achieve its objective, technology that Bird and Layzell hadn’t considered as part of the evolutionary “environment.” Bird and Layzell gave their virtual inventor a handful of building blocks to experiment with, and then let Darwinian selection run its own experiments. And those experiments led to the discovery of new building blocks, escaping the boundaries of the simulation.
This is, in a nutshell, the promise and the potential peril of AI. We invent thinking machines using evolutionary or recursive methods that will themselves exceed human understanding, and the machines at the end of that process are so smart that our attempts to contain them inevitably fail. We won’t be smart enough to realize that they’ve hijacked the oscillators in neighboring computers–only it won’t be oscillators they’ll be hijacking, it’ll be the NASDAQ or a swarm of nanobots or the internet itself.
Reckoning with the severity of this problem mandates that you shed some of your human biases about the scales of intellectual ability. As AI theorist Eliezer Yudkowsky puts it, we have a “human tendency to think of “‘village idiot’ and “‘Einstein’ as the extreme ends of the intelligence scale, instead of nearly indistinguishable points on the scale of minds-in-general.” From the point of view of, say, a mouse, the village idiot and Einstein are both unfathomably intelligent.
We spent the first decades of AI research mostly dreaming about building machines that might function at a village-idiot-level of intelligence, or perhaps reach the Einsteinian summit. But as Bostrom and Yudowsky both argue, there’s no reason to think that the Einsteinian summit is some sort of fundamental limit. “Far from being the smartest possible biological species,” Bostrom writes, “we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization–a niche we filled because we got there first, not because we are in any sense optimally adapted to it.” Powering by recursive, self-learning algorithms, the first true AI might well march right past Mount Einstein and ascend to some higher plateau well beyond our perception.
How likely is it that some form of superintelligence will emerge in the next century or so? Answering that question is beyond the capacity of my middling intelligence, residing as it does somewhere on the slope between village idiot and Einstein. I will say that Bostrom’s book (and Tim Urban’s adaptation at Wait But Why) has made me think that the whole scenario is much more likely than I had previously believed. Certainly, it is less likely than already existing threats like climate change or nuclear proliferation, in that no one so far has designed a computer that comes close to human intelligence, much less some advanced intellectual capacity beyond our understanding. But at the same time, if it does emerge, I suspect artificial superintelligence poses more of an “existential threat” to humanity–i.e., a threat that could potentially put an end to the entire species–than environmental or nuclear catastrophes.
It’s important to remember two ways in which the real-world dystopian AI forecast deviates from the science-fiction version. First, it is not at all necessary that the AI become conscious (or “self-aware” as the original Terminator put it). A superintelligent AI might develop some kind of alternative consciousness, likely completely different from ours. But it also might remain a vast assemblage of insentient calculations, capable of expression and action and long-term planning, but lacking a sense of self.
Secondly, the AI need not suddenly turn evil or vengeful or ambitious (or any other anthropomorphic emotion) to destroy human civilization. Bostrom, for instance, spends almost no time in Superintelligence imagining machines becoming evil overlords; instead, he worries about small miscommunications in defining the AI’s goals or motivations that could lead to global or even cosmic transformations. For example: humans endow the AI superintelligence with an overarching goal of maximizing human happiness, and before long the AI has distributed nanobots into every human brain on the planet, permanently stimulating the pleasure centers of the brain, and turning us all into grinning zombies. (Tim Urban has a comparably tragicomic allegory of an AI attached to a robotic arm; instructed to hand-write a note that says “We love our customers,” the AI inadvertently destroys the universe.)
The threat is not that we’ll ask the AI to build us an oscillator, and it will actively disobey us and instead hack into the Department of Defense network and detonate its entire nuclear arsenal. The threat is that we will ask it to build an oscillator, and it will decide to turn the entire planet–and every living creature on it–into a sine-wave-generating circuit because we haven’t framed the objective clearly enough.
Much of Superintelligence is devoted to thinking through what is sometimes called the “containment problem,” brilliantly explored in Alex Garland’s film Ex Machina: how to keep the genie of AI inside the bottle, while still tapping into its powers. Could humans evolve an AI that was truly superintelligent, but at the same time keep it safely bounded so that a runaway instruction doesn’t trigger a global catastrophe? In Bostrom’s convincing presentation, the problem is much harder than it might first appear, in large part because the humans would be trying to out-think an intelligence that is orders of magnitude more advanced than their own. Containing the AI will be like a mouse scheming to influence human technological advancement to prevent the invention of mousetraps.
Whether you find Bostrom’s argument convincing or not, one fact is undeniable. The AI-as-existential-threat camp is growing in number and in influence, including many minds residing on the upper slopes of Mount Einstein: Bill Gates, Stephen Hawking, Elon Musk. In a way, we are at a point equivalent to where the global warming debate was in the late 1980s: a small group of scientists and researchers and public intellectuals extrapolating out from current trends and predicting a major crisis looming several generations down the line. According to a survey conducted by Bostrom, most of the AI research community believes superhuman-level AI is still at least 50 years away.
That multi-generational time scale may be the most encouraging element in a debate filled with doomsday scenarios. Climate advocates often complain about the sluggish pace of political and corporate reform given the magnitude of the global warming threat. But we should remind ourselves that with climate change we are trying to do something that is arguably without precedent in human history: put in motion sweeping institutional and technological solutions to prevent a threat that will not have a severe impact on most humans for several decades, if not longer. One of the hallmarks of human intelligence is our long-term planning; our ability to make short-term sacrifices in the service of more distant goals. But that planning has almost never extended beyond the range of months or, at best, a few years. Wherever each of us individually happens to reside on Mount Einstein, as a species we are brilliant problem-solvers. But we have never used our intelligence to solve a genuinely novel problem that doesn’t exist yet, a problem we anticipate arising in the distant future based on our examination of current trends.
“This is the function of science fiction. To parse, debate, rehearse, question, and prepare us for the future of new.”
To be clear, humans have engineered many ingenious projects with the explicit aim of ensuring that they last for centuries: pyramids, dynasties, monuments, democracies. Some of these creations, like democratic governance, have been explicitly designed to solve as-of-yet-undiscovered problems by engineering resilience and flexibility into their codes and conventions. But mostly those exercises in long-term planning have been all about preserving the current order, not making a preemptive move against threats that might erupt three generations later. In a way, the closest analogue to the current interventions on climate (and the growing AI discussion) are eschatological: in religious traditions that encourage us to make present-day decisions based on an anticipated Judgement Day that may not arrive for decades, or millennia.
No institution in my knowledge has thought more about the history and future of genuinely long-term planning than the Long Now Foundation, and so I sent an email to a few of its founders asking if there were comparable examples of collective foresight in the historical record. “The Dutch in planning their dykes may have been planning for 100-year flood levels, and the Japanese apparently had generational tsunami levels for village buildings. However, both of these expectations are more cyclical than emergent,” Kevin Kelly wrote back. He went on to say:
I think you are right that this kind of exercise is generally new, because we all now accept that the world of our grandchildren will be markedly different than our world–which was not true before.
I believe this is the function of science fiction. To parse, debate, rehearse, question, and prepare us for the future of new. For at least a century, science fiction has served to anticipate the future. I think you are suggesting that we have gone beyond science fiction by crafting laws, social manners, regulations, etc., that anticipate the future in more concrete ways. In the past there have been many laws prohibiting new inventions as they appeared. But I am unaware of any that prohibited inventions before they appeared.
I read this as a cultural shift from science fiction as entertainment to science fiction as infrastructure–a necessary method of anticipation.
Stewart Brand sounded a note of caution. “Defining potential, long-term problems is a great public service,” he wrote. “Over-defining solutions early on is not. Some problems just go away on their own. For others, eventual solutions that emerge are not at all imaginable from the start.”
Given the accelerating rates of change of modern society, the current debate about AI and its potential threats is a bit like a group of inventors and scientists gathering together in the early 1800s, and saying: “This industrial revolution is certainly going to make us much more productive, and in the long run raise standards of living, but we also appear to be pumping a lot of carbon into the atmosphere, which will likely come back to haunt us in a couple of centuries, so we should think about how to prevent that problem.” But of course, that conversation didn’t happen, because we didn’t have the tools to measure the carbon in the air, or computer simulations to help us predict how that carbon would influence global temperature, or a history of battling other industrial pollutants, or government and academic institutions that monitor climate and ecosystem change. We were smart enough to invent coal-powered engines, but not yet smart enough to predict their ultimate impact on the environment.
This is where the AI debate helps us perceive one decidedly positive trend: all those tools and institutions and sensors that have enabled us to identify the threat of climate change–and start the battle against it–constitute, en masse, their own kind of superintelligence. Until very recently, human beings lacked the capacity to forecast long-term emergent problems, and to engineer laws or reforms or new technologies to prevent those problems from coming into being. These skills simply weren’t part of our mental toolbox. Now they are.
The fact that some of us are still debating whether global warming is even happening shows us that we’re still not experts at this kind of thinking. But in a way, those failures are understandable: It’s hard enough to project forward 50 years as an individual, much less as a society. But debates like the ones we are having today about AI make it clear that we are beginning to exercise these cognitive muscles. With AI, all the projections of future threats may well turn out to be false alarms, either because true AI turns out to be far more difficult to achieve, or because, as Stewart Brand suggests, we discover new techniques that minimize the danger before the machines march on past Mount Einstein. But if artificial superintelligence does turn out to pose an existential threat, our best defense will likely come out of our own new powers of human superintelligence: hypothesizing, predicting, simulating, thinking ahead.
How We Get To Next was a magazine that explored the future of science, technology, and culture from 2014 to 2019. This article is part of our Robots vs Animals section, which examines human attempts to build machines better than nature’s. Click the logo to read more.
