Enlightenment Now by Steven Pinker (2018)

The Case for Reason, Science, Humanism and Progress

31st December 2018

Presents a positive vision of a new Age of Enlightenment in which human intelligence and culture is shown to be improving the lot of humanity.

This vision is presented as a counterpoint to what Pinker sees as all too prevalent intellectual pessimism, but sadly it ends up becoming smug, unenlightened sneering at other contemporary intellectual discovery.

Chapters 1-18

TBC

Chapter 19 -- Existential Threats

Considers whether the positive signs discussed in earlier chapters are overshadowed by catastrophic or existential risks, meaning a class of threats which may have a low probability but an enormous negative impact. Pinker's stated aim here is not to deny that these risks exist, but to set out a framework in which to think about them. He sees this as necessary, since overemphasising major risks is at best a distraction from more immediate threats and at worst creates dangers in itself, his main example being the claim that nuclear proliferation occurred as a result of unwarranted fear.

Pinker points out that human risk assessment is poor and is based on whether something is imaginable rather than whether it is probable. But, accepting that this is true, this tendency is just one example of built-in human cognitive bias, so this justification seems wide of the mark, since the main windmill that Pinker tilts at here is the 'rationality movement' and that group in fact stresses the need to be aware of and to use techniques to correct for many kinds of cognitive bias.

Although his example of nuclear proliferation needs to be adjusted to take account of the political and economic forces that were the main motors of the post-war arms race, there is a real issue here too of fear caused by ignorance, but the solution to that is more openness. Perversely his argument, that the discussion of low probability risks should be curtailed because it might paralyse people, seems the opposite both of such openness and of the Enlightenment spirit itself.

Turning to the danger of AI specifically, Pinker argues that this is based more our association of intelligence (the ability to reason and plan) with a will to power (seeking dominance and the pursuit of personal goals). Pinker argues that while systems that have evolved in survival of the fittest competition do sometimes have this combination, there is no good reason to suppose that a machine intelligence will have both. Whether that is true however seems to depend a lot on how that machine intelligence is arrived at: if is designed and programmed by software engineers then perhaps it is true that there need be no will to power, but if it is largely a system created using Machine Learning then there is less reason to think it can't happen, particularly if the system development was a hybrid of Genetic Algorithms and Machine Learning. And if machine intelligence is achieved by functional emulation of a human brain then there is no reason to believe that some unwanted evolutionary traits will not be carried over into the machine intelligence.

An important point to make here is that the risk of powerful general-purpose AI doesn't require the AI to be self-aware, or conscious in the sense of having subjective feelings. We don't have to suppose it has its own ends. If an intelligent system vastly exceeds human capability then a large risk can come from even a small misalignment between the goals of the AI and the goals of its creator. This is known as the AI Control Problem, see for example Armstrong, 2017.

A second issue that Pinker identifies is about the nature of intelligence itself, which he defines as how to "pursue various goals in various domains", and he makes the point that the domains which are allocated to machine intelligences will not be the same domains that humans have evolved to deal with. As examples, he says that a machine intelligence might not need to solve problems like "charming mates", just as our human intelligence has not evolved to solve problems like "sorting millions of accounting records". Because there are so many different domains, and so little overlap between human and machine spheres of interest, he says the concept of Artificial General Intelligence itself "is barely coherent". Knowledge, he says, doesn't come from "running an algorithm faster and faster".

There are numerous problems with this line of argument. First, he doesn't consider what goals the sponsor of an advanced AI might have, but some of the best-funded advanced AI initiatives are likely to be from nation-states with military objectives. Given such a sponsor, then problematic goals and inconsistent rules about how humans can be treated may be baked-in to the goals themselves, in which case running the algorithm faster and faster does create a problem if that means there is insufficient time for its human monitors to understand the strategies the AI has devised to achieve those goals. Secondly, once a sufficiently advanced machine intelligence has achieved considerable superiority to humans in one area of expertise, it may rapidly learn how to extend this superiority into other domains too, if it calculates that this will help it achieve its goals. This is the problem of the expansion of cognitive superpowers discussed by Bostrom, 2014, who identified six different superpowers that a machine intelligence might have, any one of which might be used to bootstrap itself to achieve other superpowers if this furthered its goals. Overall then, despite the talk of different domains of intelligence, the problem here is still one of goals: who sets the goals, how the goals can be defined rigorously, how they are translated into targets for the AI and how to protect against interim sub-goals that may emerge.

Once again here, an important point is that the Control Problem doesn't require that the AI is self-aware or conscious in the sense of having subjective experience. It may be sufficient if the AI places a high value on a particular final goal and it reasons that some indirect course of action is the best means to that end. It may then seek abilities in different domains simply as a by-product of pursing its final goal.

The chapter then addresses the "Value Alignment Problem" and pokes fun at some scenarios raised by Bostrom. Pinker's argument is simplistic however, saying that if we are smart enough to build an AGI then we are smart enough to test it for safety, and his conclusion is trite: "The way to deal with this threat [of a Doomsday Computer] is straightforward: don’t build one". This misses the point that even a tiny mistake in establishing the goals of a system will become a huge problem if that system is smarter and faster than you are. And mistakes do happen in software definition all the time.

The most confusing thing about the overall thrust of Pinker's argument in this chapter is that on the one hand he derides 'sci-fi scaremongering' and 'scientific pessimism', while on the other fully accepting that some risks are real and also arguing that awareness of risks will allow intelligent progress towards risk-reduction. That is, the argument seems to be:

This simply makes no sense, as the rationality movement is itself an attempt to bring knowledge and intelligence to bear on issues, while using techniques from statistics and social sciences to compensate for innate predispositions and biases. In reaction to perceived alarmists, Pinker throws the baby out with the bathwater, seemingly because the bathwater wasn't just the right temperature, i.e. didn't strike the right balance between ignorance and over-sensitivity for his liking.

Furthermore in order to make this argument he caricatures the rationality movement as alarmist, but examination of its methods and texts leads to the opposite conclusion: it doesn't say that "humanity is screwed", but on the contrary that a thoughtful and bias-aware investigation of risk-adjusted costs is needed, hopefully leading to informed and effective mitigation of those risks.

It's also a lazy critique, in that Pinker pokes fun at the term "existential risk" but doesn't engage here with a central problem in this area: the problem of how to account for very small probabilities of extremely bad outcomes. The point of rationalist thought-experiments is to expand the frame of reference so that we can make better choices, but Pinker doesn't want any fresh perspectives that might pop his progress-bubble.

The chapter finishes with a review of nuclear proliferation, and concludes that we should continue steady progress towards reducing warhead stockpiles and seek to extend the 'no first use' doctrine. There is nothing much to note about this, except that he doesn't show that progress on this front really requires us to abstain from investigation of other risks.

Overall this chapter misses his mark because it misrepresents its opponents in several ways and it doesn't engage with the central problems:

Despite his initial claim that existential threats are "problems to be solved", all that Pinker's review of the field ends up doing is to suggest that -- aside from the issue of nuclear warhead decommissioning -- these problems can be expected to solve themselves in the normal course of events.

Chapters 19-23

TBC