Prey is a thriller, well constructed and fun to read, like Michael Crichton’s other books. The main characters are the narrator, Jack, and his wife, Julia, parents of three lively children, successfully combining the joys of parenthood with the pursuit of brilliant careers in the high-tech world of Silicon Valley. Julia works for a company called Xymos that is developing nanorobots, tiny machines that can move around and function autonomously but are programmed to work together like an army of ants. Jack works for a company called MediaTronics that makes software to coordinate the actions of large groups of autonomous agents. His programs give intelligence and flexibility to her machines.
Things start to go wrong when Jack loses his job and is left to take care of the kids, while Julia is working longer and longer hours at her laboratory and losing interest in the family. She is engaged in a secret struggle to develop her nanorobots into a stealthy photo-reconnaissance system that can be sold to the United States Army. To increase the power and performance of the system, she incorporates living bacteria into the nanorobots so that they can reproduce and evolve rapidly. She reprograms them with Jack’s newest autonomous-agent software so that they can learn from experience.
Even with these improvements the nanorobots fail to meet the Army’s specifications, and Xymos loses its Army funding. After that, Julia desperately tries to convert the photo-reconnaissance system into a medical diagnostic system that can be sold on the civilian market. Her idea is to train the nanorobots to enter and explore the human body, so that they can locate tumors and other pathological conditions more precisely than can be done with X-rays and ultrasound working from the outside.
Experimenting with the medical applications of her nanorobots, she uses herself as a guinea pig and becomes chronically infected. The nanorobots learn how to establish themselves as symbionts within her body, and then gradually gain control over her mind. In her deranged state, she deliberately infects three of her colleagues at the laboratory with nanorobots. She also lets a swarm of nanorobots loose into the environment where they prey upon wildlife and rapidly increase in numbers.
The main part of the story concerns Jack’s slow realization that something is seriously amiss with his wife and with the project in which she is engaged. Only at the end does he understand the full horror of her transformation. With the help of a loyal young woman friend, he confronts Julia and douses her with a spray of bacteriophage that is lethal to the bacteria inside her. But Julia and her infected colleagues are no longer able to survive without the symbiotic nanorobots that have taken over their minds. Under the spray of bacteriophage they collapse and die, like the Wicked Witch of the West in The Wizard of Oz when Dorothy throws a bucket of water over her. After Julia’s demise, Jack and his girlfriend finish the job of destroying the nanorobots inside and outside the laboratory with fire and high explosives. In the final scene, Jack is back with his kids, wondering whether the nanorobots are gone for good, or whether the Xymos corporation may still be developing other nanorobot projects that will turn into nightmares.
What are we to make of this fairy story? There are two ways to look at it. On the one hand, we may enjoy it as a story and not worry whether some parts of it might come true. On the other hand, we may read it as an urgent warning of dangers lying ahead if present-day technological developments are allowed to continue. The author says plainly, in an introductory chapter with the title “Artificial Evolution in the Twenty-first Century,” that he intends his story to be taken seriously.
It is easy to demonstrate that the details of the story are technically flawed. Consider for example the size of the nanorobots. In a commercial presentation advertising the Xymos medical diagnostic system, Julia says, “We can do all this because the camera is smaller than a red blood cell.” The camera is one of her nanorobots. It must be as small as that, since Julia describes it swimming in the human bloodstream inside the capillaries that carry blood through the lungs. The capillaries are only just wide enough for red blood cells to pass through. But later in the book Jack encounters swarms of nanorobots chasing him in the open air like a swarm of ants or bees. These nanorobots are flying through the air as fast as he can run. Fortunately for Jack and unfortunately for the story, the laws of physics do not allow very small creatures to fly fast. The viscous drag of air or water becomes stronger as the creature becomes smaller. Flying through air, for a nanorobot the size of a red blood cell, would be like swimming through molasses for a human being. Roughly speaking, the top speed of a swimmer or flyer is proportional to its length. A generous upper limit to the speed of a nanorobot flying through air or swimming through water would be a tenth of an inch per second, barely fast enough to chase a snail. For nanorobots to behave like a swarm of insects, they would have to be as large as insects.
Other technical flaws in the story are easy to find. The swarms of nanorobots flying in the open air are said to be powered by solar energy. But the solar energy falling onto their very small area is insufficient to power their movements, even if we credit them with a magical ability to use solar energy with 100 percent efficiency. I could continue with a list of technical details that are scientifically impossible for one reason or another, but that would miss the main point of the story. The story is about human beings and not about nanorobots. The main point is that Julia is a credible human being. She is a capable and well-meaning woman in a responsible position, with the fate of a company resting on her shoulders. She decides that the only way to save the company from bankruptcy is to push ahead with a risky technology. Unable to face the failure of her company and her career, she continues with her experiments regardless of the risks. She is a gambler playing for such high stakes that she cannot afford to lose. In the end she loses not only her company and her career but her family and her life. It is a credible human story, and in the end the technical details do not matter.
This story reminds me of Nevil Shute’s On the Beach, published in 1957, a novel describing the extinction of mankind by radiological warfare. Shute’s poignant translation of apocalyptic disaster into the everyday voices of real people caught the imagination of the world. His book became an international best seller and was made into a successful film. The book and the film created an enduring myth, a myth which entered consciously or subconsciously into all subsequent thinking about nuclear war. The myth pictures nuclear war as silent inexorable death from which there is no escape, with radioactive cobalt sweeping slowly down the sky from the Northern to the Southern Hemisphere. The people of Australia, after the northern hemisphere is dead, live out their lives quietly and bravely to the end. The Australian government provides a supply of euthanasia pills for citizens to use when the symptoms of radiation sickness become unpleasant. Parents are advised to give the pills to their children first before they become sick. There is no hope of survival; there is no talk of building an underground Noah’s Ark to keep earth’s creatures alive until the cobalt decays. Shute imagined the human species calmly acquiescing in its extinction.
The myth of On the Beach is technically flawed in many ways. Almost all the details are wrong: radioactive cobalt would not substantially increase the lethality of large hydrogen bombs; fallout would not descend uniformly over large areas but would fall sporadically in space and time; people could protect themselves from the radioactivity by sheltering under a few feet of dirt; and the war is supposed to have happened in 1961, too soon for even the most malevolent country to have acquired the megatonnage needed to give a lethal dose of radiation to the entire earth. Nevertheless, the myth did what Shute intended it to do. On the fundamental human level, in spite of the technical inaccuracies, it spoke truth. It told the world, in language that everyone could understand, that nuclear war means death. And the world listened.
Prey is not as good as On the Beach, but it is bringing us an equally important message. The message is that biotechnology in the twenty-first century is as dangerous as nuclear technology in the twentieth. The dangers do not lie in any particular gadgets such as nanorobots or autonomous agents. The dangers arise from knowledge, from our inexorably growing understanding of the basic processes of life. The message is that biological knowledge irresponsibly applied means death. And we may hope that the world will listen.
From this point on, I assume that the basic message of Prey is true. I assume that the growth of biological knowledge during the century now beginning will bring grave dangers to human society and to the ecology of our planet. The rest of this review is concerned with the question of what we should do to mitigate the dangers. What is the appropriate response to dangers that are hypothetical and poorly understood? In this matter, as in other situations where public health hazards and environmental risks must be assessed and regulated, there are two strongly opposed points of view. One point of view is based on the “precautionary principle.” The precautionary principle says that when there is any risk of a major disaster, no action should be permitted that increases the risk. If, as often happens, an action promises to bring substantial benefits together with some risk of a major disaster, no balancing of benefits against risks is to be allowed. Any action carrying a risk of major disaster must be prohibited, regardless of the costs of prohibition.
The opposing point of view holds that risks are unavoidable, that no possible course of action or inaction will eliminate risks, and that a prudent course of action must be based on a balancing of risks against benefits and costs. In particular, when any prohibition of dangerous science and technology is contemplated, one of the costs that must be considered is the cost to human freedom. I call the first point of view precautionary and the second point of view libertarian. In April 2000, Bill Joy, co-founder and chief scientist at Sun Microsystems, a large and successful computer company, published an article in Wired magazine with the title “Why the Future Doesn’t Need Us,” and the subtitle “Our most powerful 21st-century technologies—robotics, genetic engineering, and nanotech—are threatening to make humans an endangered species.” It was a big surprise to see one of the leaders of high-tech industry arguing passionately for a slowing-down of technology that might become dangerous. Bill Joy became a spokesman for the precautionary view.
Nine months later, in January 2001, the annual meeting of the World Economic Forum was held in Davos, Switzerland. Most of the people at the Forum are captains of industry, presidents of foundations, or government officials. But in 2001 they decided to invite some scientists and writers and artists to add some intellectual sparkle to the meeting. Bill Joy and I were both invited and asked to debate the question: Is our technology out of control? Bill Joy was taking an extreme position on the precautionary side, and I was asked to take an extreme position on the libertarian side, to make the debate interesting. In what follows I shall summarize our debate. To be sure that I am not misrepresenting Bill Joy, I quote here only from his published writings.
My first quote is from Bill’s article in Wired:
The 21st-century technologies—genetics, nanotechnology, and robotics (GNR)—are so powerful that they can spawn whole new classes of accidents and abuses. Most dangerously, for the first time, these accidents and abuses are widely within the reach of individuals or small groups. They will not require large facilities or rare raw materials. Knowledge alone will enable the use of them.
Thus we have the possibility not just of weapons of mass destruction but of knowledge-enabled mass destruction (KMD), this destructiveness hugely amplified by the power of self-replication.
I think it is no exaggeration to say we are on the cusp of the further perfection of extreme evil, an evil whose possibility spreads well beyond that which weapons of mass destruction bequeathed to the nation-states, on to a surprising and terrible empowerment of extreme individuals.
This was written a year and a half before the events of September 2001. I don’t know whether Bill at that time had Osama bin Laden in mind. He certainly had in mind the possibility of a Unabomber taking his revenge on society with genetically engineered microbes rather than with chemical explosives.
Second quote. Here Bill is quoting Eric Drexler, the chief prophet of nanotechnology. Drexler set up the Foresight Institute to promote the benign uses of nanotechnology and to warn against the dangerous uses. Here is Drexler:
Tough omnivorous [synthetic] “bacteria” could out-compete real bacteria: They could spread like blowing pollen, replicate swiftly, and reduce the biosphere to dust in a matter of days. Dangerous replicators could easily be too tough, small, and rapidly spreading to stop—at least if we make no preparation. We have trouble enough controlling viruses and fruit flies….
We cannot afford certain kinds of accidents with replicating assemblers.
The idea of nanotechnology is to build machines on a tiny scale that are as capable as living cells, but made of different materials so that they are more rugged and more versatile. One kind of nanomachine is the assembler, which is a tiny factory that can manufacture other machines, including replicas of itself. Drexler understood from the beginning that a replicating assembler would be a tool of immense power for good or for evil. Fortunately or unfortunately, nanotechnology has moved more slowly than Drexler expected. Nothing remotely resembling an assembler has yet emerged. The most useful products of nanotechnology so far are computer chips. They have no capacity for replicating either themselves or anything else.
My last quote from Bill Joy comes from an article he published in The Washington Post, summing up the dangers that he foresees and recommending a program of action to avoid them:
We who are involved in advancing the new technologies must devote our best efforts to heading off disaster. I offer here a list of first steps suggested by our history with weapons of mass destruction:
(1) Have scientists and technologists (and corporate leaders as well) take a vow, along the lines of the Hippocratic Oath, to avoid work on potential and actual weapons of mass destruction….
(2) Create an international body to publicly examine the dangers and ethical issues of new technology….
(3) Use stricter notions of liability, forcing companies to take responsibility for consequences through a private-sector mechanism—insurance….
(4) Internationalize control of knowledge and technologies that have great potential but are judged too dangerous to be made commercially available….
(5) Relinquish pursuit of that knowledge and development of those technologies so dangerous that we judge it better that they never be available. I too believe in the pursuit of knowledge and development of technologies; yet, we already have seen cases, such as biological weapons, where relinquishment is the obvious wise choice.
Next comes my response to Bill Joy. I agreed that the dangers he described are real, but I disagreed with some details of his argument, and I disagreed strongly with his remedies. I began by speaking about the history of biological weapons and gene-splicing experiments, and the successes and failures of efforts to regulate them. Bill Joy ignores the long history of effective action by the international biological community to regulate and prohibit dangerous technologies. Gene-splicing experiments began in many countries when the technique of sticking pieces of DNA together was discovered in 1975. Two leading biologists, Maxine Singer and Paul Berg, issued a call for a moratorium on all such experiments until the dangers could be carefully assessed. There were obvious dangers to public health, for example if genes for deadly toxins could be inserted into bacteria that are normally endemic in human populations. Biologists all over the world quickly agreed to the moratorium, and experiments were halted everywhere for ten months. During the ten months, two international conferences were held to work out the guidelines for permissible and forbidden experiments. The guidelines established rules of physical and biological containment for permitted experiments involving various degrees of risk. The most dangerous experiments were forbidden outright. These guidelines were adopted voluntarily by the biologists and have been observed ever since, with changes made from time to time in response to new discoveries. As a result, no serious health hazards have arisen from the experiments in twenty-five years. This is a shining example of responsible citizenship, showing that it is possible for scientists to protect the public from injury while preserving the freedom of science.
The history of biological weapons is a more complicated story. The United States, Britain, and the Soviet Union all had large programs to develop and stockpile biological weapons during and after the Second World War. But these were low-key efforts compared with the programs to develop nuclear weapons. Unlike the well-known physicists who pushed the nuclear bomb programs ahead with great enthusiasm, the biologists never pushed hard for biological weapons. The great majority of biologists had nothing to do with weapons. The few biologists who were involved with the weapons program were mostly opposed to it.
The strongest of the opponents in the United States was Matthew Meselson, who had the good luck to be a neighbor and friend of Henry Kissinger in 1968 when Nixon became president. Kissinger became national security adviser to President Nixon. Meselson seized the opportunity to convince Kissinger, and Kissinger convinced Nixon, that the American biological weapons program was far more dangerous to the United States than to any possible enemy. On the one hand, it was difficult to imagine any circumstances in which the United States would wish to use these weapons, and on the other hand, it was easy to imagine circumstances in which some of the weapons could fall into the hands of terrorists.
So Nixon in 1969 boldly declared that the United States was dismantling the entire program and destroying the stockpile of weapons. This was a unilateral move, not requiring any international agreement or ratification by the American Senate. The development of weapons was duly stopped and the weapons were destroyed. Britain quickly followed suit. In 1972, as a result of Nixon’s initiative, an international convention was signed by the US, the UK, and the USSR, imposing a permanent prohibition of biological weapons on all three countries. Many other countries subsequently signed the convention.
As we now know, the Soviet Union violated the Biological Weapons Convention of 1972 on an extensive scale, continuing to develop new weapons and to accumulate stockpiles until its collapse in 1991. After the collapse, Russia declared its adherence to the convention and announced that the Soviet program had now finally been stopped. But many of the old Soviet research and production centers remain hidden behind walls of secrecy, and Russia has never provided the world with convincing evidence that the program is not continuing. It is quite possible that stockpiles of biological weapons continue to exist in Russia and in other countries. Nevertheless, the 1972 convention remains legally in force and most countries have signed it. Even if the convention is unverifiable and even if it is violated, we are far better off with it than without it. Without the convention, we would not have any legal ground for complaint or for preventive action, whenever a biological weapons program anywhere in the world is discovered. With the convention, the danger of biological weapons is not eliminated but it is significantly reduced. Again, biologists in general and Matthew Meselson in particular deserve credit for making this happen in the real world of national politics and international rivalries.
The last part of my reply to Bill Joy concerns remedies for the dangers that we all agree exist. Bill says, “Internationalize control of knowledge,” and “Relinquish pursuit of that knowledge…so dangerous that we judge it better that [it] never be available.” Bill is advocating censorship of scientific inquiry, either by international or national authorities. I am opposed to this kind of censorship. It is often said that the risks of modern biotechnology are historically unparalleled because the consequences of letting a new living creature loose in the world may be irreversible. I think we can find a good historical parallel where a government was trying to guard against dangers that were equally irreversible.
Three hundred and fifty-nine years ago, the poet John Milton wrote a speech with the title Areopagitica, addressed to the Parliament of England. He was arguing for the liberty of unlicensed printing. I am suggesting that there is an analogy between the seventeenth-century fear of moral contagion by soul-corrupting books and the twenty-first-century fear of physical contagion by pathogenic microbes. In both cases, the fear was neither groundless nor unreasonable. In 1644, when Milton was writing, England had just emerged from a long and bloody civil war, and the Thirty Years’ War, which devastated Germany, had four years still to run. These seventeenth- century wars were religious wars, in which differences of doctrine played a great part. In that century, books not only corrupted souls but also mangled bodies. The risks of letting books go free into the world were rightly regarded by the English Parliament as potentially lethal as well as irreversible. Milton argued that the risks must nevertheless be accepted. I believe his message may still have value for our own times, if the word “book” is replaced by the word “experiment.” Here is Milton:
I deny not, but that it is of greatest concernment in the Church and Commonwealth, to have a vigilant eye how books demean themselves as well as men; and thereafter to confine, imprison, and do sharpest justice on them as malefactors…. I know they are as lively, and as vigorously productive, as those fabulous dragon’s teeth; and being sown up and down, may chance to spring up armed men.
The important word in Milton’s statement is “thereafter.” Books should not be convicted and imprisoned until after they have done some damage. What Milton declared unacceptable was prior censorship, prohibiting books from ever seeing the light of day. Next, Milton comes to the heart of the matter, the difficulty of regulating “things, uncertainly and yet equally working to good and to evil”:
Suppose we could expel sin by this means; look how much we thus expel of sin, so much we expel of virtue: for the matter of them both is the same; remove that, and ye remove them both alike.
This justifies the high providence of God, who, though he commands us temperance, justice, continence, yet pours out before us even to a profuseness all desirable things, and gives us minds that can wander beyond all limit and satiety. Why should we then affect a rigor contrary to the manner of God and of nature, by abridging or scanting those means, which books freely permitted are, both to the trial of virtue, and the exercise of truth. It would be better done to learn that the law must needs be frivolous which goes to restrain things, uncertainly and yet equally working to good, and to evil.
My last quotation expresses Milton’s patriotic pride in the intellectual vitality of seventeenth-century England, a pride that twenty-first-century Americans have good reason to share:
Lords and Commons of England, consider what Nation it is whereof ye are, and whereof ye are the governors: a Nation not slow and dull, but of a quick, ingenious, and piercing spirit, acute to invent, subtle and sinewy to discourse, not beneath the reach of any point the highest that human capacity can soar to…. Nor is it for nothing that the grave and frugal Transylvanian sends out yearly from as far as the mountainous borders of Russia, and beyond the Hercynian wilderness, not their youth, but their staid men, to learn our language, and our theologic arts.
Perhaps, after all, as we struggle to deal with the enduring problems of reconciling individual freedom with public safety, the wisdom of a great poet who died more than three hundred years ago may still be helpful.
That was the end of the debate. No vote was taken to determine who won. The purpose of the debate was not to win but to educate. Bill Joy and I remain friends.
February 13, 2003