Nine months later, in January 2001, the annual meeting of the World Economic Forum was held in Davos, Switzerland. Most of the people at the Forum are captains of industry, presidents of foundations, or government officials. But in 2001 they decided to invite some scientists and writers and artists to add some intellectual sparkle to the meeting. Bill Joy and I were both invited and asked to debate the question: Is our technology out of control? Bill Joy was taking an extreme position on the precautionary side, and I was asked to take an extreme position on the libertarian side, to make the debate interesting. In what follows I shall summarize our debate. To be sure that I am not misrepresenting Bill Joy, I quote here only from his published writings.
My first quote is from Bill’s article in Wired:
The 21st-century technologies—genetics, nanotechnology, and robotics (GNR)—are so powerful that they can spawn whole new classes of accidents and abuses. Most dangerously, for the first time, these accidents and abuses are widely within the reach of individuals or small groups. They will not require large facilities or rare raw materials. Knowledge alone will enable the use of them.
Thus we have the possibility not just of weapons of mass destruction but of knowledge-enabled mass destruction (KMD), this destructiveness hugely amplified by the power of self-replication.
I think it is no exaggeration to say we are on the cusp of the further perfection of extreme evil, an evil whose possibility spreads well beyond that which weapons of mass destruction bequeathed to the nation-states, on to a surprising and terrible empowerment of extreme individuals.
This was written a year and a half before the events of September 2001. I don’t know whether Bill at that time had Osama bin Laden in mind. He certainly had in mind the possibility of a Unabomber taking his revenge on society with genetically engineered microbes rather than with chemical explosives.
Second quote. Here Bill is quoting Eric Drexler, the chief prophet of nanotechnology. Drexler set up the Foresight Institute to promote the benign uses of nanotechnology and to warn against the dangerous uses. Here is Drexler:
Tough omnivorous [synthetic] “bacteria” could out-compete real bacteria: They could spread like blowing pollen, replicate swiftly, and reduce the biosphere to dust in a matter of days. Dangerous replicators could easily be too tough, small, and rapidly spreading to stop—at least if we make no preparation. We have trouble enough controlling viruses and fruit flies….
We cannot afford certain kinds of accidents with replicating assemblers.
The idea of nanotechnology is to build machines on a tiny scale that are as capable as living cells, but made of different materials so that they are more rugged and more versatile. One kind of nanomachine is the assembler, which is a tiny factory that can manufacture other machines, including replicas of itself. Drexler understood from the beginning that a replicating assembler would be a tool of immense power for good or for evil. Fortunately or unfortunately, nanotechnology has moved more slowly than Drexler expected. Nothing remotely resembling an assembler has yet emerged. The most useful products of nanotechnology so far are computer chips. They have no capacity for replicating either themselves or anything else.
My last quote from Bill Joy comes from an article he published in The Washington Post, summing up the dangers that he foresees and recommending a program of action to avoid them:
We who are involved in advancing the new technologies must devote our best efforts to heading off disaster. I offer here a list of first steps suggested by our history with weapons of mass destruction:
(1) Have scientists and technologists (and corporate leaders as well) take a vow, along the lines of the Hippocratic Oath, to avoid work on potential and actual weapons of mass destruction….
(2) Create an international body to publicly examine the dangers and ethical issues of new technology….
(3) Use stricter notions of liability, forcing companies to take responsibility for consequences through a private-sector mechanism—insurance….
(4) Internationalize control of knowledge and technologies that have great potential but are judged too dangerous to be made commercially available….
(5) Relinquish pursuit of that knowledge and development of those technologies so dangerous that we judge it better that they never be available. I too believe in the pursuit of knowledge and development of technologies; yet, we already have seen cases, such as biological weapons, where relinquishment is the obvious wise choice.
Next comes my response to Bill Joy. I agreed that the dangers he described are real, but I disagreed with some details of his argument, and I disagreed strongly with his remedies. I began by speaking about the history of biological weapons and gene-splicing experiments, and the successes and failures of efforts to regulate them. Bill Joy ignores the long history of effective action by the international biological community to regulate and prohibit dangerous technologies. Gene-splicing experiments began in many countries when the technique of sticking pieces of DNA together was discovered in 1975. Two leading biologists, Maxine Singer and Paul Berg, issued a call for a moratorium on all such experiments until the dangers could be carefully assessed. There were obvious dangers to public health, for example if genes for deadly toxins could be inserted into bacteria that are normally endemic in human populations. Biologists all over the world quickly agreed to the moratorium, and experiments were halted everywhere for ten months. During the ten months, two international conferences were held to work out the guidelines for permissible and forbidden experiments. The guidelines established rules of physical and biological containment for permitted experiments involving various degrees of risk. The most dangerous experiments were forbidden outright. These guidelines were adopted voluntarily by the biologists and have been observed ever since, with changes made from time to time in response to new discoveries. As a result, no serious health hazards have arisen from the experiments in twenty-five years. This is a shining example of responsible citizenship, showing that it is possible for scientists to protect the public from injury while preserving the freedom of science.
The history of biological weapons is a more complicated story. The United States, Britain, and the Soviet Union all had large programs to develop and stockpile biological weapons during and after the Second World War. But these were low-key efforts compared with the programs to develop nuclear weapons. Unlike the well-known physicists who pushed the nuclear bomb programs ahead with great enthusiasm, the biologists never pushed hard for biological weapons. The great majority of biologists had nothing to do with weapons. The few biologists who were involved with the weapons program were mostly opposed to it.
The strongest of the opponents in the United States was Matthew Meselson, who had the good luck to be a neighbor and friend of Henry Kissinger in 1968 when Nixon became president. Kissinger became national security adviser to President Nixon. Meselson seized the opportunity to convince Kissinger, and Kissinger convinced Nixon, that the American biological weapons program was far more dangerous to the United States than to any possible enemy. On the one hand, it was difficult to imagine any circumstances in which the United States would wish to use these weapons, and on the other hand, it was easy to imagine circumstances in which some of the weapons could fall into the hands of terrorists.
So Nixon in 1969 boldly declared that the United States was dismantling the entire program and destroying the stockpile of weapons. This was a unilateral move, not requiring any international agreement or ratification by the American Senate. The development of weapons was duly stopped and the weapons were destroyed. Britain quickly followed suit. In 1972, as a result of Nixon’s initiative, an international convention was signed by the US, the UK, and the USSR, imposing a permanent prohibition of biological weapons on all three countries. Many other countries subsequently signed the convention.
As we now know, the Soviet Union violated the Biological Weapons Convention of 1972 on an extensive scale, continuing to develop new weapons and to accumulate stockpiles until its collapse in 1991. After the collapse, Russia declared its adherence to the convention and announced that the Soviet program had now finally been stopped. But many of the old Soviet research and production centers remain hidden behind walls of secrecy, and Russia has never provided the world with convincing evidence that the program is not continuing. It is quite possible that stockpiles of biological weapons continue to exist in Russia and in other countries. Nevertheless, the 1972 convention remains legally in force and most countries have signed it. Even if the convention is unverifiable and even if it is violated, we are far better off with it than without it. Without the convention, we would not have any legal ground for complaint or for preventive action, whenever a biological weapons program anywhere in the world is discovered. With the convention, the danger of biological weapons is not eliminated but it is significantly reduced. Again, biologists in general and Matthew Meselson in particular deserve credit for making this happen in the real world of national politics and international rivalries.
The last part of my reply to Bill Joy concerns remedies for the dangers that we all agree exist. Bill says, “Internationalize control of knowledge,” and “Relinquish pursuit of that knowledge…so dangerous that we judge it better that [it] never be available.” Bill is advocating censorship of scientific inquiry, either by international or national authorities. I am opposed to this kind of censorship. It is often said that the risks of modern biotechnology are historically unparalleled because the consequences of letting a new living creature loose in the world may be irreversible. I think we can find a good historical parallel where a government was trying to guard against dangers that were equally irreversible.
Three hundred and fifty-nine years ago, the poet John Milton wrote a speech with the title Areopagitica, addressed to the Parliament of England. He was arguing for the liberty of unlicensed printing. I am suggesting that there is an analogy between the seventeenth-century fear of moral contagion by soul-corrupting books and the twenty-first-century fear of physical contagion by pathogenic microbes. In both cases, the fear was neither groundless nor unreasonable. In 1644, when Milton was writing, England had just emerged from a long and bloody civil war, and the Thirty Years’ War, which devastated Germany, had four years still to run. These seventeenth- century wars were religious wars, in which differences of doctrine played a great part. In that century, books not only corrupted souls but also mangled bodies. The risks of letting books go free into the world were rightly regarded by the English Parliament as potentially lethal as well as irreversible. Milton argued that the risks must nevertheless be accepted. I believe his message may still have value for our own times, if the word “book” is replaced by the word “experiment.” Here is Milton:
I deny not, but that it is of greatest concernment in the Church and Commonwealth, to have a vigilant eye how books demean themselves as well as men; and thereafter to confine, imprison, and do sharpest justice on them as malefactors…. I know they are as lively, and as vigorously productive, as those fabulous dragon’s teeth; and being sown up and down, may chance to spring up armed men.
The important word in Milton’s statement is “thereafter.” Books should not be convicted and imprisoned until after they have done some damage. What Milton declared unacceptable was prior censorship, prohibiting books from ever seeing the light of day. Next, Milton comes to the heart of the matter, the difficulty of regulating “things, uncertainly and yet equally working to good and to evil”:
Suppose we could expel sin by this means; look how much we thus expel of sin, so much we expel of virtue: for the matter of them both is the same; remove that, and ye remove them both alike.
This justifies the high providence of God, who, though he commands us temperance, justice, continence, yet pours out before us even to a profuseness all desirable things, and gives us minds that can wander beyond all limit and satiety. Why should we then affect a rigor contrary to the manner of God and of nature, by abridging or scanting those means, which books freely permitted are, both to the trial of virtue, and the exercise of truth. It would be better done to learn that the law must needs be frivolous which goes to restrain things, uncertainly and yet equally working to good, and to evil.
My last quotation expresses Milton’s patriotic pride in the intellectual vitality of seventeenth-century England, a pride that twenty-first-century Americans have good reason to share:
Lords and Commons of England, consider what Nation it is whereof ye are, and whereof ye are the governors: a Nation not slow and dull, but of a quick, ingenious, and piercing spirit, acute to invent, subtle and sinewy to discourse, not beneath the reach of any point the highest that human capacity can soar to…. Nor is it for nothing that the grave and frugal Transylvanian sends out yearly from as far as the mountainous borders of Russia, and beyond the Hercynian wilderness, not their youth, but their staid men, to learn our language, and our theologic arts.
Perhaps, after all, as we struggle to deal with the enduring problems of reconciling individual freedom with public safety, the wisdom of a great poet who died more than three hundred years ago may still be helpful.
That was the end of the debate. No vote was taken to determine who won. The purpose of the debate was not to win but to educate. Bill Joy and I remain friends.