The Puzzle Palace: A Report on America's Most Secret Agency
“Brute force” is a phrase which turns up often in James Bamford’s useful investigation of the National Security Agency, the largest but least known of American intelligence services. As used by NSA officials, “brute force” refers to the method of last resort in cracking secret codes. The best method is to obtain the key to the code by stealing it, buying it, or figuring it out, but that is often impossible. A “brute force” attack simply tries out all the possible keys to a code identified through intercepted messages.
These possibilities can be very numerous. A commercial cipher with a fifty-six-bit key devised by IBM in the mid-1970s, for example, would present an inquisitive outsider with about seventy quadrillion possibilities. Large as that number is, a computer could be built with speed and capacity enough to try out every last one of them in less than a day. This helps to explain why the NSA’s headquarters at Fort Meade, Maryland, is the computer capital of the world. Lt. Gen. Marshall S. Carter, director of the NSA (DIRNSA) for just over four years between 1965 and 1969, told Bamford he was in charge of five and a half acres of computers. Another NSA official told him that computer acreage is now about double what it was in Carter’s day.
It is the scale of the NSA that impresses. According to Bamford, the NSA’s main building at Fort Meade contains about 1.9 million square feet of floor space, roughly equal to the CIA’s headquarters in Langley, Virginia, and the Capitol building in Washington combined. It has employed as many as 95,000 people, classifies 50 million to 100 million documents a year, produces 40 tons of classified waste per day, has up to $1 billion in contracts out at any given time, and has an annual budget of perhaps $10 billion.
Making, breaking, and protecting codes is only part of the work conducted by this vast establishment, and the smaller part at that. The larger part is collecting and analysing COMINT, SIGINT, and ELINT. COMINT and SIGINT are communications and signals intelligence—messages of every conceivable type, from ordinary commercial cables to Soviet naval communications to ships at sea. When the Soviet space capsule Soyuz I ran into trouble during reentry in April 1967, technicians at an NSA listening station in Turkey taped the whole awful event, from the first discussion of problems with the parachute through the cosmonaut’s final farewell to his wife and a terminal scream when the capsule burned up as it plunged into the earth’s atmosphere. Also included under the general heading of SIGINT are radio transmissions of a technical nature such as radar or telemetry broadcast by Soviet missiles during test flights, the principal source of American intelligence about the performance of Soviet strategic weapons. ELINT is electronics intelligence, defined as electromagnetic radiators with a non-atomic origin. (HUMINT—human intelligence collected by traditional espionage—is the province of the CIA.)
The size of the NSA budget is partly explained by the huge technical apparatus required to collect so many radio transmissions of so many different types. Here, too, the approach is one of brute force. The NSA attempts to collect all Soviet transmissions—the full daily broadcast of every conventional radio station in all the Soviet republics, every transmission to every Soviet embassy abroad, every broadcast to a ship at sea, every transmission by military units on maneuvers in Eastern Europe, the radio traffic of every control tower at Soviet airports, the radar signature of every Soviet system. (If B-52s or a new bomber are to penetrate Soviet air space in the event of war, for example, they have got to know where Soviet air defenses are located. The NSA helps to provide the maps.)
Even when the transmissions can’t be read, which is often the case, the volume of traffic itself is analysed for whatever it might reveal Shortly before the Berlin Wall was built in 1961, it is said, the sheer number of radio messages and the level of the codes in which they were encrypted should have indicated that something was up. In August 1968 Soviet military units on the periphery of Czechoslovakia were “lost” for several days. Apparently this was achieved by sophisticated jamming which disguised actual Soviet military radio traffic. Richard Helms, then the director of Central Intelligence and nominally in charge of the NSA, confessed to the President’s Foreign Intelligence Advisory Board that he was much embarrassed by the failure, but assured them the intelligence community would have done better if the Soviets had been heading West.
Monitoring Soviet radio transmissions is only half the work. Since it’s always possible that a code might be cracked in future, the unread transmissions—all the unread transmissions—are stored, more or less forever, on magnetic tape.
The National Security Agency was established by President Truman—secretly: its official charter has never been made public—in the fall of 1952, but its true origins go back to the First World War when a State Department code clerk, Herbert O. Yardley, tried his hand at a secret message addressed to Woodrow Wilson in May 1916. He solved it within a couple of hours and immediately concluded that the British, who controlled the eastern terminal of transatlantic cables to North America, were reading the US government’s most secret diplomatic traffic. When the United States entered the war Yardley transferred to the military, founded a Code and Cipher Solution Subsection in the office devoted to military intelligence, and at the war’s end was arranging liaison with the French Chambre noire, or “black chamber,” a name Yardley borrowed when he was appointed to head a permanent code-breaking office in the War Department in 1919.
Yardley’s initial budget was $100,000, with which he hired a staff of fifty at salaries ranging from $1,200 per year for clerks up to $3,000 for senior code breakers. But the end of the war presented Yardley with two major problems. The first was the end of official censorship, which meant he no longer had automatic access to international cable traffic. Indeed, it was now against federal law to intercept messages, but Yardley quietly arranged with the management of Western Union and Postal Telegraph, the two major international carriers, for the Black Chamber to temporarily “borrow” messages. The two companies agreed to this illegal arrangement in the interest of national security, a precedent that was to be enduring.
Yardley’s major coup during this period was cracking the Japanese diplomatic code, which allowed the US to read Tokyo’s fallback position during the negotiations in 1921 to establish a fixed ratio of capital ships among the navies of Britain, the United States, and Japan. Knowing that Japanese negotiators had been instructed to accept the 10 to 6 ratio pressed by Britain and the United States, the Western team simply sat tight until the Japanese gave in.
But this was the high point of the Black Chamber and of Yardley’s career as well. His second major problem in the immediate years after World War I—indifference at high levels—was never overcome. It is war and the threat of war that turn a government’s thoughts to espionage. The Black Chamber’s budget was gradually reduced to $25,000 a year, most of it provided by the State Department. When Herbert Hoover’s secretary of state, Henry L. Stimson, took office in March 1929, Yardley hesitated to brief him on the Black Chamber’s work. When he finally did so a few months later, Stimson was appalled by the whole undertaking and ordered it to cease immediately with words about as close to immortal as any ever uttered by an American statesman: “Gentlemen do not read each other’s mail.”
But the end of the Black Chamber and Yardley’s departure from government did not end American code-breaking efforts. The War Department established a Signal Intelligence Service in 1930 which limped along on tiny budgets (never more than $17,400) until 1937, when the obvious approach of war brought high-level interest and funds to match. By September 1939 the staff of the SIS had grown from seven to nineteen. By December 7, 1941, it had reached 331. (The SIS had even cracked the new Japanese diplomatic code, called “Purple,” and had intercepted a message that indicated war was near, but a comedy of errors delayed a warning to the US commander in Hawaii until several hours after the attack on Pearl Harbor had begun.)
By the war’s end the SIS staff exceeded 10,000 and its official duties had vastly expanded to include the collection and analysis of all sorts of communications and signals intelligence. In 1949 the various communications intelligence branches were combined and named the Armed Forces Security Agency; three years later—largely as the result of high-level dissatisfaction with the AFSA’s performance during the Korean War, as well as weariness with the service rivalries which drove the White House half crazy during the postwar years—the entire process of reorganization was repeated. In its present form the National Security Agency not only is bigger than the postal, telephone, and telegraph services (known as PT&Ts) of most major nations, but it draws heavily on the services of the Army, the Navy, and the Air Force as well.
James Bamford has assembled all that was known, and much that was unknown, in his history of the NSA, but the result does not make for light reading. The chapter about Yardley and a handful of stories scattered throughout the rest of the text provide the only narrative. For the rest, his book reads like a study of AT&T, with methodical lists of its directors and deputy directors, the divisions and subdivisions of the NSA, and the listening stations of various kinds scattered about the world. A great many officials are identified by name and job title. Much hardware is paraded across the page, from elaborate computer retrieval systems to giant dish antennae. Much of this material is new, gleaned from 6,000 pages of NSA newsletters and Justice Department documents that Bamford obtained under the Freedom of Information Act (and which the government, a year ago, attempted to retrieve with the claim they had been released in error). Bamford has also conducted several revealing interviews, especially with General Carter and Francis Raven, formerly in charge of reading Soviet and, later, third-world communications traffic.
Bamford deserves special praise for largely avoiding the mocking, ironic, superior tone adopted by many journalists when writing about the intelligence and defense communities. These virtues add up to a considerable achievement. But that should not obscure the fact that the secrecy surrounding the NSA is still largely intact. Bamford has mapped the landscape, much as David Wise and Thomas B. Ross did in 1965 with their similar trail-breaking study of the CIA, The Invisible Government. Future researchers will owe Bamford a considerable debt, but they will also have plenty to do.
They might well begin with a knotty subject that Bamford never treats at length—the NSA’s contribution to the national security. This is hard to gauge. Investigation of the intelligence community by House and Senate select committees in the mid-1970s for the most part mentioned only the failures and “excesses,” like NSA’s Operation Minaret, which maintained watch lists of about 1,680 Americans for the Bureau of Narcotics and Dangerous Drugs (BNDD), the FBI, the Secret Service, the CIA, and the Defense Intelligence Agency. Between 1967 and 1973 the NSA distributed nearly four thousand reports concerning these Americans, but it would be hard to say what practical use they served. One director told the Senate select committee that NSA’s monitoring of international phone calls and cable traffic had helped the BNDD intercept some large drug shipments and had prevented “a major terrorist act,” presumably a Palestinian attack on American Jews. Another, much larger NSA program, Operation Shamrock, intercepted just about all cable traffic entering and leaving the United States between 1945 and 1975.
This was a very large undertaking indeed. Presumably it gave the NSA access to all diplomatic cable traffic with the exception of messages hand-carried by courier. How many of these messages could have been read by the NSA is not known, but it must have been a large number. Were they useful? Bamford doesn’t know, and I don’t either. Officials were doubtless glad to have them, but there is no public evidence that reading other countries’ secret messages has really served the American national interest. There is also no evidence that the counterintelligence arms of the FBI and the CIA were aided by NSA interceptions, or what, if anything, was done with the staggering quantities of information that must have been obtained relating to international trade in oil, grain, or high-technology equipment.
My guess is that the NSA’s interception of telephone and cable messages picked up something about everything, that a lot of it fell into a nice-to-know category, and that very little of it was both critical in importance and unobtainable by other means. Whether that little is worth what it costs is impossible for an outsider to say. Officials in the national security community clearly think that it is. The only public fruits of the NSA’s work are the seven thick volumes of translations published daily by the Foreign Broadcast Information Service. These are consulted by scholars and occasionally by journalists. At the moment, for example, they are a principal source of information about Iran.
There can be no question about the utility of the NSA’s technical collection of radio signals related to military matters. It is well known that the verification of arms agreements, as well as general intelligence about Soviet military programs, depend all but exclusively on “national technical means,” most of which are operated by the NSA. (Reconnaissance satellites are run by the Air Force under the direction of the National Reconnaissance Office. The contribution from spies run by the CIA is minuscule.) But that is only part of it. If the United States and the Soviet Union ever fight a big general war, all aspects of that war will involve intelligence of the sort collected by the NSA, from the first shots or missile launches until the final armistice or exhausted silence. The “winner”—we will not try here to settle whether there can be one—will very likely be the side with the most enduring system of “C3I,” or command, control, communications, and intelligence. This is certainly the Pentagon’s belief and it explains why the Reagan administration is planning to spend $18 billion on C3I over the next few years, most of it for ways to maintain or replace facilities destroyed—“stressed” in the Pentagon’s term—in the course of the war.
Allied success in cracking German and Japanese codes during World War II is now well known, and the NSA’s acres of computers are evidence that it hopes to do the same. But code breaking is no longer the primary focus of the intelligence part of C3I. Modern armies emit a continual buzz and hum of radio messages and signals. The fire-control and target-finding devices of tanks and fighter planes, for example, give them away in the very process of aiding the tanks and planes to attack or to defend themselves from an enemy. Even before the ignition of a missile’s rocket mortar could be picked up by infrared sensors on a satellite in geosychronous orbit high overhead, the launch order would have been suggested through radio signals. Firing a missile is not like throwing a light switch. The preparation to launch involves a blizzard of microwave transmissions.
It is the same with conventional forces. An army requires an intricate command web linked by radio. Mapping that web reveals where everybody is and what he’s doing. But figuring these things out—finding the ways in which military units, and even weapons themselves, talk to each other—can’t wait until the war begins. It has to be done in advance. This is where the NSA comes in, and why its ELINT planes and SIGINT ships have so often been attacked by target countries. The 1950s were the heyday of “spoofing”—deliberate intrusions of Soviet airspace, sometimes by formations of bombers, in order to trigger radar defenses so that they could be monitored, mapped, and identified. Several US aircraft were shot down during these provocative and dangerous exercises. In 1964 the US destroyers Maddox and Turner Joy were on SIGINT missions in the Tonkin Gulf where they were apparently attacked by North Vietnamese patrol boats. The Pueblo, captured by North Korea in 1968, was on a similar mission.
Perhaps the most notorious of these incidents occurred in June 1967, when Israeli aircraft attacked the SIGINT ship Liberty cruising along the Mediterranean coast of the Sinai Peninsula. The territorial limit claimed by Egypt was twelve miles, by Israel six. The Liberty, decks crowded with electronic gear, scrupulously remained in international waters. Why was she crowding in on the battle zone? “Somebody wanted to listen to some close tactical program,” Raven told Bamford, “or communications or something which nobody in the world gave a damn about….”
On the morning of the third day of the war, June 8, Israeli aircraft repeatedly buzzed the ship but made no attempt to contact her by radio. At 2 PM the aircraft returned and attacked the Liberty without warning of any kind, using rockets, cannon fire, and napalm. Eight US Navy men were killed outright. The ship was set afire and punctured by 800 shell holes big enough to put a fist through. At 2:24 PM three Israeli torpedo boats appeared and renewed the attack. A forty-foot-wide hole was blown in the side of the Liberty. Lifeboats were machine-gunned. Another twenty-four men were killed and over a hundred wounded. At 4:10 PM the Israeli government reported the attack to the American embassy in Tel Aviv and apologized. The following day an explanation was offered. The Israelis had mistaken the Liberty (455 feet long) for the Egyptian coastal steamer El Quseir (275 feet long). The US government formally accepted the apology and the explanation, and in 1980, after long negotiations, finally received $6 million in compensation for the ship (which had cost more than $30 million).
It is not hard to understand why Israel insisted the attack was a mistake, or why Washington accepted the explanation at face value, but scholars and military experts who have studied the episode mostly take a different view. For them the hard question is not whether the attack was deliberate, but why the Israelis thought it necessary. Writing in the United States Naval Institute Proceedings of June 1978, the naval historian Richard K. Smith concluded that Israeli military authorities, hoping to launch an attack on the Golan Heights in Syria before a cease-fire could be imposed upon them, feared that the Liberty was scooping up enough battlefield radio traffic to see that Israel had already won the war. According to Smith (as quoted by Bamford), the Israelis wanted to maintain the confusions natural to a rapidly unfolding campaign—what Clausewitz called “the fog of war”—in order to retain freedom of action for another few but critical days.
In the absence of definite proof Smith’s view must be taken as only educated surmise. The point here is that the Liberty, like other “platforms” for collecting intelligence run by the NSA, really did have the capacity to gather in and process enough radio transmissions to give American authorities almost as good a view of the progress of the war, and at almost the same time, as could be obtained by the Israelis themselves. The utility of SIGINT is suggested by Israel’s alleged willingness to take such drastic action against its main, indeed its only important, ally. Outsiders have sometimes criticized the NSA for its “vacuum-cleaner” approach to collection, especially where the information sucked up includes the conversations and cables of American citizens. But it is precisely this omniverousness—the “brute force” scale of collecting—that makes the NSA worth $10 billion a year to American defense planners.
Like the rest of the intelligence community, the NSA is not very good at predicting things—especially large things which come as an unpleasant surprise. It failed to predict the outbreak of the Korean War, the Chinese crossing of the Yalu River, the building of the Berlin Wall, the Soviet emplacement of missiles in Cuba in 1962, the Tet offensive in 1968, or the Soviet invasion of Czechoslovakia later that year. This list might be much extended. But the truth is that intelligence services are best at more mundane tasks—finding things, counting things, and describing things. The NSA has helped to provide US officials with a comprehensive, reliable, and extremely detailed knowledge of Soviet strategic forces. It is probably this fact which explains the willingness of Admiral Noel Gayler, the NSA director between 1969 and 1972, to support publicly the campaign to freeze nuclear weapons at current levels, rather than build the vast new systems which the Reagan administration claims are needed to “catch up” with the Russians. The focus of the NSA since its birth has been the Soviet Union. Occasionally it has gathered juicy political information—the conversations over radio-telephone of Soviet leaders traveling about Moscow in their limousines, for example. One ought not to slight such achievements. But the NSA’s main job has been on a much vaster scale—to paint a comprehensive electronic portait, a kind of wiring diagram, of the Soviet armed forces.
The Soviets conduct similar operations against the West. Among other things they maintain a world-wide fleet of fishing trawlers that double as SIGINT ships. One is stationed at all times, for example, near the US missile testing site at Vandenberg Air Force Base on the coast of California, and another waits downrange near Kwajalein Lagoon, the target area in the Marshall Islands. The Soviets also spare no pains to keep track of what the NSA is doing. A retired intelligence officer once asked me to remove from something I had written the names of four CIA technicians who had handled secret communications traffic at a US consulate in Africa during the 1960s. He told me the Russians went to great lengths to identify Americans who had anything to do with codes, tried to recruit or blackmail them, and at times had even physically attacked and injured them in the hope that a replacement would prove an easier target. The officer insisted that the four men I had identified might be in physical danger if I published their names. When I finally admitted to myself that this could really be so, I dropped the names.
From time to time Soviet efforts of this sort are successful, as suggested last July when the British arrested Geoffrey Arthur Prime, a Russian-language translator who worked for Britain’s equivalent of the NSA—Government Communications Headquarters—for nine years between 1968 and 1977. A New York Times story on October 24 reported that US intelligence officials were much worried about the Prime case because the NSA and GCH cooperate closely, and because Prime’s job as a translator put him, and the Russians, in an ideal position to determine which of their codes had been compromised. The officials also raised the possibility that the Russians had been systematically feeding false information to the West through messages they knew we could read. Intelligence officials do not usually tell reporters of such matters. The motive in this case was clearly to pressure the British into greater candor.
The damage caused by penetration of this sort in peacetime can mostly be repaired; in wartime—or in the sort of crisis situations that can lead to war—it could be nothing short of a disaster. If the Germans had ever learned that the British at Bletchley Park were reading the daily location reports of submarines in the North Atlantic during World War II, to give only one example, the campaign to starve Britain might have gone the other way. It is in the nature of intelligence work that something very large, like the NSA, can be seriously injured by something very small—the compromise of even a single troubled employee (such as Prime), or the acquisition of a briefcase full of documents. The obsession with secrecy that follows from this fact helps to explain why the NSA is so unhappy about the publication of Bamford’s book.
Frederick the Great of Prussia once said that an ordinary citizen should never know his country was at war. American policy since World War II has been to go Frederick one better, anticipating nothing but trouble from kibitzers. For thirty years the NSA, despite its size, remained happily in shadow. Bamford has given us our first good, clear look at it, and what we see confirms the impression—to my mind, at least—that the principal undertaking of our time is the preparation for war.