BIG DATA, BIGGER DATA . . . TOO BIG

How the NSA Became Stupid

While we most of us were busy unwrapping presents on Christmas, The Wall Street Journal published an article on the NSA’s data collection. The point was not that it violated privacy – which is true – but that they were gathering far more than they could use. In short, it was also stupid.

The Journal cited William Binney, a retired, high ranking and long time NSA employee who worked on its computer code, speaking at a conference on privacy in Switzerland: “What they are doing is making themselves dysfunctional by taking all this data.”

The article went on to comment: “The agency is drowning in useless data, which harms its ability to conduct legitimate surveillance . . . Analysts are swamped with so much information that they can’t do their jobs effectively, and the enormous stockpile is an irresistible temptation for misuse.”

The article described efforts by Binney, Ed Loomis, head of a research center to monitor the data, and others, to sift through the vast troves of messages to find important and useful information. They “built a system to scrape data from the Internet, throw away the content about U.S. citizens and zoom in on the leftover metadata—or the ‘to’ and ‘from’ information in Internet traffic. They called it ThinThread.”

The agency scotched that plan, and went for one called “Trailblazer.” But, the Journal noted, “Trailblazer’s data-filtering system was never built, either. Instead, NSA officials secretly sought and won support for an array of programs to conduct warrantless wiretapping of phone and Internet content. They got similar approval to collect and analyze metadata from nearly every U.S. phone call and vast swaths of Internet traffic.” (See, “NSA Struggles to Make Sense of Flood of Surveillance Data.”)

The net result is the bloated, inefficient system we now have, one that succeeds in compromising constitutional safeguards against the invasion of privacy while failing to find the information we need to be safe. How to account for these failures?

On an individual level, it is a product of obsession, a heightened focus on one thing that leads to a kind of tunnel vision, ignoring context and meaning. In short, if your job is to collect secret data, you can’t know when enough is enough or when to stop, because your job is to get as much data as possible, not to think about what to do with it or why we need it. And if you have real enemies, it easily becomes a paranoid obsession. Then you really can’t stop, because you all too easily succumb to the illusion that one more bit of information will make you safe.

On a group or organizational level, it means lack of leadership or oversight. The technical functions and the technology required to succeed exist in a silo or a vacuum. It is similar to what happened to the banks during the credit crisis when ambitious rogue traders were generating huge profits by manipulating algorithms that their managers did not understand. Their job was just to make money, as much money as possible, while the managers, mesmerized by the profits and afraid to show their ignorance, stood by and let it happen – until the system blew up.

How smart do you have to be in order not to be too smart?