Data and Goliath The Hidden Battles to Collect Your Data and Control Your World

PART THREE

WHAT TO DO ABOUT IT

Principles

The harms from mass surveillance are many, and the costs to individuals and society as a whole disproportionately outweigh the benefits. We can and must do something to rein it in. Before offering specific legal, technical, and social proposals, I want to start this section with some general principles. These are universal truths about surveillance and how we should deal with it that apply to both governments and corporations.

Articulating principles is the easy part. It’s far more difficult to apply them in specific circumstances. “Life, liberty, and the pursuit of happiness” are principles we all agree on, but we only need to look at Washington, DC, to see how difficult it can be to apply them. I’ve been on many panels and debates where people on all sides of this issue agree on general principles about data collection, surveillance, oversight, security, and privacy, even though they disagree vehemently on how to apply those principles to the world at hand.

SECURITY AND PRIVACY

Often the debate is characterized as “security versus privacy.” This simplistic view requires us to make some kind of fundamental trade-off between the two: in order to become secure, we must sacrifice our privacy and subject ourselves to surveillance. And if we want some level of privacy, we must recognize that we must sacrifice some security in order to get it.

It’s a false trade-off. First, some security measures require people to give up privacy, but others don’t impinge on privacy at all: door locks, tall fences, guards, reinforced cockpit doors on airplanes. And second, privacy and security are fundamentally aligned. When we have no privacy, we feel exposed and vulnerable; we feel less secure. Similarly, if our personal spaces and records are not secure, we have less privacy. The Fourth Amendment of the US Constitution talks about “the right of the people to be secure in their persons, houses, papers, and effects” (italics mine). Its authors recognized that privacy is fundamental to the security of the individual.

Framing the conversation as trading security for privacy leads to lopsided evaluations. Often, the trade-off is presented in terms of monetary cost: “How much would you pay for privacy?” or “How much would you pay for security?” But that’s a false trade-off, too. The costs of insecurity are real and visceral, even in the abstract; the costs of privacy loss are nebulous in the abstract, and only become tangible when someone is faced with their aftereffects. This is why we undervalue privacy when we have it, and only recognize its true value when we don’t. This is also why we often hear that no one wants to pay for privacy and that therefore security trumps privacy absolutely.

12

When the security versus privacy trade-off is framed as a life-and-death choice, all rational debate ends. How can anyone talk about privacy when lives are at stake? People who are scared will more readily sacrifice privacy in order to feel safer. This explains why the US government was given such free rein to conduct mass surveillance after 9/11. The government basically said that we all had to give up our privacy in exchange for security; most of us didn’t know better, and thus accepted the Faustian bargain.

The problem is that the entire weight of insecurity is compared with the incremental invasion of privacy. US courts do this a lot, saying things on the order of, “We agree that there is a loss of privacy at stake in this or that government program, but the risk of a nuclear bomb going off in New York is just too great.” That’s a sloppy characterization of the trade-off. It’s not the case that a nuclear detonation is impossible if we surveil, or inevitable if we don’t. The probability is already very small, and the theoretical privacyinvading security program being considered could only reduce that number very slightly. That’s the trade-off that needs to be considered.

More generally, our goal shouldn’t be to find an acceptable trade-off between security and privacy, because we can and should maintain both together.

SECURITY OVER SURVEILLANCE

Security and surveillance are conflicting design requirements. A system built for security is harder to surveil. Conversely, a system built for easy surveillance is harder to secure. A built-in surveillance capability in a system is insecure, because we don’t know how to build a system that only permits surveillance by the right sort of people. We saw this in Chapter 11.

We need to recognize that, to society as a whole, security is more critical than surveillance. That is, we need to choose a secure information infrastructure that inhibits surveillance instead of an insecure infrastructure that allows for easy surveillance.

The reasoning applies generally. Our infrastructure can be used for both good and bad purposes. Bank robbers drive on highways, use electricity, shop at hardware stores, and eat at all-night restaurants, just like honest people. Innocents and criminals alike use cell phones, e-mail, and Dropbox. It rains on the just and the unjust.

Despite this, society continues to function, because the honest, positive, and beneficial uses of our infrastructure far outweigh the dishonest, negative, and harmful ones. The percentage of the drivers on our highways who are bank robbers is negligible, as is the percentage of e-mail users who are criminals. It makes far more sense to design all of these systems for the majority of us who need security from criminals, telemarketers, and sometimes our own governments.

By prioritizing security, we would be protecting the world’s information flows— including our own—from eavesdropping as well as more damaging attacks like theft and destruction. We would protect our information flows from governments, non-state actors, and criminals. We would be making the world safer overall.

Tor is an excellent example. It’s free open-source software that you can use to browse anonymously on the Internet. First developed with funding from the US Naval Research Laboratory and then from the State Department, it’s used by dissidents all over the world to evade surveillance and censorship. Of course, it’s also used by criminals for the same purpose. Tor’s developers are constantly updating the program to evade the Chinese government’s attempts to ban it. We know that the NSA is continually trying to break it, and—at least as of a 2007 NSA document disclosed by Snowden—has been unsuccessful. We know that the FBI was hacking into computers in 2013 and 2014 because it couldn’t break Tor. At the same time, we believe that individuals who work at both the NSA and the GCHQ are anonymously helping keep Tor secure. But this is the quandary: Tor is either strong enough to protect the anonymity of both those we like and those we don’t like, or it’s not strong enough to protect the anonymity of either.

Of course, there will never be a future in which no one spies. That’s naïve. Governments have always spied, since the beginning of history; there are even a few spy stories in the Old Testament. The question is which sort of world we want to move towards. Do we want to reduce power imbalances by limiting government’s abilities to monitor, censor, and control? Or do we allow governments to have increasingly more power over us?

“Security over surveillance” isn’t an absolute rule, of course. There are times when it’s necessary to design a system for protection from the minority of us who are dishonest. Airplane security is an example of that. The number of terrorists flying on planes is negligible compared with the number of nonterrorists, yet we design entire airports around those few, because a failure of security on an airplane is catastrophically more deadly than a terrorist bomb just about anywhere else. We don’t (yet) design our entire society around terrorism prevention, though.

There are also times when we need to design appropriate surveillance into systems. We want shipping services to be able to track packages in real time. We want first responders to know where an emergency cell phone call is coming from. We don’t use the word “surveillance” in these cases, of course; we use some less emotionally laden term like “package tracking.”

The general principle here is that systems should be designed with the minimum surveillance necessary for them to function, and where surveillance is required they should gather the minimum necessary amount of information and retain it for the shortest time possible.

TRANSPARENCY

Transparency is vital to any open and free society. Open government laws and freedom of information laws let citizens know what the government is doing, and enable them to carry out their democratic duty to oversee its activities. Corporate disclosure laws perform similar functions in the private sphere. Of course, both corporations and governments have some need for secrecy, but the more they can be open, the more we can knowledgeably decide whether to trust them. Right now in the US, we have strong open government and freedom of information laws, but far too much information is exempted from them.

For personal data, transparency is pretty straightforward: people should be entitled to know what data is being collected about them, what data is being archived about them, and how data about them is being used—and by whom. And in a world that combines an international Internet with country-specific laws about surveillance and control, we need to know where data about us is being stored. We are much more likely to be comfortable with surveillance at any level if we know these things. Privacy policies should provide this information, instead of being so long and deliberately obfuscating that they shed little light. We also need transparency in the algorithms that judge us on the basis of our data, either by publishing the code or by explaining how they work. Right now, we cannot judge the fairness of TSA algorithms that select some of us for “special screening.” Nor can we judge the IRS’s algorithms that select some of us for auditing. It’s the same with search engine algorithms that determine what Internet pages we see, predictive policing algorithms that decide whom to bring in for questioning and what neighborhoods to patrol, or credit score algorithms that determine who gets a mortgage. Some of this secrecy is necessary so people don’t figure out how to game the system, but much of it is not. The EU Data Protection Directive already requires disclosure of much of this information. It may seem as if I am contradicting myself. On one hand, I am advocating for individual privacy over forced surveillance. On the other, I am advocating for government and corporate transparency over institutional secrecy. The reason I say yes to both lies in the existing power imbalance between people and institutions. Institutions naturally wield more power than people. Institutional secrecy increases institutional power, and that power differential grows. That’s inherently bad for personal liberty. Individual privacy increases individual power, thereby reducing that power differential. That’s good for liberty. It’s exactly the same with transparency and surveillance. Institutional transparency reduces the power imbalance, and that’s good. Institutional surveillance of individuals increases the power imbalance, and that’s bad. Transparency doesn’t come easily. The powerful do not like to be watched. For example, the police are increasingly averse to being monitored. All over the US, police harass and prosecute people who videotape them, and some jurisdictions have ruled it illegal. Cops in Chicago have deliberately obscured cameras, apparently attempting to conceal their own behavior. The San Diego Police Department denies all requests for police videos, claiming that they’re part of ongoing investigations. During the 2014 protests in Ferguson, Missouri, after the police killed an unarmed black man, police routinely prevented protesters from recording them, and several reporters were arrested for documenting events. Los Angeles police even went so far as to sabotage court-mandated voice recorders in their patrol cars. Governments and corporations routinely resist transparency laws of all kinds. But the world of secrecy is changing. Privacy-law scholar Peter Swire writes about a declining half-life of secrets. What he observed is that, in general, secrets get exposed sooner than they used to. Technology is making secrets harder to keep, and the nature of the Internet makes secrets much harder to keep long-term. The push of a “send” button can deliver

gigabytes across the Internet in a trice. A single thumb drive can hold more data every year. Both governments and organizations need to assume that their secrets are more likely to be exposed, and sooner, than ever before.

One of the effects of a shrinking half-life for secrets is that their disclosure is more damaging. One of Snowden’s documents indicated that the NSA spied on the cell phone of German chancellor Angela Merkel. The document is undated, but it’s obviously from the last few years. If that document had become public 20 years from now, the reaction in Germany would have been very different from the public uproar that occurred in 2013, when Merkel was still in office and the incident was current events rather than historical.

Cultural changes are also making secrets harder to keep. In the old days, guarding institutional secrets was part of a lifelong culture. The intelligence community would recruit people early in their careers and give them jobs for life. It was a private men’s club, one filled with code words and secret knowledge. The corporate world, too, was filled with lifers. Those days are gone. Many jobs in intelligence are now outsourced, and there is no job-for-life culture in the corporate world anymore. Workforces are flexible, jobs are outsourced, and people are expendable. Moving from employer to employer is now the norm. This means that secrets are shared with more people, and those people care less about them. Recall that five million people in the US have a security clearance, and that a majority of them are contractors rather than government employees.

There is also a greater belief in the value of openness, especially among younger people. Younger people are much more comfortable with sharing personal information than their elders. They believe that information wants to be free, and that security comes from public knowledge and debate. They have said very personal things online, and have had embarrassing photographs of themselves posted on social networking sites. They have been dumped by lovers in public online forums. They have overshared in the most compromising ways—and they survived intact. It is a tougher sell convincing this crowd that government secrecy trumps the public’s right to know.

These technological and social trends are a good thing. Whenever possible, we should strive for transparency.

OVERSIGHT AND ACCOUNTABILITY

In order for most societies to function, people must give others power over themselves. Ceding power is an inherently risky thing to do, and over the millennia we have developed a framework for protecting ourselves even as we do this: transparency, oversight, and accountability. If we know how people are using the power we give them, if we can assure ourselves that they’re not abusing it, and if we can punish them if they do, then we can more safely entrust them with power. This is the fundamental social contract of a democracy.

There are two levels of oversight. The first is strategic: are the rules we’re imposing the correct ones? For example, the NSA can implement its own procedures to ensure that it’s following the rules, but it should not get to decide what rules it should follow. This

was nicely explained by former NSA director Michael Hayden: “Give me the box you will allow me to operate in. I’m going to play to the very edges of that box… . You, the American people, through your elected representatives, give me the field of play and I will play very aggressively in it.” In one sense he’s right; it’s not his job to make the rules. But in another he’s wrong, and I’ll talk about that in Chapter 13.

In either case, we need to get much better at strategic oversight. We need more open debate about what limitations should be placed on government and government surveillance. We need legislatures conducting meaningful oversight and developing forward-looking responses. We also need open, independent courts enforcing laws rather than rubber-stamping agency practices, routine reporting of government actions, vibrant public-sector press and watchdog groups analyzing and debating the actions of those who wield power, and—yes—a legal framework for whistleblowing. And a public that cares. I’ll talk about that in Chapter 13, too.

The other kind of oversight is tactical: are the rules being followed? Mechanisms for this kind of oversight include procedures, audits, approvals, troubleshooting protocols, and so on. The NSA, for example, trains its analysts in the regulations governing their work, audits systems to ensure that those regulations are actually followed, and has instituted reporting and disciplinary procedures for occasions when they’re not.

Different organizations provide tactical oversight of one another. The warrant process is an example of this. Sure, we could trust the police forces to only conduct searches when they’re supposed to, but instead we require them to bring their requests before a neutral third party—a judge—who ensures they’re following the rules before issuing a court order.

The key to robust oversight is independence. This is why we’re always suspicious of internal reviews, even when they are conducted by a vigorous advocate. This was a key issue with the chief privacy officer at the DHS. I remember Mary Ellen Callahan, who held that job from 2009 to 2012. She was a great advocate for privacy, and recommended that the agency cancel several programs because of privacy concerns. But she reported to Janet Napolitano, the DHS secretary, and all she could do was make suggestions. If Callahan had been outside of DHS, she would have had more formal regulatory powers. Oversight is much better conducted by well-staffed and knowledgeable outside evaluators.

You can think of the difference between tactical and strategic oversight as the difference between doing things right and doing the right things. Both are required.

Neither kind of oversight works without accountability. Those entrusted with power can’t be free to abuse it with impunity; there must be penalties for abuse. Oversight without accountability means that nothing changes, as we’ve learned again and again. Or, as risk analyst Nassim Taleb points out, organizations are less likely to abuse their power when people have skin in the game.

It’s easy to say “transparency, oversight, and accountability,” but much harder to make those principles work in practice. Still, we have to try—and I’ll get to how to do that in the next chapter. These three things give us the confidence to trust powerful institutions. If

we’re going to give them power over us, we need reassurance that they will act in our interests and not abuse that power.

RESILIENT DESIGN

Designing for resilience is an important, almost philosophical, principle in systems architecture. Technological solutions are often presumed to be perfect. Yet, as we all know, perfection is impossible—people, organizations, and systems are inherently flawed. From government agencies to large multinational corporations, all organizations suffer imperfections.

These imperfections aren’t just the result of bad actors inside otherwise good systems. Imperfections can be mundane, run-of-the-mill, bureaucratic drift. One form of imperfection is mission creep. Another comes from people inside organizations focusing on the narrow needs of that organization and not on the broader implications of their actions. Imperfections also come from social change: changes in our values over time. Advancing technology adds new perturbations into existing systems, creating instabilities.

If systemic imperfections are inevitable, we have to accept them—in laws, in government institutions, in corporations, in individuals, in society. We have to design systems that expect them and can work despite them. If something is going to fail or break, we need it to fail in a predictable way. That’s resilience.

In systems design, resilience comes from a combination of elements: fault-tolerance, mitigation, redundancy, adaptability, recoverability, and survivability. It’s what we need in the complex and ever-changing threat landscape I’ve described in this book.

I am advocating for several flavors of resilience for both our systems of surveillance and our systems that control surveillance: resilience to hardware and software failure, resilience to technological innovation, resilience to political change, and resilience to coercion. An architecture of security provides resilience to changing political whims that might legitimize political surveillance. Multiple overlapping authorities provide resilience to coercive pressures. Properly written laws provide resilience to changing technological capabilities. Liberty provides resilience to authoritarianism. Of course, full resilience against any of these things, let alone all of them, is impossible. But we must do as well as we can, even to the point of assuming imperfections in our resilience.

ONE WORLD, ONE NETWORK, ONE ANSWER

Much of the current surveillance debate in the US is over the NSA’s authority, and whether limiting the NSA somehow empowers others. That’s the wrong debate. We don’t get to choose a world in which the Chinese, Russians, and Israelis will stop spying if the NSA does. What we have to decide is whether we want to develop an information infrastructure that is vulnerable to all attackers, or one that is secure for all users.

Since its formation in 1952, the NSA has been entrusted with dual missions. First, signals intelligence, or SIGINT, involved intercepting the communications of America’s enemies. Second, communications security, or COMSEC, involved protecting American military—and some government—communications from interception. It made sense to combine these two missions, because knowledge about how to eavesdrop is necessary to protect yourself from eavesdropping, and vice versa.

The two missions were complementary because different countries used different communications systems, and military personnel and civilians used different ones as well. But as I described in Chapter 5, that world is gone. Today, the NSA’s two missions are in conflict.

Laws might determine what methods of surveillance are legal, but technologies determine which are possible. When we consider what security technologies we should implement, we can’t just look at our own countries. We have to look at the world.

We cannot simultaneously weaken the enemy’s networks while still protecting our own. The same vulnerabilities used by intelligence agencies to spy on each other are used by criminals to steal your financial passwords. Because we all use the same products, technologies, protocols, and standards, we either make it easier for everyone to spy on everyone, or harder for anyone to spy on anyone. It’s liberty versus control, and we all rise and fall together. Jack Goldsmith, a Harvard law professor and former assistant attorney general under George W. Bush, wrote, “every offensive weapon is a (potential) chink in our defense—and vice versa.”

For example, the US CALEA law requires telephone switches to enable eavesdropping. We might be okay with giving police in the US that capability, because we generally trust the judicial warrant process and assume that the police won’t abuse their authority. But those telephone switches are sold worldwide—remember the story about cell phone wiretapping in Greece in Chapter 11—with that same technical eavesdropping capability. It’s our choice: either everyone gets that capability, or no one does.

It’s the same with IMSI-catchers that intercept cell phone calls and metadata. StingRay might have been the FBI’s secret, but the technology isn’t secret anymore. There are dozens of these devices scattered around Washington, DC, and the rest of the country run by who-knows-what government or organization. Criminal uses are next. By ensuring that the cell phone network is vulnerable to these devices so we can use them to solve crimes, we necessarily allow foreign governments and criminals to use them against us.

I gave more examples in Chapter 11. In general, we get to decide how we’re going to build our communications infrastructure: for security or not, for surveillance or not, for privacy or not, for resilience or not. And then everyone gets to use that infrastructure.

Solutions for Government

In the wake of Snowden’s disclosures about NSA surveillance, there has been no shortage of recommendations on how to reform national intelligence. In 2013, President Obama set up a review commission on surveillance and national intelligence; that commission came up with 46 NSA policy recommendations. In 2014, 500 organizations, experts, and officials worldwide, including me, signed the International Principles on the Application of Human Rights to Communications Surveillance, often called the “Necessary and Proportionate” principles. The US Congress has debated several bills offering minor reforms; one or two might have passed by the time you read this.

In this chapter, I look at both national security and law enforcement, and make general policy recommendations, rather than detailed legislative prescriptions. Some of these are easily doable, and others are more aspirational. All of them are where I think government needs to go in the long term.

I am not arguing that governments should never be allowed to conduct surveillance or sabotage. We already give the police broad powers to invade citizens’ privacy and access our data. We do this knowingly—and willingly—because it helps to solve crimes, which in turn makes us safer. The trick is to give government agencies this power without giving them the ability to abuse it. We need the security provided by government as well as security from government. This is what documents like the US Constitution and the European Charter try to do, it’s what the warrant process does, and it’s what fell out of balance in our mad scramble for security against terrorists after the 9/11 attacks.

International Principles on the Application of Human Rights to Communications Surveillance—Summary (2014)

LEGALITY: Limits on the right to privacy must be set out clearly and precisely in laws, and should be regularly reviewed to make sure privacy protections keep up with rapid technological changes.

LEGITIMATE AIM: Communications surveillance should only be permitted in pursuit of the most important state objectives.

NECESSITY: The State has the obligation to prove that its communications surveillance activities are necessary to achieving a legitimate objective.

ADEQUACY: A communications surveillance mechanism must be effective in achieving its legitimate objective.

PROPORTIONALITY: Communications surveillance should be regarded as a highly intrusive act that interferes with the rights to privacy and freedom of opinion and expression, threatening the foundations of a democratic society. Proportionate communications surveillance will typically require prior authorization from a competent judicial authority.

COMPETENT JUDICIAL AUTHORITY: Determinations related to communications surveillance must be made by a competent judicial authority that is impartial and independent.

DUE PROCESS: Due process requires that any interference with human rights is governed by lawful

13