Data and Goliath The Hidden Battles to Collect Your Data and Control Your World

Price discrimination is also a big deal these days. It’s not discrimination in the same classic racial or gender sense as weblining; it’s companies charging different people different prices to realize as much profit as possible. We’re most familiar with this concept with respect to airline tickets. Prices change all the time, and depend on factors like how far in advance we purchase, what days we’re traveling, and how full the flight is. The airline’s goal is to sell tickets to vacationers at the bargain prices they’re willing to pay, while at the same time extracting from business travelers the much higher amounts that they’re willing to pay. There is nothing nefarious about the practice; it’s just a way of maximizing revenues and profits. Even so, price discrimination can be very unpopular. Raising the price of snow shovels after a snowstorm, for example, is considered pricegouging. This is why it is often cloaked in things like special offers, coupons, or rebates.

Some types of price discrimination are illegal. For example, a restaurant cannot charge different prices depending on the gender or race of the customer. But it can charge different prices based on time of day, which is why you see lunch and dinner menus with the same items and different prices. Offering senior discounts and special children’s menus is legal price discrimination. Uber’s surge pricing is also legal.

In many industries, the options you’re offered, the price you pay, and the service you receive depend on information about you: bank loans, auto insurance, credit cards, and so on. Internet surveillance facilitates a fine-tuning of this practice. Online merchants already show you different prices and options based on your history and what they know about you. Depending on who you are, you might see a picture of a red convertible or a picture of a minivan in online car ads, and be offered different options for financing and discounting when you visit dealer websites. According to a 2010 Wall Street Journal article, the price you pay on the Staples website depends on where you are located, and how close a competitor’s store is to you. The article states that other companies, like Rosetta Stone and Home Depot, are also adjusting prices on the basis of information about the individual user.

More broadly, we all have a customer score. Data brokers assign it to us. It’s like a credit score, but it’s not a single number, and it’s focused on what you buy, based on things like purchasing data from retailers, personal financial information, survey data, warranty card registrations, social media interactions, loyalty card data, public records, website interactions, charity donor lists, online and offline subscriptions, and health and fitness information. All of this is used to determine what ads and offers you see when you browse the Internet.

In 2011, the US Army created a series of recruiting ads showing soldiers of different genders and racial backgrounds. It partnered with a cable company to deliver those ads according to the demographics of the people living in the house.

There are other ways to discriminate. In 2012, Orbitz highlighted different prices for hotel rooms depending on whether viewers were using Mac or Windows. Other travel sites showed people different offers based on their browsing history. Many sites estimate your income level, and show you different pages based on that. Much of this is subtle. It’s not that you can’t see certain airfares or hotel rooms, it’s just that they’re ordered so that the ones the site wants to show you are easier to see and click on. We saw in Chapter 3 how data about us can be used to predict age, gender, race, sexual preference, relationship status, and many other things. This gives corporations a greater advantage over consumers, and as they amass more data, both on individuals and on classes of people, that edge will only increase. For example, marketers know that women feel less attractive on Mondays, and that that’s the best time to advertise cosmetics to them. And they know that different ages and genders respond better to different ads. In the future, they might know enough about specific individuals to know you’re not very susceptible to offers at 8:00 am because you haven’t had your coffee yet and are grouchy, you get more susceptible around 9:30 because you’re fully caffeinated, and then are less susceptible again by 11:00 because your blood sugar is low just before lunch.

People are also judged by their social networks. Lenddo is a Philippine company that assesses people’s credit risk by looking at the creditworthiness of the people they interact with frequently on Facebook. In another weblining example, American Express has reduced people’s credit limits based on the types of stores they shop at.

University of Pennsylvania law professor Oscar Gandy presciently described all this in 1993 as the “panoptic sort”: “The collection, processing, and sharing of information about individuals and groups that is generated through their daily lives as citizens, employees, and consumers and is used to coordinate and control their access to the goods and services that define life in the modern capitalist economy.” Those who have this power have enormous power indeed. It’s the power to use discriminatory criteria to dole out different opportunities, access, eligibility, prices (mostly in terms of special offers and discounts), attention (both positive and negative), and exposure.

This practice can get very intrusive. High-end restaurants are starting to Google their customers, to better personalize their dining experiences. They can’t give people menus with different prices, but they can certainly hand them the wine list with either the cheaper side up or the more expensive side up. Automobile insurance companies are experimenting with usage-based insurance. If you allow your insurance company to monitor when, how far, and how fast you drive, you could get a lower insurance rate.

The potential for intrusiveness increases considerably when it’s an employer– employee relationship. At least one company negotiated a significant reduction in its health insurance costs by distributing Fitbits to its employees, which gave the insurance company an unprecedented view into its subscribers’ health habits. Similarly, several schools are requiring students to wear smart heart rate monitors in gym class; there’s no word about what happens to that data afterwards. In 2011, Hewlett-Packard analyzed employee data to predict who was likely to leave the company, then informed their

managers.

Workplace surveillance is another area of enormous potential harm. For many of us, our employer is the most dangerous power that has us under surveillance. Employees who are regularly surveilled include call center workers, truck drivers, manufacturing workers, sales teams, retail workers, and others. More of us have our corporate electronic communications constantly monitored. A lot of this comes from a new field called “workplace analytics,” which is basically surveillance-driven human resources management. If you use a corporate computer or cell phone, you have almost certainly given your employer the right to monitor everything you do on those devices. Some of this is legitimate; employers have a right to make sure you’re not playing Farmville on your computer all day. But you probably use those devices on your own time as well, for personal as well as work communications.

Any time we’re monitored and profiled, there’s the potential for getting it wrong. You are already familiar with this; just think of all the irrelevant advertisements you’ve been shown on the Internet, on the basis of some algorithm misinterpreting your interests. For some people, that’s okay; for others, there’s low-level psychological harm from being categorized, whether correctly or incorrectly. The opportunity for harm rises as the judging becomes more important: our credit ratings depend on algorithms; how we’re treated at airport security depends partly on corporate-collected data.

There are chilling effects as well. For example, people are refraining from looking up information about diseases they might have because they’re afraid their insurance companies will drop them.

It’s true that a lot of corporate profiling starts from good intentions. Some people might be denied a bank loan because of their deadbeat Facebook friends, but Lenddo’s system is designed to enable banks to give loans to people without credit ratings. If their friends had good credit ratings, that would be a mark in their favor. Using personal data to determine insurance rates or credit card spending limits might cause some people to get a worse deal than they otherwise would have, but it also gives many people a better deal than they otherwise would have.

In general, however, surveillance data is being used by powerful corporations to increase their profits at the expense of consumers. Customers don’t like this, but as long as (1) sellers are competing with each other for our money, (2) software systems make price discrimination easier, and (3) the discrimination can be hidden from customers, it is going to be hard for corporations to resist doing it.

SURVEILLANCE-BASED MANIPULATION

Someone who knows things about us has some measure of control over us, and someone who knows everything about us has a lot of control over us. Surveillance facilitates control.

Manipulation doesn’t have to involve overt advertising. It can be product placement ensuring you see pictures that have a certain brand of car in the background. Or just an increase in how often you see that car. This is, essentially, the business model of search engines. In their early days, there was talk about how an advertiser could pay for better placement in search results. After public outcry and subsequent guidance from the FTC, search engines visually differentiated between “natural” results by algorithm and paid ones. So now you get paid search results in Google framed in yellow, and paid search results in Bing framed in pale blue. This worked for a while, but recently the trend has shifted back. Google is now accepting money to insert particular URLs into search results, and not just in the separate advertising areas. We don’t know how extensive this is, but the FTC is again taking an interest.

When you’re scrolling through your Facebook feed, you don’t see every post by every friend; what you see has been selected by an automatic algorithm that’s not made public. But people can pay to increase the likelihood that their friends or fans will see their posts. Payments for placement represent a significant portion of Facebook’s income. Similarly, a lot of those links to additional articles at the bottom of news pages are paid placements.

The potential for manipulation here is enormous. Here’s one example. During the 2012 election, Facebook users had the opportunity to post an “I Voted” icon, much like the real stickers many of us get at polling places after voting. There is a documented bandwagon effect with respect to voting; you are more likely to vote if you believe your friends are voting, too. This manipulation had the effect of increasing voter turnout 0.4% nationwide. So far, so good. But now imagine if Facebook manipulated the visibility of the “I Voted” icon on the basis of either party affiliation or some decent proxy of it: ZIP code of residence, blogs linked to, URLs liked, and so on. It didn’t, but if it had, it would have had the effect of increasing voter turnout in one direction. It would be hard to detect, and it wouldn’t even be illegal. Facebook could easily tilt a close election by selectively manipulating what posts its users see. Google might do something similar with its search results.

A truly sinister social networking platform could manipulate public opinion even more effectively. By amplifying the voices of people it agrees with, and dampening those of people it disagrees with, it could profoundly distort public discourse. China does this with its 50 Cent Party: people hired by the government to post comments on social networking sites supporting, and to challenge comments opposing, party positions. Samsung has done much the same thing.

Many companies manipulate what you see according to your user profile: Google search, Yahoo News, even online newspapers like the New York Times. This is a big deal. The first listing in a Google search result gets a third of the clicks, and if you’re not on the first page, you might as well not exist. The result is that the Internet you see is increasingly tailored to what your profile indicates your interests are. This leads to a phenomenon that political activist Eli Pariser has called the “filter bubble”: an Internet optimized to your preferences, where you never have to encounter an opinion you don’t agree with. You might think that’s not too bad, but on a large scale it’s harmful. We don’t want to live in a society where everybody only ever reads things that reinforce their existing opinions, where we never have spontaneous encounters that enliven, confound,

confront, and teach us.

In 2012, Facebook ran an experiment in control. It selectively manipulated the newsfeeds of 680,000 users, showing them either happier or sadder status updates. Because Facebook constantly monitors its users—that’s how it turns its users into advertising revenue—it could easily monitor the experimental subjects and collect the results. It found that people who saw happier posts tended to write happier posts, and vice versa. I don’t want to make too much of this result. Facebook only did this for a week, and the effect was small. But once sites like Facebook figure out how to do this effectively, they will be able to monetize this. Not only do women feel less attractive on Mondays; they also feel less attractive when they feel depressed. We’re already seeing the beginnings of systems that analyze people’s voices and body language to determine mood; companies want to better determine when customers are getting frustrated, and when they can be most profitably upsold. Manipulating those emotions to market products better is the sort of thing that’s acceptable in the advertising world, even if it sounds pretty horrible to us.

Manipulation is made easier because of the centralized architecture of so many of our systems. Companies like Google and Facebook sit at the center of our communications. That gives them enormous power to manipulate and control.

Unique harms can arise from the use of surveillance data in politics. Election politics is very much a type of marketing, and politicians are starting to use personalized marketing’s capability to discriminate as a way to track voting patterns and better “sell” a candidate or policy position. Candidates and advocacy groups can create ads and fundraising appeals targeted to particular categories: people who earn more than \$100,000 a year, gun owners, people who have read news articles on one side of a particular issue, unemployed veterans … anything you can think of. They can target outraged ads to one group of people, and thoughtful policy-based ads to another. They can also fine-tune their get-out-the-vote campaigns on Election Day, and more efficiently gerrymander districts between elections. Such use of data will likely have fundamental effects on democracy and voting.

Psychological manipulation—based both on personal information and on control of the underlying systems—will get better and better. Even worse, it will become so good that we won’t know we’re being manipulated. This is a hard reality for us to accept, because we all like to believe we are too smart to fall for any such ploy. We’re not.

PRIVACY BREACHES

In 1995, the hacker Kevin Mitnick broke into the network of an Internet company called Netcom and stole 20,000 customer credit card numbers. In 2004, hackers broke into the network of the data broker ChoicePoint, stole data on over 100,000 people, and used it to commit fraud. In late 2014, hackers broke into Home Depot’s corporate networks and stole over 60 million credit card numbers; a month later, we learned about a heist of 83 million households’ contact information from JPMorgan Chase. Two decades of the Internet, and it seems as if nothing has changed except the scale.

One reasonable question to ask is: how well do the Internet companies, data brokers, and our government protect our data? In one way, the question makes little sense. In the US, anyone willing to pay for data can get it. In some cases, criminals have legally purchased and used data to commit fraud.

Cybercrime is older than the Internet, and it’s big business. Numbers are hard to come by, but the cost to the US is easily in the tens of billions of dollars. And with that kind of money involved, the business of cybercrime is both organized and international.

Much of this crime involves some sort of identity theft, which is the fancy Internet-era term for impersonation fraud. A criminal hacks into a database somewhere, steals your account information and maybe your passwords, and uses them to impersonate you to secure credit in your name. Or he steals your credit card number and charges purchases to you. Or he files a fake tax return in your name and gets a refund that you’re later liable for.

This isn’t personal. Criminals aren’t really after your intimate details; they just want enough information about your financial accounts to access them. Or sufficient personal information to obtain credit.

A dozen years ago, the risk was that the criminals would hack into your computer and steal your personal data. But the scale of data thefts is increasing all the time. These days, criminals are more likely to hack into large corporate databases and steal your personal information, along with that of millions of other people. It’s just more efficient. Government databases are also regularly hacked. Again and again we have learned that our data isn’t well-protected. Thefts happen regularly, much more often than the news services report. Privacy lawyers I know tell me that there are many more data vulnerabilities and breaches than get reported—and many companies never even figure out that their networks have been hacked and their data is being stolen. It’s actually amazing how bad corporate security can be. And because institutions legitimately have your data, you often have no recourse if they lose it.

Sometimes the hackers aren’t after money. Californian Luis Mijangos was arrested in 2010 for “sextortion.” He would hack into the computers of his female victims, search them for sexual and incriminating photos and videos, surreptitiously turn on the camera and take his own, then threaten to publish them if they didn’t provide him with more racy photos and videos. People who do this are known as “ratters,” for RAT, or remote access Trojan. That’s the piece of malware they use to take over your computer. The most insidious RATs can turn your computer’s camera on without turning the indicator light on. Not all ratters extort their victims; some just trade photos, videos, and files with each other.

It’s not just hackers who spy on people remotely. In Chapter 7, I talked about a school district that spied on its students through their computers. In 2012, the Federal Trade Commission successfully prosecuted seven rent-to-own computer companies that spied on their customers through their webcams.

While writing this book, I heard a similar story from two different people. A few years after a friend—or a friend’s daughter—applied to colleges, she received a letter from a

college she’d never applied to—different colleges in each story. The letter basically said that the college had been storing her personal data and that hackers had broken in and stolen it all; it recommended that she place a fraud alert on her account with the major credit bureaus. In each instance, the college had bought the data from a broker back when she was a high school senior and had been trying to entice her to consider attending. In both cases, she hadn’t even applied to the college. Yet the colleges were still storing that data years later. Neither had secured it very well.

As long as our personal data sloshes around like this, our security is at risk.

Business Competitiveness

In 1993, the Internet was a very different place from what it is today. There was no electronic commerce; the World Wide Web was in its infancy. The Internet was a communications tool for techies and academics, and we used e-mail, newsgroups, and a chat protocol called IRC. Computers were primitive, as was computer security. For about 20 years, the NSA had managed to keep cryptography software out of the mainstream by classifying it as a munition and restricting its export. US products with strong cryptography couldn’t be sold overseas, which meant that US hardware and software companies put weak—and by that I mean easily breakable—cryptography into both their domestic and their international products, because that was easier than maintaining two versions.

But the world was changing. Cryptographic discoveries couldn’t be quashed, and the academic world was catching up to the capabilities of the NSA. In 1993, I wrote my first book, Applied Cryptography, which made those discoveries accessible to a more general audience. It was a big deal, and I sold 180,000 copies in two editions. Wired magazine called it “the book the National Security Agency wanted never to be published,” because it taught cryptographic expertise to non-experts. Research was international, and non-US companies started springing up, offering strong cryptography in their products. One study from 1993 found over 250 cryptography products made and marketed outside the US. US companies feared that they wouldn’t be able to compete, because of the export restrictions in force.

At the same time, the FBI started to worry that strong cryptography would make it harder for the bureau to eavesdrop on the conversations of criminals. It was concerned about e-mail, but it was most concerned about voice encryption boxes that could be attached to telephones. This was the first time the FBI used the term “going dark” to describe its imagined future of ubiquitous encryption. It was a scare story with no justification to support it, just as it is today—but lawmakers believed it. They passed the CALEA law I mentioned in Chapter 6, and the FBI pushed for them to ban all cryptography without a backdoor.

Instead, the Clinton administration came up with a solution: the Clipper Chip. It was a system of encryption with surveillance capabilities for FBI and NSA access built in. The encryption algorithm was alleged to be strong enough to prevent eavesdropping, but there was a backdoor that allowed someone who knew the special key to get at the plaintext. This was marketed as “key escrow” and was billed as a great compromise; trusted US companies could compete in the world market with strong encryption, and the FBI and NSA could maintain their eavesdropping capabilities.

The first encryption device with the Clipper Chip built in was an AT&T secure phone.

It wasn’t a cell phone; this was 1993. It was a small box that plugged in between the wired telephone and the wired handset and encrypted the voice conversation. For the time, it was kind of neat. The voice quality was only okay, but it worked.

No one bought it.

In retrospect, it was rather obvious. Nobody wanted encryption with a US government backdoor built in. Privacy-minded individuals didn’t want it. US companies didn’t want it. And people outside the US didn’t want it, especially when there were non-US alternatives available with strong cryptography and no backdoors. The US government was the only major customer for the AT&T devices, and most of those were never even used.

Over the next few years, the government tried other key escrow initiatives, all designed to give the US government backdoor access to all encryption, but the market soundly rejected all of those as well.

The demise of the Clipper Chip, and key escrow in general, heralded the death of US government restrictions on strong cryptography. Export controls were gradually lifted over the next few years, first on software in 1996 and then on most hardware a few years later. The change came not a moment too soon. By 1999, over 800 encryption products from 35 countries other than the US had filled the market.

What killed both the Clipper Chip and crypto export controls were not demands for privacy from consumers. Rather, they were killed by the threat of foreign competition and demands from US industry. Electronic commerce needed strong cryptography, and even the FBI and the NSA could not stop its development and adoption.

GOVERNMENT SURVEILLANCE COSTS BUSINESS

Those of us who fought the crypto wars, as we call them, thought we had won them in the 1990s. What the Snowden documents have shown us is that instead of dropping the notion of getting backdoor government access, the NSA and FBI just kept doing it in secret. Now that this has become public, US companies are losing business overseas because their non-US customers don’t want their data collected by the US government.

NSA surveillance is costing US companies business in three different ways: people fleeing US cloud providers, people not buying US computer and networking equipment, and people not trusting US companies.

When the story about the NSA’s getting user data directly from US cloud providers— the PRISM program—broke in 2013, businesses involved faced a severe public relations backlash. Almost immediately, articles appeared noting that US cloud companies were losing business and their counterparts in countries perceived as neutral, such as Switzerland, were gaining. One survey of British and Canadian companies from 2014 found that 25% of them were moving their data outside the US, even if it meant decreased performance. Another survey of companies found that NSA revelations made executives much more concerned about where their data was being stored.

Estimates of how much business will be lost by US cloud providers vary. One 2013 study by the Information Technology and Innovation Foundation foresees the loss of revenue at \$22 to \$35 billion over three years; that’s 10% to 20% of US cloud providers’ foreign market share. The Internet analysis firm Forrester Research believes that’s low; it estimates three-year losses at \$180 billion because some US companies will also move to foreign cloud providers.

US computer and networking companies are also taking severe hits. Cisco reported 2013 fourth quarter revenue declines of 8% to 10%. AT&T also reported earnings losses, and had problems with its European expansion plans. IBM lost sales in China. So did Qualcomm. Verizon lost a large German government contract. There’s more. I have attended private meetings where large US software companies complained about significant loss of foreign sales. Cisco’s CEO John Chambers wrote to the Obama administration, saying that NSA’s hacking of US equipment “will undermine confidence in our industry and in the ability of technology companies to deliver products globally.”

Chambers’s comments echo the third aspect of the competitiveness problem facing US companies in the wake of Snowden: they’re no longer trusted. The world now knows that US telcos give the NSA access to the Internet backbone and that US cloud providers give it access to user accounts. The world now knows that the NSA intercepts US-sold computer equipment in transit and surreptitiously installs monitoring hardware. The world knows that a secret court compels US companies to make themselves available for NSA eavesdropping, and then orders them to lie about it in public. Remember the Lavabit story from Chapter 5?

All of this mistrust was exacerbated by the Obama administration’s repeated reassurances that only non-Americans were the focus of most of the NSA’s efforts. More than half of the revenue of many cloud companies comes from outside the US. Facebook’s Mark Zuckerberg said it best in a 2013 interview: “The government response was, ‘Oh don’t worry, we’re not spying on any Americans.’ Oh, wonderful: that’s really helpful to companies trying to serve people around the world, and that’s really going to inspire confidence in American internet companies.”

To be fair, we don’t know how much of this backlash is a temporary blip because NSA surveillance was in the news, and how much of it will be permanent. We know that several countries—Germany is the big one—are trying to build a domestic cloud infrastructure to keep their national data out of the NSA’s hands. German courts have recently ruled against data collection practices by Google, Facebook, and Apple, and the German government is considering banning all US companies that cooperate with the NSA. Data privacy is shaping up to be the new public safety requirement for international commerce.

It’s also a new contractual requirement. Increasingly, large US companies are requiring their IT vendors to sign contracts warranting that there are no backdoors in their IT systems. More specifically, the contractual language requires the vendors to warrant that there is nothing that would allow a third party to access their corporate data. This makes it harder for IT companies to cooperate with the NSA or with any other government agency, because it exposes them to direct contractual liability to their biggest and most sophisticated customers. And to the extent they cannot sign such a guarantee, they’re