If...Then Algorithmic Power and Politics

What algorithms do in these cases, however, cannot simply be understood by opening up the black box, as it were. This is not because, as Winner (1993) suggests, we risk finding it empty when we open the box; but, rather, as Latour (1999) reminds us, all black boxes are black boxes because they obscure the networks and assemblages they assume and were constituted by. For Latour, all scientific and technical work is made invisible by its own success through a process of blackboxing. In a much-cited example of an overhead projector breaking down, Latour suggests that the black box reveals itself as what it really is—not a stable thing but, rather, an assemblage of many interrelated parts (1999: 183). When a machine runs smoothly, nobody pays much atention, and the actors and work required to make it run smoothly disappear from view (Latour, 1999: 34). For Latour, the black box ultimately hides its constitution and character as a network, while blackboxing refers to the process in which practices become reified. If the metaphor of the black box is too readily used as a way of critiquing algorithms, Latour’s notion of blackboxing reminds us that we might want to scrutinize critically the ways in which algorithms become.

A core tenet of a relational ontology is the principle of relational materialism, the idea that “objects are no mere props for performance but parts and parcel of hybrid assemblages endowed with difused personhood and relational agency” (Vannini, 2015: 5). Concepts such as sociotechnical and sociomateriality are often used to express the idea of a radical symmetry between human and nonhuman actors.15 According to this view, the social and technical are not seen as separate entities that can be considered independently of each other. The social and technical are always already engaging in symbiotic relationships organized in networks, assemblages, or hybrids.16 What is important on a relational account is that the enactive powers of new assemblages or composite entities cannot merely be reduced to its constituent parts. Rather, these new composite entities are able to “produce new territorial organisations, new behaviours, new expressions, new actors and new  realities” (Müller,  2015: 29). This agential force is, perhaps, most explicitly expressed in the concept of assemblage or, more specifically the French term agencement. As Callon points out, “agencement has the same root as agency: agencements are arrangements endowed with the capacity of acting in diferent ways depending on their configuration” (2007: 320).

These discussions, of course, raise the tricky question of how we should think about agency in the first place. After all, it is not for nothing that agency has been called “the most dificult problem there is in philosophy” (Latour, 2005: 51). If algorithms are multiple and part of hybrid assemblages or even hybrid assemblages themselves, then where is agency located? Who or what is acting when we say that algorithms do this or that? Although scholars commited to a relational ontology may difer in terms of the ontological status they ascribe to entities and relations, the general answer would be to see agency as distributed.17 For a theory of the agential capacities of algorithms, adopting a relational view implies discarding any “neatly ordered flow of agency” (Introna, 2016: 9). As Karen Barad puts it, agency is not an atribute that someone or something may possess but, rather, a name for the process of the ongoing reconfiguration of the world (2003: 818). In a similar vein, actornetwork theory sees agency as a mediated achievement, brought about through forging associations (Müller, 2015: 30). Anything—whether human or nonhuman— can potentially forge an association. As Barad emphasizes, “agency is not aligned with human intentionality or subjectivity” (2003: 826). According to Latour: “any thing that does modify a state of afairs by making a diference is an actor”; one needs simply to ask whether something “makes a diference in the course of some other agent’s action or not” (2005: 71). At the core of a relational ontology lies the importance of acknowledging the relationality and agential capacities of nonhumans. Perhaps more so than any other concept, the notion of assemblage has served as way to account for the ways in which relations are assembled for diferent purposes. Deleuze and Parnet view assemblage as a “multiplicity which is made up of many heterogeneous terms and which establishes liaisons, relations between them,” where the only unity “is that of co-functioning” (2007: 69). This notion of co-functioning usefully describes “how diferent agents within the assemblage may possess diferent resources and capacities to act” (Anderson et al., 2012: 181). Viewed in these terms, the agency of algorithms cannot be located in the algorithm as such but in the “ever-changing outcome of its enactment” (Passoth et al., 2012: 4).

50

IF . . . THEN

The implications of viewing agency as distributed are far from trivial. When we hear that “algorithms discriminate” (Miller, 2015) or that “discrimination is baked into algorithms” (Kirchner, 2015), it may easily be understood as saying algorithms possess the agency to discriminate. While cases such as the Google “gorilla” incident (see chapter 2), Amazon’s exclusion of predominately black ZIP codes from their same-day deliveries, or Google’s display of ads for arrest records when distinctively black names are searched (Sweeney, 2013) leave much to be desired in terms of algorithmic fairness and performance, the question of who or what is actually discriminating in these cases is not as straightforward to answer as the media headlines seem to suggest.

Take the controversy over Facebook’s trending feature. In May 2016, Facebook hit the news (again) after it became clear that their trending feature was not, in fact, “the result of a neutral, objective algorithm” but, partly, the accomplishment of human curation and oversight.18 Facebook had employed journalism graduates to keep checks on algorithmically produced trending topics, approving the topics and writing headlines to describe them. The problem was that the human editors employed to oversee the trending topics happened to lean to the political left, and this, according to the news stories, could be seen in the kinds of stories that were made to “trend.” As the Gizmodo article first reported, “In other words, Facebook’s news section operates like a traditional newsroom, reflecting the biases of its workers and the institutional imperatives of the corporation” (Nunez, 2016). Shortly after the story broke, Tom Stocky, a Facebook executive, wrote that “there are rigorous guidelines in place for the review team to ensure consistency and neutrality” (Stocky, 2016). The incident also prompted a leter from the US Senate to Mark Zuckerberg, demanding more transparency about how Facebook operates. The leter, signed by Republican Senator John Thune, asked Facebook to elaborate on questions such as: “Have Facebook news curators in fact manipulated the content of the Trending Topics section” and “what steps will Facebook take to hold the responsible individuals accountable?” As Thune later told reporters, any level of subjectivity associated with the trending topics would indeed be “to mislead the American public” (Corasaniti & Isaac, 2016).

Neither Black nor Box

51

What remained puzzling throughout the ordeal was the apparent lack of vocabulary available to talk about what it is that algorithms do or are even capable of doing, as exemplified in the repeated atribution of bias either to the algorithm or to the humans involved. Words such as bias, neutrality, manipulation, and subjectivity abound, making the controversy one of locating agency in the right place. The prevailing sense in the discourse surrounding the event seemed to be that Facebook should not claim to use algorithms to make decisions when, in fact, humans make the decisions. Of course, what was being slightly overlooked in all of this was the fact that algorithms are always already made, maintained, and sustained by humans. Yet, if only the responsible people could be held accountable, the story went, it would make it easier to control or regulate such “manipulations” and “subjective orderings” in the future. From a relational perspective, however, determining the origin of action as if it belonged to one source only would be misleading. After all, as Latour puts it, “to use the word ‘actor’ means that it’s never clear who and what is acting when we act since an actor on stage is never alone in acting” (2005: 46). The stage in this case was clearly composed of a myriad of participants, including journalism graduates, professional culture, political beliefs, work guidelines, the trending product team, Facebook executives and management, algorithms, users, news agencies, and so on.

So, then, what about the supposed bias of algorithmic processes? As John Naughton, professor of the public understanding of technology, writes in an op-ed in the Guardian, bias or, human values, is embedded in algorithms right from the beginning simply because engineers are humans:

Any algorithm that has to make choices has criteria that are specified by its designers. And those criteria are expressions of human values. Engineers may think they are ‘neutral’, but long experience has shown us they are babes in the woods of politics, economics and ideology. (Naughton, 2016)

52

I F . . . T H E N

Of course, with machine-learning algorithms, some would, perhaps, be tempted to argue that, because the engineers or designers of the system are not necessarily human, concerns over the influence of human values or bias becomes less of a problem. However, just as algorithms may “inherit the prejudices of prior decision-makers,” they may “reflect the widespread biases that persist in society at large” (Barocas & Selbst, 2016: 1). To understand where bias might be located in the case of the Facebook trending topic controversy, it certainly helps to know something about how the trending topic algorithm works. It helps to know, for instance, that, at the time of the controversy, Facebook relied on a few news outlets to determine whether “a subject is newsworthy or not” (Stat, 2016). This meant that heavy hiters such as the New York Times and CNN—traditional media institutions—played a significant part in determining whether something would actually be considered a trend or not. As suggested by the leaked internal documents used by Facebook to guide the work of its trending topic editors, “The document designates 10 outlets as particularly central, and instructs editors to only label a topic as a ‘National Story’ or ‘Major Story’ based on how many of those publications placed the news on their front page” (Brandom, 2016). These guidelines and the information about the work practices of editors and the types of decisions they have to make when faced with labeling something as particularly newsworthy or not sheds important light on the diferent values, mechanisms, and ideologies at play in a seemingly neutral and objective decision-making process. Furthermore, it helps to know how central a role users have in shaping algorithmic outcomes. As reporter Ezra Klein puts it, the user is “Facebook’s most biased curator” (Klein, 2016). Users mater because it is their data, their clicking behavior, preferences, network relations, and communicative actions that provide the data for algorithms to act on.

Although Facebook’s news feed is often heralded as a prime example of algorithmic finesse, the feed is far from simply automatic or without human intervention. As with the trending section, Facebook enlists human intervention of the news feed as well. During the summer of 2014, Facebook set up a “feed quality panel,” a group of several hundred people located in Knoxville, Tennessee, whom the company paid  to provide detailed feedback on what they were seeing in their news feed (Oremus, 2016). Later, Facebook took its panel nationwide, paying a representative sample of users to rate and review their feeds on a daily basis. They even expanded the panel overseas. More recently, Facebook has been running a survey asking a subset of users to choose between two posts that are shown side-by-side and pick the one that appeals to them the most. Facebook is not the only platform that “humanizes” its algorithmic systems. Netflix, for example, employs a wide range of “taggers,” whose responsibility it is to assess the genre, tone, and style of a film’s content to help determine what users might want to watch next. Humans are also in the loop when it comes to music recommendations in which music is often presented as belonging to a special category of content that computers cannot necessarily “understand.” One of the selling points of Apple Music, the music-streaming service developed by Apple and launched in the summer of 2015, was the fact that it relied heavily on human curators and radio hosts to provide the recommendations as opposed to mere algorithms. In an interview with the Wall Street Journal, Apple CEO Tim Cook claimed, “People love the human curation” aspect of the streaming service (WSJ,  2015). Similarly, Ajaj Kalia, product head of Taste Profiles at Spotify, thinks that for “something as emotional as music [ . . . ] it is crucial to keep humans in the loop” (Popper, 2015). Because computers “cannot truly appreciate music,” Spotify employs a team of 32 music experts around the world to curate playlists that are updated on a weekly basis (Knight, 2015). Around the same time that Apple Music was launched, Spotify introduced Discover Weekly, an algorithmically generated playlist that would get closer to cracking the emotional standard of human taste. The algorithms behind Discover Weekly provide users with a recommended playlist every Monday, based on the user’s unique taste profile, compiled by comparing her tastes to the playlists of other users who have featured the same songs and artists that she liked (Heath, 2015; Pasick, 2015). Then, the algorithm goes through the other songs that the related people have added to their playlists but the target user has not yet heard, based on the assumption that there is a good chance she might like them, too (Popper, 2015). Mathew Ogle, who oversees Discover Weekly, describes this process as one of “magic filtering” (Pasick,  2015). However, algo rithms are just one aspect to this magic. As the majority of media reports on Discover Weekly emphasize, the main ingredient is other people and their curated playlists. Reminiscent of what seems to be a broader rhetorical strategy surrounding big data and algorithms on part of media platforms, Ogle insists that thinking of Discover Weekly in terms of algorithms misses the point since the “whole thing is built on data created by humans— it's just that algorithms are connecting the dots on a massive scale [ . . . ] Discover Weekly is humans all the way down” (Heath, 2015, my emphasis). These examples do not merely show how humans are always already implicated in algorithms, whether as end-users providing the data input from which algorithms learn or the human experts employed to help algorithms compute things such as musical taste or the tone of a film, they are also meant to illustrate the pothole of atributing agency to either an algorithm or a human. The point here is not that we need to choose to whom or what agency most obviously belongs. It is not that agency is with the designers, the users or the algorithm. A relational perspective precisely “disavows any essentialist or isolated explanation of either human or nonhuman agency” (Schubert, 2012: 126). Rather, it is a mater of accounting for the manner in which the assemblage or coming together of entities becomes more or less human, more or less nonhuman. As it was suggested in the previous chapter, algorithms have a variable ontology, meaning that the question of where agency operates cannot be conclusively answered. What needs explaining, then, is the continuum of variation. In the following, I want to suggest that one way of answering what makes an algorithm more or less technical/nonhuman or more or less social/human can be done by shifting atention away from questions of where agency is located to questions of when agency is mobilized and on whose behalf.

Neither Black nor Box

53

54

IF . . . THEN

When Is an Algorithm?

The case of Facebook’s trending controversy and other instances of “humanizing algorithms” do not merely call the source of agency into question but suggest that the more politically poignant question to ask is when agency is located, on whose behalf and for what purpose? The real controversy was not the fact that Facebook employs journalism graduates to intervene in an algorithmic decision-making process by adding their editorial human judgments of newsworthiness to the mix. The controversy lies in the selective human and nonhuman agency. What became apparent in the public reaction to the Facebook trending controversy was the fact that algorithms only mater sometimes. This, I want to suggest, constitutes an important dimension of thinking around the politics of algorithms—politics not as what algorithms do per se but how and under what circumstances diferent aspects of algorithms and the algorithmic are made available—or unavailable—to specific actors in particular setings.19 By shifting atention away from the proper source of action toward the practices of mobilizing sources of action in specific circumstances, we might be able to understand both how and when algorithms come to mater (the matering of algorithms is taken up again in chapter 6).20 Why is it that, sometimes, an algorithm is blamed for discrimination when, on a similar but diferent occasion, a human is “accused” of bias? Why is it contentious that Facebook employs journalism graduates in the process of curating the trending feature when, in the context of news organizations, the opposite seems to be considered problematic? Put diferently, why is it OK for Facebook to use algorithms when doing the same thing is considered problematic for a news organization? These and similar questions cannot simply be answered by adhering to a vocabulary of essences. Algorithms are not given; they are not either mathematical expressions or expressions of human intent but emerge as situated, ongoing accomplishments. That is, they emerge as more or less technical/nonhuman or more or less social/human because of what else they are related to. In the case of the Facebook trending topic incident, the algorithms shifted its configuration as a result of controversy. In the face of widespread accusations against the subjective biases of the human editors, Facebook decided to fire the 26 journalism graduates contracted to edit and write short descriptions for the trending topics module. In a bid to reduce bias, Facebook announced, instead, that they would replace them with robots. While still keeping humans in the loop, “a more algorithmically driven process,” according to Facebook, would “allow our team to make fewer individual decisions about topics” (Facebook, 2016). What is of interest in this case is not whether algorithms or humans govern Facebook’s trending topic but how the diferent categories were enlisted and made relevant in the context of the controversy. That is, the “algorithm” emerged as objective and neutral while humans were seen as subjective and biased. The point is that there is nothing inherently neutral about algorithms or biased about humans, these descriptive markers emerge from particular contexts and practices. It could have been otherwise.21

Neither Black nor Box

55

From the perspective of relational materialism, the questions that mater the most are “not philosophical in character, but political” (Mol, 2013: 381). Returning to the notion of ontological politics introduced at the end of the previous chapter, engaging with the question of how algorithms come to mater in contemporary society is not about trying to define what they are or at what points they act but, rather, about questioning the ways in which they are enacted and come together to make diferent versions of reality. I think what the many contrasting examples of controversies involving the algorithm-human continuum show is how algorithms are not inherently good or bad, neutral or biased, but are made to appear in one way or the other, depending on a whole range of diferent factors, interests, stakeholders, strategies and, indeed, politics. The term “ontological politics” is precisely meant to highlight how realities are never given but shaped and emerge through interactions.22 Instead of determining who acts (or discriminates, for that mater), the interesting question is what and when an actor becomes in the particular ways in which the entity is active (Passoth et al., 2012: 4).

By shifting atention away from asking what and where agency is to when agency is and to whom agency belongs in specific situations, we may begin to see how the notion of algorithms as black boxes may not just be an ontological and epistemological claim but, ultimately, a political one as well. As I said in the beginning of this chapter, positioning something as a black box serves diferent functions. The black box of algorithm is not simply an unknown but, in many cases, constitutes what Linsey McGoey (2012) has called a strategic unknown, understood as the strategic harnessing of ignorance. As McGoey asserts, strategic unknowns highlight the ways in which “cultivating ignorance is often more advantageous, both institutionally and personally, than cultivating knowledge” (2012: 555). In the case of disaster management, for example, experts’ claims of ignorance soften any alleged responsibility for the disaster or scandal in question. By mobilizing unknowns strategically, organizations and individuals can insist that detection or prior knowledge was impossible. These forms of ignorance are frequently mobilized in discourses on algorithms and software as well.

In fact, it seems that machine learning and the field of artificial intelligence often appear as entire fields of strategic unknowns. At the most fundamental level, the operations of machine-learning algorithms seem to preclude any form of certainty. Because the machine learns “on its own” without being explicitly programmed to do so, there is no way of knowing what exactly caused a certain outcome. As Dourish reflects on the advent of machine learning. “During my years of computer science training, to have an algorithm was to know something. Algorithms were definitive procedures that lead to predictable results. The outcome of the algorithmic operation was known and certain” (2016: 7). Machine-learning techniques, on the other hand, produce unknowns:

56

I F . . . T H E N

When my credit card company deems a particular purchase or stream of purchases “suspicious” and puts a security hold on my card, the company cannot explain exactly what was suspicious–they know that there’s something odd but they don’t know what it is. (2016: 7)

Indeed, in context of machine learning, the decisional rule emerges “from the specific data under analysis, in ways that no human can explain” (Kroll et al., 2016: 6). While this fundamental uncertainty may be disgruntling to those interested in knowing algorithms, it can also be used to restore what McGoey calls knowledge alibis, “the ability to defend one’s ignorance by mobilizing the ignorance of higherplaced experts” (2012: 563-564). As McGoey writes:

A curious feature of knowledge alibis is that experts who should know something are particularly useful for not knowing it. This is because their expertise helps to legitimate claims that a phenomenon is impossible to know, rather than simply unknowable by the unenlightened. If the experts didn’t know it, nobody could. (2012: 564, emphasis in the original)

People routinely defend their lack of knowledge of algorithms by alluding to the notion that they could not have known because no one else can, either. As the title of a recent article in The Atlantic tellingly puts it: “Not even the people who write algorithms really know how they work” (LaFrance, 2015). To what extent does this notion of unknowability hold true? While it might indeed be dificult to explain what paterns in the data are being caught by the model of a machine learning algorithm as the above statements seem to suggest, the general principles and operational logics of these systems are very well-known. That’s why they are used in the first place. The claim that no human can really explain the working of an algorithm or the decisional rules emerging from patern recognition might be more about the organizational construction of software than the algorithm itself. As discussed in the previous chapter, machine learning systems of massive scale are really networked sets of machine-learning algorithms that are wired together to compute some emergent property (e.g., search). When we are dealing with a newsfeed or a search engine we never deal with single algorithms that train single models from a table of data. They are machine learning systems pieced together and layered like Lego. Often, several machine-learning systems are doing the same task, but a user will only see the results aggregated from the top-three performing ones. If any of the top three start to perform poorly, they will fall out and be replaced automatically.

Neither Black nor Box

57

What we need to remember is that all of these layers and structures are built by people in teams, and so the ability for one person to understand everything is very much the same challenge as understanding how the University of Copenhagen works. Yet at the same time teams can build heuristic understanding of their algorithmic systems, feelings of what might cause it to break or what is causing the bottleneck, which allow them to work and do their job.23

The ignorance of higher-placed experts, however, should not detract us from knowing diferently. Now, if the exact configuration of the algorithmic logic cannot be easily traced—say, for example, by examining what the machine learned at diferent layers in a neural net, it should not keep us from interrogating the oddity itself, particularly since the allusion to ignorance often comes in handy for the platforms themselves. As Christian Sandvig (2015) suggests, “platform providers often encourage the notion that their algorithms operate without any human intervention, and that they are not designed but rather ‘discovered’ or invented as the logical pinnacle of science and engineering research in the area.” When things do not go exactly as planned and platforms are accused of censorship, discrimination or bias, the algorithm is often conveniently enlisted as a strategic unknown. Of course, as Callon and Latour aptly remind us, “black boxes never remain fully closed or properly fastened [ . . . ] but macro-actors can do as if they were closed and dark” (1981: 285). It is certainly odd that a particular purchase is deemed suspicious, or that a credit card company cannot explain why an algorithm came to this or that conclusion. However, the knowledge of an algorithmic event does not so much hinge on its causes as it does on the capacity to produce certain “orderings and disorderings” (Michael, 2004). As an event whose identity is uncertain, there is an opportunity to ask new questions. In the case of the credit card company, the question might not necessarily be phrased in terms of why the algorithm came to a certain conclusion but what that conclusion suggests about the kinds of realities that are emerging because people are using algorithms and what these algorithmic practices do to various people.

In terms of transparency, questions also need to be asked of functions and expectations. When the Norwegian newspaper Aftenposten and Norway’s Prime Minister Erna Solberg called out Facebook in September 2016, accusing its algorithms of censoring the iconic Pulitzer Prize image “Terror of War,” they did so because they expected the algorithm to behave in a specific way. An algorithm, they suggested, should have been able to distinguish between an award-winning iconic image and “normal nudity.” Not only was this incident, to which I shall return later in chapter 6, yet another example of the diferential enactment of agency as exemplified in the question of whether the fault was with the algorithms or the humans: It also shows that, sometimes, what there is to know about algorithms may not be about the algorithm itself but, rather, our own limits of understanding. Should we, perhaps, not just worry about algorithms being wrong but also ask whether they do what they are supposed to do? Why was Facebook called out on censoring an image of a naked girl when, in fact, this is what is expected of them? Knowing algorithms, I want to suggest, may be just as much about interrogating “negative knowledge” in Knorr-Cetina’s sense as it is about trying to peel the layers of a neural network or geting access to the actual source code. In other words, when trying to know algorithms we also have to take into account what things interfere with our knowing, what we are not interested in, what we do not want to know and why.

58

IF . . . THEN

What the figure of the black box conceals is not just the inner workings of algorithms but also the ways in which the unknown can be used strategically as a resource to maintain control or deny liability in certain situations. In suggesting, as I  have done, that the black box metaphor constitutes a red herring of sorts, the metaphor itself becomes a strategic unknown, enabling knowledge to be deflected and obscured. What needs to be critically scrutinized, then, is not necessarily the hidden content of the box but the very political and social practices that help sustain the notion of algorithms as black boxes. The question is not simply whether we  can know algorithms but when the realm of its intelligibility is made more or less probable. That is, when are algorithms framed as unknowns, for whom and for what purpose?

Three Methodological Tactics

Despite the obvious epistemological limits addressed so far, I want to finish this chapter by ofering three methodological tactics for (un)knowing algorithms. Broadly construed, these tactics correspond to the diferent kinds of ignorance discussed throughout this chapter and focus on tackling the seemingly black-boxed nature of algorithms. These tactics are not meant as an exhaustive list but as a few possible methodological routes that scholars may take when they examine algorithms as objects for cultural analysis.

“REVERSE ENGINEERING” KNOWN UNKNOWNS

Instead of following the Enlightenment impetus seeking to reveal the origin of actions, one way to proceed would be to follow the cybernetic lead. Cybernetics is  the science of control and communication, in the animal and the machine (Wiener, 1948). It is concerned with understanding the relationship and feedback mechanisms between a system and its environment. As the neuropsychiatrist Ross Ashby put it in An Introduction to Cybernetics, cybernetics is not concerned with what things are but what they do (1999 [1956]: 1). In the book, Ashby dedicated a whole chapter to the problem of the black box, presented as a challenge to the engineer who had “to deduce what he can of its contents” (1999 [1956]: 86). For Ashby, the black box was not necessarily an obstacle but simply part of everyday life. The black box is not an exception but the norm. Lots of things are seemingly hidden and inaccessible—that is, until we find the leaks, cracks, and ruptures that allow us to see into them.

Neither Black nor Box

59