If...Then Algorithmic Power and Politics

At other times, the algorithm is less identifiable; yet, the question of what and where it is looms large. What is at stake, then, in addressing the ontological politics of algorithms is not so much an understanding of what exactly the algorithm is or the moments in which it acts (although this is important, too) but, rather, those moments in which they are enacted and made to mater as part of specific contexts and situations.

40

IF . . . THEN

Neither Black nor Box

(Un)knowing Algorithms

The power of algorithms can be felt in all spheres of society. Algorithms do not merely filter news or arrange people’s social media feeds, they also are routinely used to help law enforcement, financial institutions, and the health sector make decisions that may have a profound impact on individual lives (Pasquale, 2015). Yet algorithms are not the kinds of objective and neutral decision makers that they are frequently held out to be. Examples of bias in algorithmic decision-making now abound, and scholars and policymakers are trying to figure out the best ways to govern and make these systems more accountable. A major obstacle to holding algorithms accountable, however, is to know them in the first place. Many of the algorithms that we are most concerned with and interested in—for example, those that underlie popular media platforms—are often governed by trade-secret protection. The algorithms running Facebook and Google are the “secret sauces” that give shape to the information and data flowing online. Algorithms are made to be opaque—not just to protect businesses but due to the technical necessity of handling the complexity of the system. Knowing algorithms, then, is severely burdened by their often “black-box” nature. As Frank Pasquale writes, knowing what algorithms do maters because authority is increasingly expressed through them: “Decisions that used to be based on human reflection are now made automatically” by encoded rules that are “hidden within black boxes” (2015: 8). Algorithmic decision-making is often described as impenetrable and secretive, concealed behind a “veil of code” and trade law (Perel & Elkin-Koren,  2017). The story often goes that, if only we could make algorithms more transparent, we would stand a beter chance of governing the big corporations that make these automatic decisions on our behalf.

While “transparency is not just an end in itself,” transparency is still seen as a necessary condition for greater intelligibility (Pasquale, 2015: 8). As Barack Obama (2009) famously stated in the opening of a memorandum of the Freedom of Information Act: “A democracy requires accountability, and accountability requires transparency.” The question remains as to what exactly it is that should be made transparent and what transparency is believed to help reveal. In her review of

41

Pasquale’s The Black Box Society, Dewandre (2015) instructively connects the recent calls for greater algorithmic transparency to what feminist scholar Susan H. Williams has called the “Enlightenment Vision.” For Williams, “the liberal model of autonomy and the Cartesian model of truth are deeply connected. The autonomous liberal is the Cartesian knower” (2004: 232). According to the Enlightenment Vision, transparency is what makes rationality, autonomy, and control possible. When something is hidden, the Enlightenment impetus says we must reveal it because knowing leads to greater control.1 But what if the power believed to emanate from algorithms is not easily accessible simply because the idea of origins and sources of actions that come with the Cartesian assumption of causality are problematic to begin with? By thinking around the cusp of sensibility and knowledge, I take up in this chapter the challenge of formulating an epistemological stance on algorithms that is commited to the notion of algorithm as multiple introduced in the previous chapter. That is, how to know algorithms when the algorithm is both multiple, “concealed behind a veil of a code,” and seemingly “impenetrable”?

In this chapter, I use the concept of the black box as a heuristic device to discuss the nature of algorithms in contemporary media platforms and how we, as scholars and social actors interested in them, might atend to algorithms despite, or even because of, their seemingly secret nature. Moving beyond the notion that algorithms are black boxes, this chapter asks, instead, what is at stake in framing algorithms in this way and what such a framing might possibly distract us from asking. The questions that I want to pose in going forward have to do with the limits of using the black box as a functional analogy to algorithms. To what extent are algorithms usefully considered as black boxes? What could we possibly see if eforts to illuminate algorithms are not directed as the source code or details of its encoded instructions but elsewhere, and what would this elsewhere be? The chapter unfolds as follows: First, I address the trope of algorithms as black boxes, arguing that algorithms are neither as black nor as boxed as they are sometimes made out to be. Next, I unpack this claim more by conceptualizing algorithms in terms of a relational ontology. This implies a shift in focus from the question of what algorithms are to what they do. The argument is made that, in order to address the power and politics of algorithms, questions concerning the agency of algorithms should be focused not on where agency is located but when. Finally, I point to three methodological tactics around which to orient ways of making sense of algorithms.

Black Box

THE PROBLEMATIC OF THE UNKNOWN

The concept of the black box has become a catch-all for all the things we (seemingly) cannot know. Referring to an opaque technical device about which only the inputs and outputs are known, the figure of the black box is linked to the history of secrecy—to trade secrets, state secrets, and military secrets (Galison, 2004: 231).2 The black box is an object whose inner functioning cannot be known—at least not by observation, since the blackness of the box obscures vision. Historically, the black box refers, quite literally, to a physical black box that contained war machinery and radar equipment during World War II. In tracing the genealogy of the black box, von Hilgers (2011) describes how the black box initially referred to a “black” box that had been sent from the British to the Americans as part of the so-called Tizard Mission, which sought technical assistance for the development of new technologies for the war efort. This black box, which was sent to the radiation lab at MIT, contained another black box, the Magnetron. During wartime, crucial technologies had to be made opaque in case they fell into enemy hands. Conversely, if confronted with an enemy’s black box, one would have to assume that the box might contain a self-destruct device, making it dangerous to open. As a consequence, what emerged was a culture of secrecy or what Galison (1994) has termed “radar philosophy,” a model of thought that paved the way for the emergence of cybernetics and the analysis and design of complex “man-machine” systems. The black box readily became a metaphor for the secret, hidden, and unknown. In everyday parlance, everything from the brain to markets to nation-states is now conceptualized as a black box. Algorithms are no diferent.

42

IF . . . THEN

When algorithms are conceptualized as black boxes, they are simultaneously rendered a problem of the unknown.3 As unknowns, algorithms do not simply signify lack of knowledge or information. The black box notion points to a more specific type of unknown. What the pervasive discourses on transparency and accountability surrounding algorithms and trade secrecy suggest is that algorithms are considered knowable known unknowns (Roberts, 2012)—that is, something that, given the right resources, might be knowable in principle. All that is needed, according to popular discourse, is to find a way of opening up the black box. Indeed, a key mantra in science and technology studies, “opening up the black box,” implies disentangling the complexities and work that goes into making a technical device appear stable and singular.4 The impetus for opening up the black box can also be seen in calls for greater transparency and accountability characteristic of the “audit society” (Power,  1999). In a climate of auditing, organizations are increasingly asked to be transparent about their dealings and ways of operating. Universities, for example, are asked to produce more and more paper trails, including assessment records, numbers of research outputs, and lists of funding received. As Marilyn Strathern (2000a) puts it, axiomatic value is given to increased information. Today, scholars have extended the notion of auditing to the field of algorithms, arguing for the need to conduct audit studies of algorithms in order to detect and combat forms of algorithmic discrimination (Sandvig et al., 2014). While such eforts are certainly admirable, what I want to examine critically as part of this chapter is precisely the hope of intelligibility atached to calls for more transparency. There are many ways of knowing algorithms (broadly understood) besides opening the black box and reading the exact coded instructions that tell the machine what to do. Indeed, while some things are “fundamentally not discoverable” (von Hilgers, 2011: 42), the widespread notion of algorithms as black boxes constitutes something of a red herring— that is, a piece of information that distracts from other (perhaps, more pressing) questions and issues to be addressed. The metaphor of the black box is often too readily used as a way of critiquing algorithms without critically scrutinizing the metaphor itself. What is gained and what is lost when we draw on the metaphor of the black box to describe algorithms? To what extent does the metaphor work at all?

Neither Black nor Box

43

To say that something is a black box may not simply be a statement of facts. As I will discuss in this chapter, declaring something a black box may serve many diferent functions. Unlike the Socratic tradition, which sees the unknown as a fundamental prerequisite to wisdom, the black box renders the unknown an epistemological problem. The unknown—including the black box—is deemed problematic because it obscures vision and, ultimately, undermines the Enlightenment imperative aude sapere: “ ‘dare to know,’ ‘have the courage, the audacity, to know’” (Foucault, 2010: 306). For Kant, the Enlightenment philosopher par excellence, not knowing is characterized by immaturity, the notion that people blindly accept someone else’s authority to lead (Foucault, 2010: 305). Alas, if something is willfully obscured, the task for any enlightened mind would be to find ways of rendering it visible. Critics of the Enlightenment vision have often raised flags against the notion of exposing or decoding inner workings, as if there were a kernel of truth just waiting to be revealed by some rational and mature mind (a mind that is quite often seen as specifically male) (Chun, 2011; Foucault, 1980; Harding, 1996). In the Kantian tradition, the audacity to know is not just explicitly linked to rationalism but to the quest for the conditions under which true knowledge is possible. On the face of it, then, black boxes threaten the very possibility of knowing the truth.

While threatening access to a seemingly underlying truth, the very concept of the black box also denotes a device that releases rational subjects from their obligation, as Kant would see it, to find a “way out” of their immaturity (Foucault, 2010: 305). As Callon and Latour suggest, “[a] black box contains that which no longer needs to be reconsidered” (1981: 285). In discussions on technical or commercial black boxes and transparency, one of the arguments often raised in defense of the black box is the necessity to keep details closed and obscured. In writing about trade secrets, historian of science Peter Galison (2004) makes the point that secrecy is legitimized as a form of “antiepistemology,” knowledge that must be covered and obscured in order to protect a commercial formula or the like.5 Indeed, it seems that the entire figure of the black box is premised on the notion of antiepistemology. Without secrecy, systems would cease to work properly. From a more technical  standpoint, making the inner workings obscure helps to remedy atempts at gaming the system. As Kroll et al. write, “Secrecy discourages strategic behavior by participants in the system and prevents violations of legal restrictions on disclosure of data” (2016: 16). Finally, from an engineering point of view, concealing or obfuscating large portions of code is a necessary feature of software development. As  Galloway points out, “obfuscation, or ‘information hiding,’ is employed in order  to  make code more modular and abstract and thus easier to maintain” (2006a: 323). Black boxing code, in other words, reduces both the cognitive load of programmers and enables the writing of new portions of the code, or the design new features or functionalities without having to think about every litle detail of how the systems work.

44

I F . . . T H E N

When algorithms are positioned as black boxes in current discourses, it is usually  to problematize the fact that many have harmful or discriminatory efects. Particularly within legal circles, repeated calls have been made for opening up the black box of algorithms. As Pasquale argues, “without knowing what Google actually does when it ranks sites, we cannot assess when it is acting in good faith to help users, and when it is biasing results to favor its own commercial interests” (2015: 9). For Pasquale, what is worrisome is the knowledge asymmetry inherent in the black box society or what he calls the “one-way mirror” of knowledge: “Important corporate actors have unprecedented knowledge of the minutiae of our daily lives, while we know litle to nothing about how they use this knowledge to influence the important decisions that we—and they—make” (2015: 9). This knowledge asymmetry impinges new power relations—not just between corporations knowing more and more about the people it monitors but also between corporations themselves, ultimately undermining robust competition (Pasquale, 2015: 83). Thus, legal scholars (especially, in the US context) are now intensifying calls for greater transparency, asking corporations to make their workings more transparent (see, for example, Balkin, 2016; Benjamin, 2013; Citron & Pasquale, 2014; Cohen, 2016; Mehra, 2015).6 As Pasquale provocatively puts it, “you can’t form a trusting relationship with a black box” (2015: 83).

What, then, are some of the proposed solutions to the pervasive black boxing of technical details in the “algorithmic society” (Balkin, 2016)? For Pasquale, black boxes must be revealed in order to counteract any wrongdoings, discrimination, or bias these systems may contain: “algorithms should be open for inspection—if not by the public at large, at least by some trusted auditor” (2015: 141). Here, opening up may simply involve making source code accessible. As Citron and Pasquale argue in the context of detecting bias in credit-scoring systems, “To know for sure, we would need access to the source code, programmers’ notes and algorithms at the heart of credit-scoring systems to test for human bias, which of course we do not have” (2014: 14). Others are more skeptical about the demands for transparent source code, pointing out that such calls are not taking account of the many adequate reasons for not making every detail fully transparent (Kroll et al.,  2016). However, opening up the black box of algorithms may also imply opening up the conditions under which algorithms can be legally audited from the outside. As a recent lawsuit filed in US federal court by researchers Sandvig and colleagues atest, the law may constitute an additional barrier to access—in this case, “preventing researchers from collecting data to determine whether online algorithms result in discrimination” (Grauer, 2016). While the many calls for greater transparency certainly hold merit, what I want to argue in the following is that knowing algorithms need not necessitate opening the black box. In fact, the black box may not even be the most adequate notion to use when thinking about the ontology and epistemology of algorithms.

Neither Black nor Box

45

UNKNOWING ALGORITHMS

While it is true that proprietary algorithms are hard to know, it does not make them unknowable. Perhaps paradoxically, I want to suggest that, while algorithms are not unknowable, the first step in knowing algorithms is to un-know them. By unknowing, I mean something akin to making the familiar slightly more unfamiliar. In the previous chapter, we saw how algorithms mean diferent things to diferent stakeholders and people from diferent disciplines. This is not to say that a computer scientist needs to dismiss her knowledge of algorithms or that social scientists should somehow dismiss their ways of seeing algorithms as objects of social concern. “Unknowing” also does not imply “blackboxing” the black box even more. Rather, “unknowing” means seeing diferently, looking elsewhere, or not even looking at all. As much as calls for transparency atempt to make the object of concern more visible, visibility, too, may conceal. As Strathern notes, “there is nothing innocent about making the invisible visible” (2000a: 309). Too much information may  blind us from seeing more clearly and, ultimately, from understanding (Strathern, 2000a; Tsoukas, 1997). Unknowing does not foreclose knowledge but challenges it. In this sense, the kind of unknowing I have in mind can be likened to Bataille’s notion of “nonknowledge” as “a form of excess that challenges both our thinking and our ethics” (Yusof, 2009: 1014). For Bataille, nonknowledge is not something that must be eradicated but embraced as an enriching experience (1986, 2004).7 Baudrillard develops the distinction between “obscenity” and “seduction” in response to Bataille’s thinking about the knowledge/nonknowledge divide, stating how “nonknowledge is the seductive and magical aspect of knowledge” ( Juergenson & Rey, 2012: 290). In other words, we might think of “unknowing algorithms” as a form of distancing or a form of engaging with the seductive qualities of algorithms that cannot always be explained in fully rational terms. On a practical level, unknowing algorithms may simply imply opening the black box of one’s own assumptions about knowing algorithms. For a computer scientist, this may imply knowing more about the social dynamics impinging on the data that algorithms are designed to process. Conversely, for social scientists and humanities scholars, it might imply knowing more about how computers and algorithms make decisions (Kroll et al.,  2016: 14). On a more theoretical and conceptual level, unknowing algorithms implies confronting the limits of the metaphor of the black box itself. Grappling with what Karin Knorr-Cetina calls negative knowledge, the task is  to identify the limits and imperfections of the black box metaphor. For Knorr-Cetina:

46

IF . . . THEN

[N]egative knowledge is not nonknowledge, but knowledge of the limits of knowing, of the mistakes we make in trying to know, of the things that interfere with our knowing, of what we are not interested in and do not really want to know. (1999: 64)

Unknowing algorithms as a first step in knowing them would be to engage more actively in addressing the things that seem to interfere with or keep us from knowing. Indeed, what I want to argue is that the metaphor of the black box itself constitutes such interference. In our atempts to open the black box, what frequently gets ignored is the question of whether the metaphor of the black box holds at all. It might be easier and more tangible to think of algorithms as black boxes because it allows the analyst, policymaker, or outside critic to call for more transparency and openness as a practical means for making powerbrokers accountable. For all its alleged blackness, the box not only conceals the code the critics want to reveal, it also conceals the fact that algorithms may not even be as black or as boxy as they are often held out to be.

Multiple, Processual, Heterogeneous

RELATIONAL ONTOLOGY AND ALGORITHMS

The central focus in this chapter has to do with the possibilities and challenges of knowing algorithms that are commonly described as black boxes. What I have argued so far is that the metaphor of the black box conceals its own limitations as an epistemic device. By encouraging the Enlightenment impetus of unveiling—of opening—the black box positions visibility as a conduit for knowledge and control. Algorithms, though, are not stand-alone boxes but always part of complex systems. They often operate as a collection of algorithms in what are, ultimately, networked systems. This is particularly true for the algorithms underlying platforms such as Facebook, Twiter, Netflix, or YouTube. These systems do not contain one algorithm but a collection of algorithms, working together to create a unified experience. As product owners at Netflix, Gomez-Uribe and Hunt write, “our recommender system consists of a variety of algorithms that collectively define the Netflix experience” (2015: 2). To give an impression of the many algorithms at play in designing this overall Netflix experience, Gomez-Uribe and Hunt list at least eight diferent algorithms, including the personalized video ranker (PVR), which orders the entire catalogue of videos for each member in a personalized way; the Top-N Video ranker, which produces the recommendations in the Top Picks; and the page generation algorithm, which works to construct every single page of recommendations. Similarly, YouTube’s video recommendation system is not powered by one single algorithm but many, including the related/recommended video suggestion algorithm, search algorithms, and Content ID (Davidson et al., 2010). Facebook is no exception. As technology reporter Will Oremus (2016) writes in Slate: “the algorithm is really a collection of hundreds of smaller algorithms solving the smaller problems that make up the larger problem of what stories to show people.”

Neither Black nor Box

47

Their multiplicity, moreover, pertains to their constantly changing nature. As we saw in the previous chapter, Facebook is routinely described as a work in progress. This is not just to be understood as innovation speak but, quite literally, as an integrated business plan constantly to improve its algorithms in order to retain users. This ethic of continuous change is by no means unique to Facebook: It is part and parcel of how most online platforms now develop their products. So-called A/B testing is integral to the culture of experimentation that governs platforms as a means for assessing and comparing how well diferent versions of the algorithm perform.8 A/B tests basically work like focus groups in which people are asked for their opinion on a certain product—with the important diference that most people participating in an A/B test are not aware of it. As Wired reporter Brian Christian (2012) explains:

Without being told, a fraction of users are diverted to a slightly diferent version of a given web page and their behavior compared against the mass of users on the standard site. If the new version proves superior—gaining more clicks, longer visits, more purchases—it will displace the original; if the new version is inferior, it’s quietly phased out without most users ever seeing it.

These “invisible” experimentations have now become such an important part of product development on the Web that, for a company like Netflix, A/B test results are seen as their “most important source of information for making product decisions” (Gomez-Uribe & Hunt, 2015: 11). Typically performed as a test of two variants and on two user groups, A/B tests epitomize the logic of constant change. At any given time, multiples of these tests are executed in parallel. At Netflix, these tests are not just performed on two variants but often include 5 to 10 versions of the algorithm and, typically, run for a period of 2 to 6 months (Gomez-Uribe & Hunt, 2015: 9-12). As a result, there is not one “Netflix” to speak of but many diferent permutations (Seaver, 2013). The culture of experimentation complicates any efort to know algorithms, as the question inevitably arises as to which version, what test group, or what timeframe we are talking about. What I want to suggest, then, is that the algorithms operating in contemporary media platforms are simply neither black nor box but eventful. For a conceptualization of algorithms, this implies a rejection of essences and permanence and an ontological shift towards a world of process and relations.

48

IF . . . THEN

T H E E V E N T FU L N E S S O F A L G O R I T H MS

Theoretically, this ontological shift is indebted to diverse but interrelated perspectives, including actor-network theory and post-ANT (Latour,  2005; Mol, 1999; Law;  1999), process-relational philosophy (Deleuze & Guatari,  1987; Simondon,  2011; Whitehead, 1978), agential realism (Barad,  2003;  2007), and new materialism (Bennet, et al., 2010; Braidoti, 2006).9 While these perspectives do not represent a homogeneous style of thought or a single theoretical position, even among the thinkers referenced here as belonging to a specific category, they all,  in one way or another, emphasize a relational ontology and the extension of power and agency to a heterogeneity of actors, including non-humans or the morethan-human.10

In arguing for an understanding of algorithms as eventful, I am drawing on a Whiteheadian sensibility that puts emphasis on processes of becoming rather than being. For Whitehead (1978), actual entities or actual occasions (such as algorithms) are combinations of heterogeneous elements (or what he calls prehensions).11 Actual entities are only knowable in their becoming as opposed to their being. As Whitehead suggests, “how an actual entity becomes constitutes what that actual entity is. Its ‘being’ is constituted by its ‘becoming’. This is the ‘principle’ of process” (1978: 23). This view of entities as processes breaks with the more traditional view of entities as substance and essence. Indeed, “‘actual entities’ are the final real thing of which the world is made up. There is no going behind actual entities to find anything more real” (Whitehead, 1978: 18). This has important consequences for the analytical treatment of algorithms, since there is nothing “more real” behind the ways in which they actualize to form novel forms of togetherness.12 It is not enough simply to state that algorithms are eventful, understood as constituents that co-become. Analytical choices have to be made as to which relations and which actors to include in the study of actual entities. As Mike Michael suggests, the value of studying processes or events “rests not so much on their empirical ‘accuracy’ as on their capacity to produce ‘orderings and disorderings’ out of which certain actualities (such as practices, discourses and politics) emerge” (2004: 9, 19).13 What this means for an understanding of algorithms is a shift in atention away from questions of what algorithms are to what they do as part of specific situations.

At the most basic level, algorithms do things by virtue of embodying a command structure (Gofey, 2008: 17). For the programmer, algorithms solve computational problems by processing an input toward an output. For users, algorithms primarily do things by virtue of assistance—that is, they help users find something they are searching for, direct atention to the “most important” content, organize information in a meaningful way, provide and limit access to information, or make recommendations and suggestions for what to watch or buy. The doing of algorithms can also be seen in the various ways they shape experience and make people feel a certain way—for example, in how they animate feelings of frustration, curiosity, or joy (see chapter 5). As Introna suggests, “the doing of algorithms is not simply the execution of instructions (determined by the programmers),” algorithms “also enact the objects they are supposed to reflect or express” (2016: 4). The notion of performativity that Introna draws on here posits that algorithms, by virtue of expressing something, also have the power to act upon that world.14 When algorithms become part of people’s everyday lives, incorporated into financial markets or entangled in knowledge production, they do something to those domains.

Neither Black nor Box

49