(Surden, 2014:93) without being explicitly programmed to do so. How should we think about the politics of artifacts when the design of the system is continuously performed by the system itself? Returning to the example of the failed image recognition in the case of Google’s Photos app, the nature of the training data might be just as important to consider as the algorithm itself. As Gillespie points out:
The most common problem in algorithm design is that the new data turns out not to match the training data in some consequential way [...] Phenomena emerge that the training data simply did not include and could not have anticipated [. . .] something important was overlooked as irrelevant, or was scrubbed from the training data in preparation for the development of the algorithm. (2016a: 21)
Indeed, as Introna and Wood contend, it seems “we cannot with any degree of certainty separate the purely social from the purely technical, cause from efect, designer from user, winners from losers, and so on” (2004: 180). This does not mean, however, that taking a “value in design” approach in the study of machine learning algorithms is not a viable option. To the degree that algorithms are, in fact, capable of labeling black people as “gorillas” or to score black people as predominantly highrisk in commiting a future crime (Angwin et al., 2016), they ought to be scrutinized for the kinds of values and assumptions underlying their function. While many scholars and policymakers have worried about the discriminatory capabilities of algorithms, calling for more transparency on part of the companies involved is only part of the solution. What many of the “algorithms gone awry” cases suggest is the importance of seeing the results as reflections of more fundamental societal biases and prejudices. This is to say that, if the machine that is supposed to compute the likelihood of future crimes is fed statistical data tainted by centuries of racial bias materialized in police reports, arrests, urban planning, and the juridico-political systems, it would be misleading only to talk about the power of algorithms in producing such risk assessments. Instead of locating predictive power in the algorithm (narrowly defined), we may think of algorithms as what Foucault calls governmental technologies (Lemke, 2001; Miller & Rose, 1990).
Foucault used the notions of government and governmentality to analyze the forms of power and politics involved in the shaping and structuring of people’s fields of possibility. While Foucault used the terms government and governmentality somewhat interchangeably, especially in his later writings, “government” broadly refers to the “conduct of conduct,” whereas governmentality refers to the modes of thought, or rationalities, underlying the conduct of conduct (Foucault, 1982; 2007; 2010; Lemke, 2001; Rose, 1999).25 As Lemke remarks, “the problematic of government redirects Foucault’s analytics of power” (2012: 17). It is a notion of power that moves beyond relations of consent and force and, instead, sees power relations in the double sense of the term “conduct” as both a form of leading and “as a way of behaving within a more or less open field of possibilities” (Foucault, 1982: 789). Government is about “the right disposition of things” (Foucault, 2007: 134), which concerns “men in their relationships, bonds, and complex involvements with” various things such as resources, environment, habits, ways of acting and thinking (Foucault, 2007: 134).26 By way of example, Foucault asks what it would mean to govern a ship?
36
I F . . . T H E N