When my credit card company deems a particular purchase or stream of purchases “suspicious” and puts a security hold on my card, the company cannot explain exactly what was suspicious–they know that there’s something odd but they don’t know what it is. (2016: 7)
Indeed, in context of machine learning, the decisional rule emerges “from the specific data under analysis, in ways that no human can explain” (Kroll et al., 2016: 6). While this fundamental uncertainty may be disgruntling to those interested in knowing algorithms, it can also be used to restore what McGoey calls knowledge alibis, “the ability to defend one’s ignorance by mobilizing the ignorance of higherplaced experts” (2012: 563-564). As McGoey writes:
A curious feature of knowledge alibis is that experts who should know something are particularly useful for not knowing it. This is because their expertise helps to legitimate claims that a phenomenon is impossible to know, rather than simply unknowable by the unenlightened. If the experts didn’t know it, nobody could. (2012: 564, emphasis in the original)
People routinely defend their lack of knowledge of algorithms by alluding to the notion that they could not have known because no one else can, either. As the title of a recent article in The Atlantic tellingly puts it: “Not even the people who write algorithms really know how they work” (LaFrance, 2015). To what extent does this notion of unknowability hold true? While it might indeed be dificult to explain what paterns in the data are being caught by the model of a machine learning algorithm as the above statements seem to suggest, the general principles and operational logics of these systems are very well-known. That’s why they are used in the first place. The claim that no human can really explain the working of an algorithm or the decisional rules emerging from patern recognition might be more about the organizational construction of software than the algorithm itself. As discussed in the previous chapter, machine learning systems of massive scale are really networked sets of machine-learning algorithms that are wired together to compute some emergent property (e.g., search). When we are dealing with a newsfeed or a search engine we never deal with single algorithms that train single models from a table of data. They are machine learning systems pieced together and layered like Lego. Often, several machine-learning systems are doing the same task, but a user will only see the results aggregated from the top-three performing ones. If any of the top three start to perform poorly, they will fall out and be replaced automatically.
Neither Black nor Box
57