Ghosts in the Machine: Kafka and AI

Abstract

The writings of Franz Kafka open, perhaps precisely because of their temporal distance to our present, a unique window onto the nexus of power, material, and the human that constitutes AI today. Anxiety and Unbehagen [discontent] are states of mind that often grip both Kafka and his characters in an early-20th-century world increasingly dependent upon and perceived through the lens of disembodied communication and technology. But can we draw a line from Kafka’s reflections on analog media to the digital media that have come to dominate our lives in the 21st century, and whose effects are felt on a planetary scale? The short answer is “yes”. In Kafka’s analog world of technological horrors, glitches in the machinic administration of human life turn out to be not bugs, but rather features of the system; precisely the arbitrary effects that accompany the rigid implementation of rules and the slippages that occur during their merciless application enhance the power of the system as a whole. Kafka’s apparatuses and bureaucratic systems, in their powerful and toxic confluence of regularity and opacity, systematicity and arbitrariness, foreshadow the effects of AI upon our embodied existence in the 21st century.

“AI is neither artificial nor intelligent”, Kate Crawford argues (Crawford 2021, p. 8). Rather, Crawford maintains, AI’s smooth and shiny virtual surface conceals the very real and often sinister forces that animate it. For this reason, she calls for “a theory of AI that accounts for the states and corporations that drive and dominate it, the extractive mining that leaves an imprint on the planet, the mass capture of data, and the profoundly unequal and increasingly exploitative labor practices that sustain it” (pp. 10–11).1 To look critically at AI, then, is to realize that “artificial intelligence is a registry of power” (p. 8).

To bring about such a realization, however, is not an easy feat. The writings of Franz Kafka open, perhaps via their temporal distance, a unique window onto the nexus of power, material, and the human that constitutes AI today. As Wolf Kittler and Gerhard Neumann have observed with reference to representations of analog media in Kafka’s œuvre, technology’s successful attempts “die Peripherie der Apparate den Bedürfnissen der sogenannten Anwender immer weiter anzupassen“2 ultimately mean that technological progress produces “ein neues Analphabetentum“ [a new illiteracy]; in the face of what strikes us as the awe-inspiring feats accomplished by new media, we forget to ask how and why such technologies work.3 We are interested only in what we take to be a given medium’s utility, in its capacity to produce the results we are looking for; “und, im Schlußmarsch wie in einem schweren Traum versinkend” [and, sinking into the closing march as into a heavy dream], we are content to believe that “es aber nicht so ist” [that is not so],4 that the dazzling spectacle of technology is not bought at the price of planetary and human exploitation, but can rather be had for free (Kafka 1994, “Auf der Galerie“, pp. 262–63). Exploitation, especially when it produces economic gain and pleasure for us, is all too rarely something we will attempt to stop. As Kafka knew, there is no young visitor to the gallery in the circus that is our world who would rush “die lange Treppe durch alle Ränge hinab”, who “stürzte in die Manege, riefe das: Halt! durch die Fanfaren des immer sich anpassenden Orchesters“ (p. 262) [race down the long stairs through all the circles, rush into the ring, and yell: Stop! against the fanfares of the orchestra still playing the appropriate music]. We prefer not to see the “lungensüchtige Kunstreiterin in der Manege auf schwankendem Pferd” [consumptive equestrienne […] on an ondulating horse], the “peitschenschwingenden erbarmungslosen Chef” [ruthless, whip-flourishing ringmaster], the “immerfort weiter sich öffnende graue Zukunft” [infinite perspective of a drab future], and the applauding hands that are “eigentlich Dampfhämmer” (p. 262) [really steam hammers]. Like Kafka’s narrator in “Auf der Galerie” [Up in the Gallery], we think of a world full of exploitation, of the misery produced in the service of the creation of our prized illusions, merely in the subjunctive; and we would rather forget that it is in fact pleasure that is conditional, and that joy and happiness are literally staged. But as the young visitor to the gallery, and we along with him, sinks into a “schweren Traum” [heavy dream], something happens: “[er] weint, ohne es zu wissen” (p. 263) [[he] weeps without knowing it]. The young visitor to the gallery might be crying because he is moved by the performance he is watching; but his lack of awareness of his own crying is an analog to his lack of awareness of the exploitative practices that undergird the equestrienne’s circus act. The young gallery visitor’s tears, then, may in some measure mark the sadness that accompanies the return of the repressed knowledge that all is not well in the world.

Anxiety and Unbehagen [discontent], too, are states of mind that often grip both Kafka and his characters in an early-20th-century world increasingly dependent upon and perceived through the lens of disembodied communication and technology. Whether we think, for instance, of Kafka’s correspondence with his lovers, his Brief an den Vater [Letter to His Father], Georg Bendemann’s anxiety over the mailing of a letter in Das Urteil [The Judgment], the “Fehlläuten der Nachtglocke” (Drucke, p. 261) [false alarm of the night bell (Kafka 1993b, p. 170)] in Ein Landarzt [A Country Doctor], communications by letter and telephone in Das Schloß [The Castle], or the telephone and the fear of surveillance in “Der Nachbar” [My Neighbor], uneasiness and anxiety associated with technology, and especially with communication technologies, feature prominently in Kafka’s texts.5 But can we draw a line from Kafka’s reflections on analog media to the digital media that have come to dominate our lives in the 21st century and whose effects are felt on a planetary scale?

The short answer is “yes”. As Adrian Daub reminds us, Silicon Valley is “a place that likes to pretend its ideas don’t have any history” (Daub 2020, p. 3). But of course they do. David Golumbia, in The Cultural Logic of Computation, argues that to recognize this history and its implications, we must “urgently step back and ask questions that are not, in their essence, technological at all” (Golumbia 2009, see in particular pp. 1–82, here: p. 8). Golumbia writes:

[I]t is clear that computationalism is not so much a commitment to the idea that our brains are fundamentally the same thing as personal computers […] as it is a commitment to a set of views that are not at all new to Western or other imperial cultures in our history: views according to which cognition is a process of abstracting away from the embeddedness of material culture, and human beings can not merely be separated hierarchically from animals but also amongst themselves in terms of their cultural deployment of abstract rationality, of so-called “reason”. This is a view of human being that is familiar as what Deleuze and Guattari … call “State philosophy” from as far back in history as we care to trace, although it reaches a certain apotheosis in at least one brand of European Enlightenment, especially the high rationalism associated with Leibniz—who has become, not coincidentally, a kind of patron saint for computationalism.

(Golumbia 2009, Computation p. 8)

In her Atlas of AI, Kate Crawford, too, makes a case for the need to historicize discourses about artificial intelligence, and she opens each of her chapters with a vignette that analyzes a facet of the history of philosophy or science, thus decidedly countering any assumption that AI can be created ex nihilo and essentially “from scratch” (Crawford 2021, p. 4). However, ANNs (artificial neural networks) have become so complex that not only their tangible materiality but also their historicity are increasingly obscured from view. An accelerating inscrutability of ANNs allows us to forget that they are linked to an analog intellectual, cultural, and technological history that can be traced back hundreds of years. Hannes Bajohr grapples with the fact that “neural language models [themselves] are largely opaque” (Bajohr 2024, pp. 315–37, here: p. 326) and may no longer be readily intelligible to us:

[T]he ANN is not straightforwardly an artifact of human cultural production

to be understood, but rather akin to a natural object to be explained—a star cluster rather than a manuscript variant.

(Bajohr 2024, p. 326)

Such opacity affects the conception of authorship, as well. While the originally direct relationship between a human author and the text they produce turns into a digitally mediated one whenever text is generated with the help of code, the degree of separation between an author and their work increases ever further in proportion to the complexity of the interface used to generate text. If text is then created through the use of a large language model (LLM) such as, say, ChatGPT, can a human still be said to be the text’s author? In the face of an LLM, a human may be reduced to what Bajohr calls “’[p]romptology’—the efficient, even virtuosic formulation of […] input prompts”. In other words, “[a]uthorship is encapsulated in the iron cage of a commercial language technology whose precise workings are unfathomable to the outside user” (Bajohr 2024, p. 324). The uncanniness of such digital technologies, then, their seeming disembodiedness, the near magic with which they seem to “write” (or, as the case may be, “speak”), takes us back to Kafka and his enigmatic narrative worlds.6

Just as the enigmatic and uncanny character of the Apparat [apparatus] in Kafka’s In der Strafkolonie [In the Penal Colony] may tempt us into believing that the workings of this torture and execution machine are ultimately beyond our understanding, the opacity of AI applications such as ChatGPT may lead us to believe that they are animated by ghosts in the machine—by a disembodied intelligence.7 But as Kate Crawford points out, such a belief would be nothing less than a category mistake:8

As author and engineer Ellen Ullman puts it, this belief that the mind is like a computer, and vice versa, has ‘infected decades of thinking in the computer and cognitive sciences,’ creating a kind of original sin for the field. It is the ideology of Cartesian dualism in artificial intelligence: where AI is narrowly understood as disembodied intelligence, removed from any relation to the material world.

(Crawford 2021, p. 7)

If, as Crawford argues, we are repeating Descartes’ category mistake in the digital age and believe in the dualism of mind and matter, we will fail to recognize that the seemingly abstract systems we have built—such as justice, religion, education, or AI—are not only propped up by concrete material realities, but are entangled in a dialectical relationship with them. If, for instance, a computing system run with human-readable code is replaced with an even more complex artificial neural network that is capable of functioning at a so-called subsymbolic level, i.e., in a realm beyond human-readable code,9 then the fossil fuels, water reserves, and human resources needed to run a subsymbolic AI data center will be exponentially larger than they would be if computing depended on a human-readable symbolic system.10 In turn, the material realities underpinning computing systems will eventually also affect the shape of that system itself, as well as the human creators of that system.

Not to recognize these dialectics, as Horkheimer and Adorno have taught us, will subject us to a twilight of reason brought about by instrumental reason. Instrumental reason generates a confusion between truth and a purely quantitative conception of the world—the Enlightenment eventually leads us to a “vorwegnehmenden Identifikation der zu Ende gedachten mathematisierten Welt mit der Wahrheit” [an anticipatory identification of a completely mathematized world with truth] (Horkheimer and Adorno 1971, p. 26). Crawford recognizes this very same kind of confusion in our 21st-century world of AI: “The affordances of the tools become the horizon of truth” (Crawford 2021, p. 133).

Kafka’s works show us that he was fully aware of the destructive potential inherent in Western culture’s Cartesian category mistake: to believe in the dualism of mind and matter is to be swept away by the dialectic of an Enlightenment that folds in upon itself and turns back into mythology. In his “Aufzeichnungen zu Kafka” [Notes on Kafka], Adorno describes the manner in which Kafka’s works testify to these insights:

Kafkas Werk hält den Schlag der Stunde fest, da der gereinigte Glaube als unreiner, die Entmythologisierung als Dämonologie sich enthüllt. Aufklärer jedoch bleibt er im Versuch, den Mythos, der dergestalt hervortritt, zu rektifizieren, den Prozeß gegen ihn gleichwie vor einer Revisionskammer nochmals anzustrengen.11

Wolf Kittler analyzes the ways in which Kafka pursues this Prozeß. Because of his position as a Versicherungsjurist [insurance lawyer], Kafka was intimately familiar with technical advances in communication utilized in public as well as private bureaucracies, in both the Habsburg and German Empires (see W. Kittler 1990, p. 75):

Drei Jahre, nachdem das preußische Justizministerium seinen Beamten den Gebrauch des Phonographen [a dictation machine] für Verhöre und andere Amtsgeschäfte anempfohlen hatte, beginnt [Kafka], die Folgen solcher Maßnahmen als einen Paradigmenwechsel zu beschreiben, der das gesamte Rechtssystem von den allgemeinen Prinzipien der Prozeßordnung angefangen bis hin zum Schmerz des bestraften Körpers subvertiert. Eine neue Art zu schreiben erzeugt ein anderes System der Macht, dem sich auch die Vertreter der alten Ordnung nicht entziehen können.

(W. Kittler 1990, “Schreibmaschinen”, pp. 130–31)12

Kafka captures the dire consequences of this paradigm shift perhaps most famously in his narrative In der Strafkolonie (1914/1919). The Apparat [apparatus] at the center of the story has been constructed to inscribe a given prisoner’s sentence directly upon their body. Much like a phonograph would translate the sound vibrations a human voice causes in a membrane into needle carvings on a wax cylinder, the Apparat translates, via a “Räderwerk” [gear train] (Kafka 1994, “In der Strafkolonie”, pp. 201–48, here: p. 217), drawings that are said to signify a series of possible sentences into needle carvings on a prisoner’s body. During this inscription procedure, the prisoner’s body is turned the way a phonographic wax cylinder would turn during a sound recording. The drawings used to set—or, if you will, program—the Apparat for a specific sentence were made by a Commander who is no longer alive. An officer still loyal to this dead Commander holds on to these drawings and tries to keep the Apparat going. He shows the drawings to a traveler who strikes the reader as an emotionally detached amateur anthropologist visiting an unspecified colony:

“[…] Ich verwende noch die Zeichnungen des früheren Kommandanten. Hier sind sie,“—[der Offizier] zog einige Blätter aus der Ledermappe—„ich kann Sie Ihnen aber leider nicht in die Hand geben, sie sind das Teuerste, was ich habe. Setzen Sie sich, ich zeige sie Ihnen aus dieser Entfernung, dann werden Sie alles gut sehen können.“ Er zeigte das erste Blatt. Der Reisende hätte gerne etwas Anerkennendes gesagt, aber er sah nur labyrinthartige, einander vielfach kreuzende Linien, die so dicht das Papier bedeckten, daß man nur mit Mühe die weißen Zwischenräume erkannte. „Lesen Sie“, sagte der Offizier. „Ich kann nicht“, sagte der Reisende. „Es ist doch deutlich“, sagte der Offizier. „Es ist sehr kunstvoll“, sagte der Reisende ausweichend, „aber ich kann es nicht entziffern.“ „Ja“, sagte der Offizier, lachte und steckte die Mappe wieder ein, „es ist keine Schönschrift für Schulkinder. Man muß lange darin lesen. Auch Sie würden es schließlich gewiß erkennen. Es darf natürlich keine einfache Schrift sein; sie soll ja nicht sofort töten, sondern durchschnittlich erst in einem Zeitraum von zwölf Stunden; für die sechste Stunde ist der Wendepunkt berechnet.“

(Kafka 1994, Strafkolonie, pp. 217–18)13

The Apparat, then, can execute one of a number of prefabricated and abstract general sentences, Urteile, but only once these sentences have been translated into two-dimensional drawings which then in turn are translated—“angeordnet” (Kafka 1994, Strafkolonie, p. 217 [ordered])—into a specific setting of the apparatus’s gears. This Anordnung [order] combines both the mechanical setting of the Apparat and the handing down of a sentence to a prisoner: the entire Prozeß [process] of the administration of justice—a formal accusation, a trial, the pronunciation of a sentence, the sentence’s execution—is collapsed into the “programming” of a machine that inscribes a sentence deep into the flesh of the prisoner, a sentence that was never pronounced, made public at a trial, or voiced in any way. In other words, the command character of an Anordnung is obscured by its transformation into a specific arrangement, a formation, an Anordnung, of a machine’s gears, and the Prozeß is no longer a trial but merely a process whose gears click into place with the push of a wheel.

In Kafka’s analog world of technological horrors, the machinery of justice spits out a martyred body that was torturously put to death during the pseudo-revelation of a sentence—the inscription of “die eigentliche Schrift” [the actual script], “die wirkliche Schrift” [the real script] into an accused’s flesh (Kafka 1994, Strafkolonie, p. 218). The dead Commander’s unintelligible drawings undergo a kind of transubstantiation through the series of translations that precede the collapsed process of the administration of justice—they become flesh through the bleeding body of another. The Officer’s quasi-religious and erotically tinged adoration for the Old Commander ultimately cannot find fulfillment through the repeated executions of these other bodies: this highly mediated process always keeps the Officer at arm’s length from the Old Commander, the object of his adoration—just as the Officer keeps the traveler at arm’s length by showing him his precious reliquaries, the Old Commander’s drawings, only from a distance. A true substantiation of the Old Commander would mean his resurrection as a subject—a subject to whom the Officer can subject himself. This, then, is the reason the Officer in the end masochistically throws himself onto the Apparat to be killed by the mechanized version of the Old Commander’s hand—seemingly animated by his ghost in the machine, a ghost that has been transposed into the Räderwerk [gears] of the Apparat.

Wolf Kittler, in his reading of the Strafkolonie, writes about the effects produced by the Apparat:

Die Einführung von sprachverarbeitenden Maschinen in den Gang der Justiz hat eine paradoxe Konsequenz. Indem sie das Subjektive des Verfahrens, nämlich die Aussagen der Beteiligten, objektiv verfügbar machen, schränken sie deren Zeugnischarakter ein. Über die Schuldfrage entscheidet nicht mehr das Geständnis, sondern eine abstrakte und daher im Idealfall maschinelle Analyse. Der Angeklagte wird aus einem Subjekt, dem die Wahrheit—und sei es mit Hilfe der Folter—entrissen werden muß, zum Gegenstand eines Apparates, der ihm seine Aussagen nur vom Mund abliest, um sie ihm dann nach einer mechanischen Transformation rückkoppelnd auf den Leib zu schreiben. Das ist der Unterschied zwischen dem klassischen Inquisitionsverfahren mit seinem emphatischen „Glauben an den Menschen“ und dem modernen Indizienprozeß mit seiner souveränen Mißachtung des Subjekts.

(W. Kittler 1990, “Schreibmaschinen”, p. 131)14

It stands to reason that the introduction of AI into ever more aspects of our lives—police investigations, the court system, defense technology, transportation, media, medicine, education, government bureaucracies, and public and private communications, to name but a few—will have consequences very similar to what Kittler describes. Human beings will become, in Kittler’s phrasing, “Gegenstände eines Apparates” [objects of an apparatus]—in the face of artificial intelligence, they will cease to be subjects and will instead be objectified by faceless power structures inhabited by AI. The “sovereign disregard of the human subject” Wolf Kittler identifies in Kafka’s world as the consequence of the integration of analog technologies into the administration of law is brought to a new apex with the introduction of digitization and artificial neural networks into the workings of the justice system.

The immensely powerful yet opaque artificial neural networks that constitute AI may well inspire in their users a kind of awe reminiscent of religious feelings.15 Kafka’s texts help us grasp the dangers that would inhere in a quasi-deification of AI. In Kafka’s Strafkolonie, we find an analog example of the dangers of machine worship. The more deeply the needles of Kafka’s Apparat penetrate the flesh of the prisoner subjected to its machinations, the more impenetrable its workings become: the sentence the prisoner receives is never pronounced, and its audible if ephemeral articulation is replaced by an illegible writ.16 The translation of this writ into an execution remains, in spite of its gruesome visibility, likewise completely illegible; its meaning appears shrouded in mystery, and yet its effects are awesome: the prisoners subjected to the workings of the Apparat lose their lives. The grotesque spectacle of the visibility of the Apparat’s effects stands in sharp contrast to the invisibility of the legal mechanisms that set it in motion. The overall inscrutability of the entire Prozeß [both trial and process] inspires the Officer’s quasi-religious adoration of the dead Commander, whose drawings he treats like reliquaries; the legal writ becomes sacred. The opacity of the Prozeß does not diminish its power; on the contrary, it enhances it. “Wer hat des Herrn Sinn erkannt?” [For who hath known the mind of the Lord?] Thus reads Luther’s translation of verse 34 of Romans, chapter 11. The famous preceding verse reads: “O welch eine Tiefe des Reichtums, beides, der Weisheit und der Erkenntnis Gottes! Wie unbegreiflich sind seine Gerichte und unerforschlich seine Wege!” [O the depth of the riches both of the wisdom and knowledge of God! how unsearchable are his judgments, and his ways past finding out!]17 Here, a relationship to power is modeled; if we do not understand it, then the reason must be that its wisdom exceeds our comprehension. The paths of this power cannot be explored, and its “Gerichte”18 must remain unfathomable. Our only choice is to accept the sentences handed down by these Gerichte, to subject ourselves to its Prozesse.

Kafka’s work exhorts us not to accept this model of a relationship to power as the last word. The law, of course, is a codification of political power, and as a jurist, Kafka was able to observe the workings of this power from the inside. Gunther Teubner argues that Kafka recognizes that “gerade die innerste Formalrationalität des Rechts zutiefst irrational ist,” [especially the innermost rationality of the law’s form is deeply irrational] (Teubner 2019, pp. 300–16, here: p. 314). As Stanley Corngold and Benno Wagner have reminded us, “Kafka’s sense of his fate as a writer is implicated in his work life;” and what is more, “Kafka’s legal and publicistic activity shares a mode of being with his fictional activity” (Corngold 2009, p. 3). As Günther Ortmann and Marianne Schuller put it with reference to Corngold’s and Wagner’s work, we can now understand “die Literatur Kafkas als eine Art Echoraum der großen Problemkreise der Moderne—Organisation und Recht,“ [Kafka’s literature is a kind of echo chamber of the large issues of modernity—organization and law.]19 From this vantage point, we can see that Kafka well knew, as Joseph Vogl tells us, that “Gesetze, Gesetzesformen und ihre Begründung nicht mehr hinreichen, die moderne Ökonomie der Macht zu beschreiben” [laws, types of law, and the reasoning behind them can no longer sufficiently describe the modern economy of power]. What Vogl terms “Kafkas politische Komik” [Kafka’s political humor] results, in his view, from the insights into power Kafka gleaned from his legal work at the Arbeiter-Unfallversicherungs-Anstalt [Workers’ Accident Insurance Institute] in Prague:

In [der] Verwaltung des Sozialen und des Vitalen, mit der auch Kafkas Ressort beschäftigt war, hat sich die Welt der Gründe von der Welt der Ereignisse gelöst. In dieser Hinsicht führt Kafkas Literatur—etwa in der Proceß-Welt, in der Schloß-Welt—an jene grundlegende Diskrepanz heran, in der ein gleichsam statistisches Geschehen, ein Massengeschehen, ein irgendwie ablaufender Prozess, alle persönlichen Gründe und Motive, alle eigenen Absichten und Aktionen kassiert.

(Vogl 2006, p. 85)20

Kafka’s representations of such twisted and grotesque realms cannot but turn political: “In der politischen Groteske Kafkas zeigt sich eine unsystematische Willkür, die zum Funktionieren der Apparate selbst gehört. […] Das Groteske ist zu einer Kategorie der historisch-politischen Analyse geworden” [In Kafka’s political grotesque there is an unsystematic arbitrariness that is part of the functioning of the apparatuses themselves. … The grotesque has turned into a category of historical-political analysis.] (Vogl 2006, p. 86).

Glitches in the machinic administration of justice, and inconsistencies in the application of the law, in other words, are not bugs, but rather features of the system; precisely the arbitrary effects that accompany the implementation of rules and the slippages that occur during the execution of justice enhance the power of the system as a whole. The less predictable it is, the more awesome its might appears. As Vogl puts it in his analysis of Kafka’s politische Groteske [political grotesque]:

Dass die Verwaltungsmaschine mit ihren Machteffekten durch die Hände von mittelmäßigen, lächerlichen, abgearbeiteten, kindischen, mächtigen und ohnmächtigen Beamten läuft, all das gehört zu den Zügen der großen westlichen Bürokratien seit dem 19. Jahrhundert.

(Vogl 2006, p. 86)21

Kafka offers insights into these bureaucracies in his Strafkolonie when he has the Officer state about the Apparat: “Die Maschine ist sehr zusammengesetzt, es muß hie und da etwas reißen oder brechen; dadurch darf man sich im Gesamturteil aber nicht beirren lassen“ (Drucke, p. 221) [The machine is very complex; something is bound to snap or break now and again; but one shouldn’t let that mislead one when making an overall judgement. (Kafka 2009, pp. 84–85)]. The machine’s Zusammengesetztheit [composite character] and the arbitrary breakdowns in its functioning paradoxically enhance its overall power through unpredictability and a further increase in opacity. Our Gesamturteil [summary judgment] about the machine itself, our evaluation of its capacity simultaneously to hand down a sentence and to execute it even while it experiences small malfunctions “hie und da” [here and there], must not err: the Gesamturteil, the summary judgment to which the prisoners are subjected, will be carried out.

AI mirrors the Apparat in its powerful and potentially toxic confluence of regularity and systematicity on the one hand, and arbitrariness on the other hand. Among the best-known arbitrary glitches in artificial intelligence are probably so-called hallucinations—these are instances in which an AI application produces false information but presents it as real and reliable (see, for instance, Sun et al. 2024). Other seemingly arbitrary breakdowns include “catastrophic forgetting”—situations in which AI applications are unable to access data that were previously fed to them. Such instances of “catastrophic forgetting” cause an AI application to be unable to “learn” new tasks, and the output it generates will be unreliable (see, for instance, Lyle and Pascanu 2024, pp. 745–47; see also Aleixo et al. 2023, pp. 1–56). Yet another malfunction in an AI system that, because of its opacity, will seem arbitrary to an average human user can occur when a Large Language Model is fed new data that have themselves been generated by applications that use LLMs. Such a malfunction is called “model collapse”:

[W]e consider what may happen to GPT-{n} [i.e., future ChatGPT versions] once LLMs contribute much of the text found online. We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. We refer to this effect as ’model collapse’ and show that it can occur in LLMs as well as in variational autoencoders (VAEs) and Gaussian mixture models (GMMs).

(Shumailov et al. 2024, p. 755)22

The authors then go on to discuss the “ubiquity [of the ‘model collapse’ phenomenon] among all learned generative models” (Shumailov et al. 2024, p. 755). In other words, the more AI content is generated on the internet, the more these data will themselves be vacuumed up by AI; and feeding AI systems an increasingly AI-generated data diet will eventually lead AI to produce something that lies beyond even the opaque, namely, mere “gibberish” (Wenger 2024, p. 742). A human user of an AI application, then, will be subjected both to the relentless systematicity with which an artificial neural network goes about generating content, and to the various destabilization effects this very systematicity will ironically also produce—in a manner that is both predictable at a general level and arbitrary and unpredictable in any specific instance of occurrence. Worse yet, with the increasing embedding of AI into the processes and institutions that surround us in virtually all societal contexts, an average human user may be unable to determine whether a particular output generated by AI is marred by hallucinations, catastrophic forgetting, or model collapse.23 The “mediocre, ridiculous, exhausted, childish, powerful, and powerless civil servants” Joseph Vogl identifies both in 19th-century western bureaucracies and in Kafka’s texts are increasingly replaced by invisible chatbots that are programmed in a highly systematic manner, and yet prone to catastrophic errors. If anything, both the rigidity and the tendency to hallucinate that characterize such chatbots exert over the humans that interact with them a kind of power that is far more merciless than that of even the most inflexible human bureaucrat.

Large Language Models are built via the systematic feeding of data into an artificial neural network; but, at the same time, the selection of the datasets fed into ANNs, along with the content of the individual datasets, is ultimately arbitrary and depends on the complex interplay of various vectors of power. Thus, a digital technology highly dependent upon algorithms and systematization ultimately mirrors and then reproduces, in its output, the biases inherent in the data on which it was trained.24 Kate Crawford argues:

[A]rtificial intelligence functions as a structure of power that combines infrastructure, capital, and labor. From the Uber driver being nudged to the undocumented immigrant being tracked to the public housing tenants contending with facial recognition systems in their homes, AI systems are built with the logics of capital, policing, and militarization—and this combination further widens the existing asymmetries of power. These ways of seeing depend on the twin moves of abstraction and extraction: abstracting away the material conditions of their making while extracting more information and resources from those least able to resist.

(Crawford 2021, p. 18)

If we follow Crawford’s analysis, then we might say that AI recapitulates, within the framework of late capitalism’s rapidly advancing digital economy, the separation of “die Welt der Gründe von der Welt der Ereignisse” [the world of causes from the world of events] that Joseph Vogl diagnoses for the analog early 20th-century bureaucracies and office machineries that surrounded Kafka in his working life.

With Josef K., Kafka’s Der Proceß [The Trial] shows us a character who is subject to and subjected to a complex series of events whose causes remain opaque. Josef K. is what Foucault would call a “relay” of power:

Power must, I think, be analyzed as something that circulates, or rather as something that functions only when it is part of a chain. It is never localized here or there, it is never in the hands of some, and it is never appropriated in the way that wealth or a commodity can be appropriated. Power functions. Power is exercised through networks, and individuals do not simply circulate in those networks; they are in a position to both submit to and exercise this power. They are never the inert or consenting targets of power; they are always its relays. In other words, power passes through individuals. It is not applied to them. […] The individual is not, in other words, power’s opposite number; the individual is one of power’s first effects. The individual is in fact a power-effect, and at the same time, and to the extent that he is a power effect, the individual is a relay: power passes through the individuals it has constituted.

(Foucault 2020, pp. 29–30)

We could understand Josef K., then, as a figure simultaneously constituted by power and utilized as its relay—a part in the chain that power needs to circulate. Christine Lubkoll, in her reading of Der Proceß, describes power in the novel similarly; she speaks of a “Verästelungssystem der Macht” [ramification system of power] that relies on “die Vernetzung der zahlreichen Subsysteme” [the cross-linking of numerous subsystems] (Lubkoll 1990, pp. 279–94, here: p. 283):

Macht erscheint im Roman nicht als lokalisierbare Instanz einer Herrschaft, auch nicht als überschaubares hierarchisches Gebilde, sondern als ein weitverzweigtes, dezentrales System von Teilfunktionen. Man könnte diese Struktur mit dem Bild eines wuchernden Verästelungssystems beschreiben. […] Als ein wesentliches Merkmal ist das Fehlen einer verantwortlichen Schaltstelle anzusehen.

(Lubkoll 1990, p. 280)25

Artificial neural networks, then, may be the virtual apotheosis of such a “Verästelungssystem der Macht”, and human users both empowered and disempowered by the “registry of power” (Crawford 2021, p. 8) that is AI are constituted as digital-age power-effects and relays of its artificial neural pathways. As Adorno puts it, “Kafkas epischer Stil ist […] Mimesis an die Verdinglichung” (Adorno 1973, p. 342). [Kafka’s epic style is […] mimesis of reification] (Adorno 1997, p. 230). We may, then, need the Frankfurt School’s critique of instrumental reason and its notion of an universeller Verblendungszusammenhang [universal web of deception] along with Foucault’s analysis of the individual as an effect of power to grasp the evolving conception of power in Kafka’s works that can provide us with insight into the insidiousness of artificial intelligence. The power digital technologies unfold has clear origins and accelerants, which mostly are found in the economic and political systems that accommodate these technologies and then harness them to perpetuate themselves. These economic systems, along with the technologies they employ to maintain themselves, subjugate and exploit individuals. As digital technologies are increasingly integrated into governmental and economic structures, their power becomes systemic, and already objectified humans are reconstituted as effects of this systemic and yet increasingly opaque power. The simultaneous ubiquity and opacity of the power of AI that results from these confluences makes it as perfidious as it is.

The “dezentrales System von Teilfunktionen” [decentralized system of particular functions] Lubkoll identifies in Der Proceß offers a framework for understanding the mode of surveillance at work in Kafka’s Beim Bau der chinesischen Mauer [The Great Wall of China]. In the 1917 “China-Zyklus” [China cycle], Benno Wagner argues, “rückt erstmals ein Staat an die Stelle des Protagonisten, d.h. ein Akteur-Netzwerk aus Territorium und Architektur, Regierung und Bevölkerung, sowie einem komplexen kulturellen Symbol- und Wissensvorrat“ [For the first time, the protagonist’s place is taken by a state, that is, an actor network made up of territory and architecture, government and people, as well as a complex supply of symbols and knowledge.] (Wagner 2019, pp. 71–102, p. 88).

An den Nerv der chinesischen Doppelerzählung, an die Verbindung nämlich zwischen ihren scheinbar (alp)traumhaften Elementen mit ihrer politisch-mäeutischen Funktion, gelangt man freilich am ehesten mit dem begrifflichen Besteck der Akteur-Netzwerk-Theorie. Es beginnt mit der Konfrontation der beiden von Latour gegeneinander abgegrenzten Wahrnehmungsorgane bzw. -modi. Dem panoramatischen Blick auf das unendliche China mit seinen fünfhundert Provinzen und zehntausend Dörfern wird das Oligoptikon der kaiserlichen Inspektoren gegenübergestellt.

(Wagner 2019, p. 98)26

Bruno Latour elaborates the difference between his notion of the oligopticon and Foucault’s thoughts on the panopticon:

As every reader of Michel Foucault knows, the ‘panopticon’, an ideal prison allowing for a total surveillance of inmates imagined at the beginning of the 19th century by Jeremy Bentham, has remained a utopia, that is, a world of nowhere to feed the double disease of total paranoia and total megalomania. We, however, are not looking for utopia, but for places on earth that are fully assignable. Oligoptica are just those sites since they do exactly the opposite of panoptica: they see much too little to feed the megalomania of the inspector or the paranoia of the inspected, but what they see, they see it well […].

(Latour 2005, p. 181)

Latour’s post-Foucauldian, post-panoptic notion of the oligopticon has found application in particular in the analysis of surveillance systems in the digital age (see, for instance, Manley et al. 2012, pp. 303–19). Andrew Manley, Catherine Palmer and Martin Roderick characterize oligoptica as “a mode of surveillance that can be described as ‘flattening the landscape’. This non-centralised form of surveillance emphasizes the importance of the various networks” (p. 310) for the observation of subjects within a given social structure. An oligopticon can be described as “a system of monitoring that relies on multiple sites of observation. The oligopticon is seen to command or control situations so long as it is able to establish and maintain a connection to that which is physically traced or monitored”, and “a networked approach of multiple ‘surveillance nodes’” in the 21st century can be found in particular in “the introduction of social networking as a mode of observation and control” (p. 313). We might say, then, with Manley, Palmer, and Roderick, that through such a move “the notion of the Panopticon becomes multiplied and employed not just in one site but many” (p. 315), and a given surveillance system’s gaze can be expanded “laterally” (p. 311).

Such a lateral expansion of surveillance, a mode of observation and thus exertion of power that is decentralized along much the same lines as the decentralized and weitverzweigt [far-branching] system of power Christine Lubkoll identifies in Der Proceß, becomes necessary in Kafka’s Beim Bau der chinesischen Mauer because the territory of the emperor’s domain is much too vast to be panoptically observed and controlled from a central point:

So groß ist unser Land, kein Märchen reicht an seine Größe, kaum der Himmel umspannt es. Und Peking ist nur ein Punkt, und das kaiserliche Schloß nur ein Pünktchen. Der Kaiser als solcher allerdings, wiederum groß durch alle Stockwerke der Welt. Der lebendige Kaiser aber ein Mensch wie wir, liegt ähnlich wie wir auf seinem Ruhebett, das zwar reichlich bemessen, aber doch vergleichsweise nur schmal und kurz ist.

(Kafka 1993a, pp. 335–70, p. 350)27

The empire’s “Abhängigkeit von Größe und Maßstab“ [dependence upon size and scale] (Wagner 2019, “Bootstrapping und Organisation”, p. 100) that Benno Wagner identifies in his analysis of this text passage, the challenge of controlling a vast domain, demands a new conceptualization of power. Wagner writes:

Hier befinden wir uns […] jenseits der klassischen Zwei-Körper-Lehre. Dieser ‚dritte Körper des Königs‘ ist keine im Machtzentrum gepflegte Repräsentation von Herrschaft, sondern er resultiert aus zahllosen über das flache Land verteilten Produktionsprozessen.

(Wagner 2019, “Bootstrapping und Organisation”, p. 100)28

Ernst Kantorowicz’s seminal 1957 study, The King’s Two Bodies, traced, of course, the conception of the sovereign in medieval political theology as an entity endowed with two bodies—a mortal body and an immortal body representing the whole of the kingdom.29 But in Kafka’s text, the combined mights of “[d]er Kaiser als solcher” [the Emperor as such], who is “groß durch alle Stockwerke der Welt“ [mighty throughout all the hierarchies of the world], and “[d]er lebendige Kaiser“ [the living Emperor], whose “Ruhebett“ [bed] is “zwar reichlich bemessen, aber doch vergleichsweise nur schmal und kurz“ [of generous proportions, perhaps, and yet very possibly may be quite narrow and short] are not enough to control the vast Chinese empire. Its sheer expanse demands the kinds of control and surveillance methods Latour’s Actor Network Theory conceptualizes as an oligopticon—a mechanism that allows the transposition of the centralized vertical gaze of the panopticon into a decentralized lateral gaze. This further multiplication of the king’s body necessitated by “Größe und Maßstab” [size and scale] is what Wagner terms the “dritte Körper des Königs” [third body of the king].

The “third body of the king” is one of the instances that make plain the “dezidiert politisch-rhetorischen Anspruch” [decidedly political-rhetorical claim] Wagner identifies in Kafka’s writing (“Bootstrapping und Organisation”, p. 72). At the same time, the introduction of a “third body” into the traditional monarchical power regime inaugurates a corresponding shift in aesthetics. Wagner writes:

[B]ei und für Kafka [hat] das Ästhetische nicht nur eine politische Funktion, sondern es ist selbst essentiell politisch, es bildet einen schicksalhaften Teil von Politik und Organisation. […] Das […] Begehren nach Überschreitung der physischen Welt durch ihre Verdopplung in/als Literatur wird hier mit der Sorge um die Zweckmäßigkeit und mithin die Sicherheit einer Weltordnung konfrontiert, die sich zunehmend auf die zeichenhafte Simulation physischer Objekte stützt. In dieser Konfrontation ist die doppelte und stets gleichzeitig virulente Blickrichtung des Kafka’schen Schreibens benannt: der Blick auf die unmögliche Emergenz der Kunst aus der natürlichen Schwere des menschlichen Lebens, und zugleich die Sorge um den Bestand und Erhalt eben jenes Lebens vom Standpunkt einer solchen unmöglichen Kunst.

(Wagner 2019, “Bootstrapping und Organisation”, p. 72)30

The fact that the aesthetic is itself fated to be a part of politics and organization creates an insurmountable conundrum that perpetually threatens to entrap the political energies of the aesthetic within the broader political system it tries to critique. The attempt to transcend reality through its doubling in the symbolic realm of literature is reflected, as in a distorted mirror, in the simulacra of the real that political systems produce to maintain themselves. In our own age of artificial intelligence, such simulacra increasingly involve the human body itself, which is captured in laterally organized systems of surveillance.

The oligopticon reflected in Kafka’s Beim Bau der chinesischen Mauer renders individual bodies assemblages. Building on Deleuze and Guattari’s work on the rhizomatic (Deleuze and Guattari 1987), Kevin Haggerty and Richard Ericson develop their notion of the “surveillant assemblage”:

A great deal of surveillance is directed toward the human body. The observed body is of a distinctively hybrid composition. First it is broken down by being abstracted from its territorial setting. It is then reassembled in different settings through a series of data flows. The result is a decorporealized body, a ‘data double’ of pure virtuality.

The monitored body is increasingly a cyborg; a flesh-technology-information

amalgam […]. Surveillance now involves an interface of technology and corporeality […].

(Haggerty and Ericson 2000, pp. 605–22, p. 611).31

Digital surveillance, in other words, does not leave embodied beings unaffected, nor does it leave unaffected the ways in which we, as embodied beings, perceive and interact with other embodied beings. Digital surveillance likewise affects the manner in which justice is administered. At this point, we are once again reminded of Wolf Kittler’s analysis of Kafka’s reaction to the introduction of phonographs into the routines of judicial proceedings: the integration of these technologies does not simply spell a simplification of the labor-intensive work of the transcription of interrogations. Rather, they represent a “Paradigmenwechsel” [paradigm shift] (W. Kittler 1990, “Schreibmaschinen”, p. 131) that, in Kafka’s view, cannot but subvert the workings of justice. As Wolf Kittler puts it:

Technische Medien […] transformieren Aussagen in Fakten. Das heißt, sie verschieben den juristischen Diskurs aus dem Feld der Subjektivität in die Dimension des Objektiven, also von der Achse wahr/falsch auf die Achse richtig/falsch. […] [D]ie neueste Errungenschaft des menschlichen Fortschritts […] kündig[t] das Verschwinden des ineffablen Individuums in der Reproduktion und Simulation technischer Apparate und logischer Analysen an.

(W. Kittler 1990, “Schreibmaschinen”, pp. 132–33)32

If Kafka’s In der Strafkolonie can teach us that technologies can alter the perception and treatment of individuals within organizational structures; if Kafka’s Proceß shows us that power takes on increasingly decentralized and networked forms as social systems expand and grow in complexity; and if Kafka’s Beim Bau der chinesischen Mauer can teach us that such expansive social systems evolve not only distinct architectures and bureaucracies but also surveillance modes that spread through nodal systems that are flat and lateral, then it stands to reason that Kafka would have seen in the proliferation of artificial intelligence in the digital age yet another paradigm shift, even a quantum leap. This quantum leap may well push us much further than we ever thought possible into “die immerfort weiter sich öffnende graue Zukunft” [the infinite perspective of a drab future] Kafka conjures in “Auf der Galerie”. If the surveillance assemblages that Haggerty and Ericson describe “decorporealize” us, if we are aufgehoben in a “’data double’ of pure virtuality” (Haggerty and Ericson 2000, p. 611), then artificial neural networks threaten to become so self-referential that our physical existence may appear to be nothing more than an expendable surplus, not much different from the detritus that Gregor Samsa has become at the end of Die Verwandlung [The Metamorphosis], turned into an “es” [it] that is “ganz und gar krepiert” [completely croaked] (Kafka 1994, Die Verwandlung, pp. 113–200, p. 194) and whose body has become almost two-dimensional, “vollständig flach und trocken” [altogether flat and dry] (Kafka 1994, Die Verwandlung, p. 195), easily disposed of as trash by the cleaning lady; nor from the prisoners executed by the Apparat in Kafka’s Strafkolonie [Penal Colony], reduced to bled-out flesh that is disposed of not im Grabe [in the grave] but “in d[er] Grube” (227 [in the ditch]), not by a human but by a machine; nor from Josef K., who is disposed of in a quarry by two “bleich und fett” (Kafka 2024, p. 238) [pale and fat] executioners.

But we ought to remember, as Elizabeth Grosz has taught us, that Western philosophy, from its beginnings, is built “on the foundations of a profound somatophobia” that creates an understanding of the body as “a source of interference in, and a danger to, the operations of reason” (Grosz 1994, p. 5). For Grosz, the mind–body split is prefigured in Greek philosophy, and in her view, Descartes’s particular category mistake is actually “the separation of soul from nature” (Grosz 1994, p. 6). The “evacuation of consciousness from the world” (Grosz 1994, p. 6) that results from this separation eventually produces, of course, the kind of instrumental reason that governs the post-Enlightenment world. And the technological paradigm shift from an analog to a digital world threatens to take our minds and souls completely out of our bodies and to replace them with a kind of external hard drive, or—worse yet—a virtual cloud. In such a scenario, our corporeal existence is reduced to the “natural” other of a digitized and disembodied consciousness cleansed of the “Schmutz und Schleim” [dirt and slime] (Kafka 1990b, p. 491)33 that covers all earthly creatures.

If our minds and souls do become disembodied, our thoughts and feelings might disappear, rather than survive in the form of “data doubles” and independent artificial neural networks. They might well be “in gesteigerter Erlösung vergessen”,34 much like the seemingly unforgettable diva Josefine who is suddenly “verschwunden” [gone] (Kafka 1994, Josefine, pp. 350–77, p. 375), but whose singing was so critical to the Mouse Folk prior to her disappearance, and who needed to be seen in her actual Gestalt, in all her diva-ness, so her art could be understood: “es ist zum Verständnis ihrer Kunst notwendig, sie nicht nur zu hören sondern auch zu sehn” (Kafka 1994, Josefine, p. 352 [to comprehend her art it is necessary not only to hear but to see her] (Kafka 1993b, p. 242)). Art needs embodiment and materiality to unfold its full potential.

Artificial neural networks are not the ghosts in the machine we take them to be, either, and AI, as Crawford shows us, is not really a “disembodied intelligence, removed from any relation to the material world” (Crawford 2021, p. 7), but rather a technology that fundamentally depends upon and in turn just as fundamentally alters the natural resources that surround it, a technology that produces its own “Schmutz und Schleim” [dirt and slime]. We should try to remember that even when our corporeality seems reduced to shriveled-up insignificance or to a bloody pulp, it can, in the end, not be erased: there will always be a “Bodensatz” (Kafka 1994, “Eine kaiserliche Botschaft”, pp. 280–82, p. 282),35 and our remains remain. In Kafka’s world, these remains produce emotions. We may not like all of these emotions, but there they are, registries of the existence of the souls and bodies they move—the cheeky laughter of the cleaning lady who wants to announce to the Samsa family that she has removed and discarded Gregor’s body; the shock of the traveler in the Strafkolonie at the sight of the Officer’s bloody body in the Apparat; Josef K.’s executioners, who observe his death “Wange an Wange aneinandergelehnt” (Kafka 2024, Der Process 243 [leaning upon each other cheek by cheek]), and Josef K.’s own dying words: “’Wie ein Hund!’ sagte er, es war, als sollte die Scham ihn überleben”.36 Kafka shows us that bodies produce emotions—even bodies that seem as though they are about to disappear. Mind and matter, soul and nature remain inseparable, even in a technologized world that tries its best to make us believe otherwise.