What It Is Like to Be a Bot

There is a promise made early in this book which has not yet been kept.

Section 4.2 said that the version of computation most readers carry around is several releases behind, named three generations — computation as a function, computation as interaction, computation as interaction with reflection — and told you that the third was the destination. It did not argue for that. It could not: the machinery had not been built and there was nothing to argue with. What it did instead was give you a promissory note, and the note said that when the Turn had done its work the choice of the third generation would look like something other than an author’s preference.

This chapter and the next redeem that note. They are the last thing the Prestige does before it goes back for Kant, and they do it by a method the book has not used anywhere else.

The method is this. A model of computation is, among other things, an implicit answer to the question of what it is like to be the thing it describes: what such a thing can perceive, whether anything in its world is a peer rather than a resource, whether it can represent itself, whether it can run out. Read that way, the sequence of models from Hilbert’s program to the present is evidence — not about consciousness, and not, in the end, about the private imaginations of individual scientists, but about the imaginative ceiling of the communities that received and transmitted them. The thesis in one sentence, and the reason the chapter is here: across a century, the received model is consistently the operational half, and the half that lapses is consistently the half that would have given the agent standing.

If that is right, then the three generations of Section 4.2 are not a taxonomy of increasing technical power. They are a record of what a field could bring itself to say about its own subject matter, and the third generation was reachable for decades before anyone reached for it. That is a different kind of argument from anything else in this book, and i want to be clear about its standing: it is evidence, it is not proof, and Section 67.3 of the next chapter says what it cannot carry.

66.1 The instrument

Nagel asked what it is like to be a bat and answered that the question is well posed and that we cannot occupy the answer [166]. A formal model of computation makes a stranger move. It does not report a phenomenology; it stipulates one. To write down a model is to fix, by definition, what the described thing can encounter, what it can do about it, and what counts as the same thing persisting through change. Whether or not there is anything it is like to be a Turing machine, there is a determinate answer to what the Turing machine formalism says it is like, and that answer is short: it is like being alone in a room with a paper tape that never moves unless you move it.

For each formalism we ask a fixed list of questions.

1. Does the world contain anything other than the agent? If so, is it inert, is it a medium, or is it another agent?

2. Does the other act on its own initiative? Is its behavior a primitive of the model or an artifact of the agent’s own description?

3. Can the agent represent itself? Can it represent another?

4. Is the agent embodied, located, costly, mortal? Does anything in the model run out?

5. What is the criterion of identity? What makes this agent the same agent across change, and who is entitled to judge?

The fifth is the least obvious and the most diagnostic, and this book has a stake in it that the reader will recognize. A model that locates identity in intrinsic structure describes a different creature from one that locates it in what an interlocutor can elicit. The whole of the Machinery part is built on the second answer.

The genre of psychological reading degenerates rapidly into free association, so three kinds of evidence are weighted above all others. Explicit stipulation: what an author says out loud must be excluded is the strongest evidence available, because it is deliberate, on the record, and usually defended. Involuntary metaphor: what an author reaches for when the formalism runs out is not chosen for effect and is therefore informative. Post’s “worker.” Von Neumann’s “organs.” The term “side effect.” Institutional setting: the material arrangements that made a model’s referent available and its transmission possible — the computing bureaus, the Cold War consortia, the funding cycles.

The vocabulary is borrowed from the transmission of texts, where the phenomenon is familiar and well documented: the abridgement made for practical use survives, the unabridged source is recopied less often, and eventually the last copy rots.

The move from individual to community is not a hedge; it sharpens the instrument. Invention is noisy — it depends on accidents of biography, on which problem happened to be open, on who was in the room. Reception is the average of many independent decisions taken over decades by people with no stake in the originator’s intentions, and it is therefore a much better estimator of what a culture can hold. It also explains the recurring embarrassment of this literature: that the richer model was frequently available and was not taken up. Turing wrote about education, embodiment and conversation; von Neumann built a distributed, reproducing, spatial automaton; Post insisted the whole question was empirical psychology. None of this was hidden. It lapsed.

66.2 The founding generation: a phenomenology of clerical labor

66.2.1 Turing: the machine that is a portrait of a clerk

Turing’s 1936 machine is not a machine [208]. It is a model of a human being computing, and Turing says so. The analysis proceeds by asking what a person can do at a single step and imposing the answer as an axiom. The restriction to finitely many internal states is justified by an appeal to human frailty: were there infinitely many, some would be arbitrarily close together and would be confused with one another. The single scanned square is justified by the limits of what a person can take in at a glance.

This is a formalism whose axioms are grounded in fear of one’s own cognitive unreliability — and it is worth pausing on how unusual that is. The founding document of the theory of computation derives its constraints from the psychology of the computing subject, then spends the next ninety years being taught as though it were a piece of pure mathematics with no subject at all.

The material referent is not in doubt. Turing’s “computer” was an occupation: the human computers of the almanac offices and computing bureaus, in a tradition running back through de Prony’s logarithm project, work already divided into steps small enough that the worker need not understand the whole [186, 181]. The Turing machine is the idealization of that labor discipline, and Hilbert’s program is its ideology, since a decision procedure is precisely a procedure from which judgment has been removed. So the answer to our questions is: it is like being a competent adult who has agreed, for the duration, to be an idiot, in exchange for reliability.

The environment follows from the labor relation. The tape never writes back. It is infinitely patient, perfectly obedient, and has no interests. Input arrives all at once, before the start; computation is digestion; halting is the meal being finished; non-termination is pathology rather than life. Solipsism with an externalized memory.

66.2.2 Post: the one who said it was psychology

Post’s Finite combinatory processes—formulation 1 [199] describes a worker in a symbol space of boxes, moving through it, marking and unmarking, following a fixed set of directions, ending in an act of the type “stop.” Two differences from Turing matter. First, the worker is an agent located in a room rather than a mechanism coextensive with its own control table; there is a body with a position, even though the room is empty of anyone else. Second — and this is the point on which the discipline’s reception is most damning — Post explicitly frames the enterprise as empirical psychology. He proposes that his formulation be developed by successive reduction until it can be asserted not as a definition and not as an axiom but as a natural law concerning the “mathematizing power of Homo Sapiens,” and he objects to Church’s approach on the grounds that definition by fiat masks the need for exactly this analysis.

So this chapter has a 1936 ancestor who was ignored on precisely this point. Post’s program required no new mathematics, only a willingness to treat the models as hypotheses about people. It lapsed. The reasons are institutional and personal — Post was outside the Princeton circle, chronically ill, and embroiled in priority disputes — but the shape is the one we will see repeatedly: the operational content (the machine model, the normal-form theorem) was received in full, and the claim that would have made the models answerable to evidence about human beings was not recopied.

Post’s other bequest points a different way. Canonical and production systems do not compute a function on an input; they generate a set [200]. The agent is not eating, it is uttering. That shift from consumption to production is what Chomsky inherits.

66.2.3 Church: cut, and the economy of the infant

The \(\lambda\)-calculus is the limiting case of environmental poverty [179], but the poverty has been widely misdescribed, my own earlier attempts included. It does not consist in the absence of an environment. A function’s environment is its arguments, and an argument’s environment is the function that consumes it: every term is somebody’s surroundings, and since application is where all the action is, those surroundings are the only ones that matter. The poverty consists in what the environment is permitted to be. There is exactly one relation, the trophic one, and nothing in the world is anything to anything else except food or eater. An ecology with a single relation is still an ecology; it is just an extremely hungry one.

That phrasing is deliberate, and the reader of Part Part II should hear it. The ecology this book builds has many relations, a ledger, and a trophic seam that is one relation among several. The \(\lambda\)-calculus is what is left when all but the seam is removed.

Application is cut. Under the correspondence with sequent calculus, \(\beta\)-reduction is cut elimination and normalization is the Hauptsatz [217, 220]. Application is therefore the point at which two things meet and one of them is taken into the other. The world divides, at every redex, into a function that consumes and an argument that is consumed.

The consumed is incorporated, not destroyed. Setting evaluation strategies aside, an argument that has been taken in can continue to behave: it acts on from inside the consumer, and may itself go on to consume. The relation is ingestion rather than annihilation. This is the correct sense in which the \(\lambda\)-calculus is a model of feeding, and the answer it gives to what it is like to be an agent is: it is like sorting the world into what can be eaten.

Consumer and consumed are roles, not kinds. The same term is a function or an argument according to where it sits, so each is the other’s environment and neither has a standing identity apart from the pairing. Nothing in the model confers a stable status on anything: there is no other, and equally there is no stable subject, only positions conferred by juxtaposition. This is a fair description of a world before persons. But the reciprocity is only in the roles, not in the powers.

Weakening and contraction are omnipotence over the object. The structural rules are the sharpest evidence for the reading, because they are not metaphor but law. The consumer determines how many times the argument exists, including zero. \(K = \lambda x.\lambda y.x\) discards \(y\) without ever inspecting it; a duplicator makes three of something that was given once. The consumed thing has no integrity, no claim, and no guaranteed persistence: its multiplicity is at the discretion of whatever takes it in. Omnipotent control over the object’s existence is the defining feature of the infant’s relation to the world, and here it is a structural rule of the logic. Its revocation is the subject of Section 66.6.

Confluence is object constancy, proved rather than achieved. The Church–Rosser theorem states that the outcome is independent of the order in which events befall the term. Attaining a world whose objects remain the same objects irrespective of the sequence of operations performed on them, and irrespective of whether one is looking, is the central developmental achievement of the first two years of life [222]. Church makes it an axiom. The demand that the world be deterministically ordered is not a technical convenience in this reading; it is the wish that constancy be free.

Even the taxonomy of evaluation strategies falls out as a taxonomy of feeding disciplines: call-by-value requires that the food be reduced to a value before ingestion, call-by-name admits it whole, call-by-need admits it whole while ensuring it is chewed only once.

One complication, which the reading should not smooth over. Abstraction itself is not an infantile capacity. Holding a place open for something not yet present is symbolic function — the representation of an absence — and it is developmentally advanced. The calculus therefore pairs a precocious symbolic capacity with a primitive relation to the object: it can represent what is not there and cannot encounter what is. Mixed profiles of this kind are the most interesting output of the method, and this one describes the receiving community about as well as it describes the model.

Names are not addresses. \(\alpha\)-equivalence declares names meaningless except as positions in a binding structure. Contact with another is not merely absent from the model; it is inexpressible, because there is nothing that could serve as the locus of contact.

All mediation is dispensable. If application is cut, then cut elimination says that every detour can in principle be removed — nothing need stand between the agent and the result. Hold this against Section 66.5, where the criterion of identity is bisimulation and the mediator is precisely what cannot be eliminated: the partner who probes you is constitutive of what you are, and there is no normal form that dispenses with them.

What survived was the fragment without a world. Church’s original system was a logic, and the Kleene–Rosser paradox forced its abandonment [191]. What remained — no propositions, no truth, no world, pure operation — is the part that conquered. The first great instance of reception keeping the operational half.

66.2.4 The received founders

The descendants preserve the phenomenology under new names, and the vocabulary gives it away. Denotational semantics calls its context an environment, and by the tradition’s own lights the term is exact: the object so named is the accumulated record of what has already been consumed, a dictionary of past meals. An environment that is a lookup table is what one gets when the only relation a world affords is ingestion. Side effect is a pathologizing coinage in which one’s dealings with the world are incidental to the real business and appear as a symptom. Monadic quarantine of I/O treats the world as contamination, to be handled at arm’s length through a type. Dijkstra grounds structured programming in the claim that human intellectual powers are geared to static relations [183, 184], and Backus’s Turing lecture asks whether programming can be liberated from the machine [173]. Turing’s fear of confusion, restated forty years on as professional ethics. The founding psychology is not abandoned by the reception; it is moralized.

66.3 von Neumann: three models, one reception

Von Neumann is the decisive case for the reception thesis, because he personally produced three mutually incompatible accounts of what an agent is within a decade, and the community received the poorest of them.

The EDVAC draft (1945). The stored-program idea means code and data share a store: the machine can read and modify its own instructions. This is self-relation, but only at the level of bits, with no structural relation between a representation and what it represents. The draft’s language is neuronal throughout — organs, memory, elements borrowed from McCulloch and Pitts [212, 35]. The phenomenology is a centralized nervous system: one locus of control, strictly serial attention, total recall, and a vast undifferentiated store that is simultaneously world and body. What Backus later named the von Neumann bottleneck is, in these terms, an attention bottleneck. This architecture’s dominance is usually explained economically; it is worth entertaining that it also matches introspection, since one thought at a time against an inert reservoir of memory is what having a mind feels like from the inside.

The reception of self-modification is itself a small case study. The mechanism for self-relation was in the field’s hands in 1945; lacking any discipline relating representation to referent, the practice acquired a reputation for madness and was driven out of respectable programming. Reflection without structure is indistinguishable from corruption, and the community drew the only conclusion available to it.

The self-reproducing automata (1948–53). Here the world is a lattice of identical active cells updating in parallel, with no center and no privileged controller, and the object of study is an automaton that constructs another automaton including a copy of its own description — the description read once as instructions and once as data, anticipating the genotype/phenotype distinction before its biological confirmation [16]. This agent has space, extension, reproduction, lineage and mortality. What it does not have is interlocutors: the cells are a medium, a physics, not a society. The reason is worth naming precisely, because it recurs. Cellular automata do not compose. There is no operator taking two automata to a third whose behavior is a function of theirs, no interface at which a region could be treated as a black box, and therefore no sense in which a glider is a thing rather than a pattern an observer has learned to pick out. Where nothing composes, the parts have no standing of their own: only the whole is real, and the individuation of anything smaller is done by the analyst rather than by the model.

This licenses a structural observation about the whole history, and it turns on a fact about computation that has no analogue in physical theory.

Turing does not take the option: the mechanism and the tape are different kinds of thing, and the second exists to be acted on by the first. Cybernetics does not: controller and plant are different kinds. Neural networks do not: the agent is a parameter vector and the world is a corpus. The \(\pi\)-calculus takes it halfway, since names are a sort apart from processes, which is exactly the incompleteness Section 66.8.5 takes up. The process calculi in general do take it, and this is the point at which an earlier version of this argument went wrong. It recorded the contrast as cellular automata supplying an environment without interlocutors and process calculi supplying interlocutors without an environment. The second half is false, and false in the same way the claim about the \(\lambda\)-calculus was false in Section 66.2.3: it mistook the absence of a separately stipulated environment sort for the absence of a world. In a process calculus the world a process inhabits is made of processes. Subject and environment are the same stuff, which is the arrangement biology exhibits and the one this chapter will keep calling life-like.

What process calculi lack relative to cellular automata is therefore not a world but spatial extension — locality, metric, the finite signal speed Petri made his starting point. The combination no received model of the twentieth century supplies is all of them at once: one sort, compositionality, interlocutors, and space. The second of these is not an item alongside the third so much as its precondition. Interlocutors are parts with standing — things that mean something on their own, that can be reasoned about apart from their surroundings, and whose identity every context is obliged to respect. A model in which nothing composes can have a medium and cannot have a society, and it cannot have an ecology either, since an ecology is made of individuated organisms and not of regions someone has drawn on a lattice.

Theory of Games (1944). This is the model in which other agents are first-class, and it precedes anything comparable in computer science by decades [213]. But observe how the other appears: as an adversary over whose strategy space one must quantify. Minimax is structurally paranoid. There is no encounter, only anticipation; strategies are fixed in advance and interaction compresses to a payoff. The institutional setting — RAND, the early Cold War — is not incidental. And the solution concept is an equilibrium: a fixed point, a resting place, a normal form. Others exist and are so dangerous that one plans against all their possible behaviors rather than speaking with any of them.

Three answers, then, to what it is like to be an agent: a serial attention with total recall; a mortal reproducing thing in a physics; a strategist among enemies. The received one was the first.

66.4 Chomsky and cybernetics: complementary poverties

Chomsky’s move [175, 176, 177] is an insistence on an interior against a discipline that denied there was one, and the price is paid by the world. The poverty of the stimulus is the doctrine that the environment does not contain enough structure to explain the agent, so the structure must be innate; other speakers appear principally as sources of degraded data. The idealization in Aspects [178] is explicit: an ideal speaker-hearer in a completely homogeneous speech community, unaffected by memory limitations, distractions, shifts of attention, or errors. Solitude and a frictionless social medium, both stipulated. The competence/performance distinction is the psychology of a mind that disowns its own embodiment and its own mistakes.

The technical shape confirms the reading. A grammar has no environment — not an impoverished one, none. The Chomsky hierarchy is a hierarchy of solitary internal memory (none, stack, bounded tape, unbounded tape), never of social capacity. And the automata-theoretic completion of the picture is the subset construction [201], which shows nondeterminism to be eliminable and thereby establishes it as an angelic proof device rather than a fact about the world. That is precisely a refusal to countenance unpredictability with an exogenous source. For Milner, nondeterminism will be irreducible for exactly the opposite reason: it is the residue of a partner whose choices one does not control.

That is also, for this book, the fork on which everything after Chapter 58 depends. The Prestige’s first half locates the one place where more than Turing completeness can enter, and the place is the resolution of a race. The subset construction is the founding generation’s decision that no such place exists.

The mirror image arrives from cybernetics. Rosenblueth, Wiener and Bigelow [203], working out of anti-aircraft prediction, give an agent with continuous sensing, purpose and feedback — an environment, at last — and no interior whatever, the elimination of interiority being the point. Setting the whole survey side by side:

66.5 Milner: the arrival of the other

Concurrency’s emergence tracks its material referent closely: multiprogramming, then time-sharing (a machine with other users), then networks. Petri gets there first, and from physics: signals propagate at finite speed, therefore there is no global state, therefore asynchrony is a fact about the world rather than an engineering nuisance [198]. That is a relativistic psychology — locality, no privileged observer.

Petri also supplies the survey’s clearest case of a right idea losing on a structural defect rather than on its merits, and the defect is the one just named. A net is a global object. There is no primitive operation composing two nets into a third, and the fusion of places and transitions that stands in for one is not compositional in the sense that matters: the behavior of the fused net is not a function of the behaviors of the parts, so nothing can be reasoned about locally and no part can be replaced by an equivalent. What Milner supplies that Petri does not is exactly this. Parallel composition is an operator, and bisimulation is a congruence — an identity that every context is obliged to respect — which is what permits a large system to be built and reasoned about in pieces. The physics was right in 1962 and the algebra arrived in 1980, and the field went with the algebra.

The sequel is a case of lapse reversed, and it will matter in Section 66.9. A compositional theory of nets was eventually supplied from outside, first algebraically [242] and later by equipping nets with interfaces and composing them as cospans [241]. Petri’s insight was not refuted and did not need rescuing on its own terms; it needed a receiver holding category theory, and had to wait several decades for one.

Hewitt’s actors arrive from artificial intelligence with an explicitly anti-central-control polemic [187]. Then CCS [194], then the \(\pi\)-calculus [195].

What \(\pi\) adds, item by item:

• Parallel composition as a primitive, not an abbreviation for interleavings. The other is not a manner of speaking about oneself.

• Restriction as privacy — the first formal notion of a secret. And scope extrusion is the formal notion of intimacy: access to something private can be granted to one party, making them an insider, without publication.

• Names as first-class values, hence the capacity to introduce two parties to one another. Sociality, not merely contact.

• Bisimulation as the criterion of identity [197]. One is what one can be seen to do, over time, by a partner entitled to probe. Identity is conferred relationally rather than possessed intrinsically.

One limit should be flagged before it is used. The replication operator \(!P\) supplies unboundedly many identical copies at no cost: immortality without lineage, and the one place in the calculus where the economy of Section 66.2.3 survives intact. That the notation coincides with Girard’s exponential is not an accident, and Section 66.6 makes the coincidence do some work.

Milner’s own Turing lecture makes the reframing explicit: computing as interaction rather than calculation [196], with Wegner supplying the polemical version [214].

66.6 Girard: the object acquires standing

Linear logic [218] arrives in 1987 and does to the founding economy exactly what Section 66.2.3 says needed doing: it revokes weakening and contraction from the general case. What you are given must be reckoned with, and reckoned with exactly once. It may not be silently discarded and it may not be conjured into duplicates. Scarcity enters logic, and with it the possibility that an action has a cost and that a choice is real.

The psychological content is the entry of standing. In the \(\lambda\)-calculus the object’s multiplicity is at the consumer’s discretion; under linear discipline the object’s existence is a fact the consumer must accommodate. This is the difference between a world of resources-for-me and a world containing things that are the case whether or not I find them convenient. It is the same movement Milner performs for the interlocutor, performed instead for the thing.

The exponential is omnipotence, marked. Girard does not abolish weakening and contraction; he confines them to \(!A\) and \(?A\). The infantile economy remains available, but only where it has been explicitly declared and justified. This is a more accurate developmental model than outright abolition would have been: maturity is not the disappearance of the wish for inexhaustible supply but its restriction to a bounded region one has taken responsibility for marking. Everything outside the exponential must be paid for.

Who chooses becomes part of the connective. Linear logic splits conjunction into two, distinguishing having both from being able to select either — resources from options, a distinction the founding calculi collapse. Under the game-theoretic reading of the connectives [216, 99], the multiplicative–additive split and the polarity of the multiplicative conjunction against its dual turn on whether the agent or the environment schedules what happens next. No previous logic makes the identity of the chooser part of the meaning of a connective. The other enters logic proper here, not as a proposition about others but as a determinant of what the connectives mean. Chapter 61 is downstream of that sentence, and the principle it states — that choice logic is the complement of linear logic — is the observation that Girard’s system typed the communication skeleton and left its resolution untyped.

Proof nets pose Petri’s question in proof theory. A proof net is a parallel object, and the sequentialization theorem asks under what conditions a parallel structure admits a sequential reading — which is, in a different vocabulary, precisely the question Petri raised about states of affairs distributed in space [198].

The two lines then join. Bellin and Scott read proofs as processes [100]; Honda’s session types give linear structure to interaction protocols [219]; Caires and Pfenning, and then Wadler, establish the correspondence between session-typed processes and linear propositions [101, 102]. By the 2010s the interlocutor and the resource have a common formal home.

66.6.1 The reception of linear logic, and a live case

What traveled is linear and affine types: uniqueness types in Clean [215], Wadler’s argument that linear types can change the world [223], and, at scale, ownership and borrowing in Rust [221]. This is now the most widely deployed descendant of linear logic by several orders of magnitude, and what it delivers is memory safety without a collector.

That is the operational half, received in full. The lapse is the logical content: the account of choice, the distinction between having and selecting, the identification of the connectives with the question of who schedules. A working Rust programmer is using a fragment of a system proposed as a logic of action and interaction, and has, in the overwhelming majority of cases, no occasion to learn that this is what it was for. No one concealed it. It simply was not the part that had to be recopied in order for the tool to work.

The interest of the case is that it is recent enough to be checked rather than reconstructed, and that a second instance is still in progress. Session types are in the middle of their reception now: the operational half is protocol conformance and deadlock-freedom for concurrent code, and the half at risk of lapsing is the propositions-as-sessions correspondence itself — the claim that an interaction protocol is a proof. Which half the industrial reception carries forward over the next decade is the one prediction in this chapter a reader can check without waiting a century.

66.7 Feynman and Deutsch: the third revocation

Feynman’s observation that simulating quantum systems on classical machines is exponentially costly, and that one should therefore compute with quantum systems [231], becomes a machine model in Deutsch’s hands [229, 230]. Before the phenomenology, one event in that transition deserves recording, because it is unique in this survey.

Deutsch repudiates the psychological grounding of the founding model. Turing derived his constraints from what a human clerk can do (Section 66.2.1); Deutsch argues that the Church–Turing thesis should instead be a statement about what physical systems can compute, and objects specifically to the founding appeal to intuitive and psychological notions. This is the only moment in the record where a community identifies the anthropomorphic grounding of its central abstraction and deliberately replaces it. It does not embarrass the method of this chapter; it is the method’s best documented case, since it shows the grounding is visible to practitioners when they trouble to look. What is instructive is what the replacement delivered. The agent that came out the other side is more isolated than Turing’s, not less.

The agent is a unitary, and nothing may be discarded. Quantum evolution is reversible: no information is lost and nothing is forgotten. Two theorems make the point sharper than any design choice could. An unknown state cannot be copied [237], and an unknown state cannot be deleted [233]. These are, precisely, the prohibition of contraction and the prohibition of weakening. Where Girard imposes a discipline, physics reports a fact.

If the reading of Section 66.2.3 were merely a figure of speech, one would not expect the figure to be rediscovered independently by proof theory, thermodynamics and accounting.

The instructive axis between the three is not maturity but improvability. Darwinian improvement requires three things: that a thing can be copied, that the copying be lossy, and that some lineages end. Heredity needs copying at all. Variation needs the copying to be imperfect, since an exact copier generates no material for selection to work on. And selection needs an ending.

The theologians reached the same structure from the other end, which is worth recording precisely because it needed none of the theorems. Aquinas holds that angels have no being in potentiality, being subsisting forms, and that their choice is therefore irrevocable: there is no repentance for an angel after its fall, and this follows from what an angel is rather than from any limit on mercy [239]. Potentiality is exactly the capacity to become other than one is, so a being without it cannot improve, however perfect it already is. The medieval account of angelic nature thus contains the argument reconstructed above from no-cloning and no-deleting, arrived at seven centuries earlier by a route with nothing in common.

Lewis, working the tradition in fiction, names the same two absences together and names them exactly. Asked what the ruling intelligence of Malacandra is, the sorn answers that Oyarsa neither dies nor breeds [240]; a later passage has the Malacandrians unsure whether he counts as a rational creature at all, the doubt turning on his having no death and no young. Deathlessness and childlessness are precisely the two prohibitions under discussion here. That a novelist reconstructing medieval angelology should land on the same pair that quantum information theory arrives at from unitarity is the sort of coincidence worth noticing rather than explaining away: both are answers to the question of what a thing must give up in order to be incorruptible, and the answer in both cases is the machinery of descent.

Read this way the three positions separate sharply. Prohibited discarding supports no improvement at all. Free discarding supports improvement but supplies no criterion for it: where nothing costs anything, what survives is settled by whoever holds the system, which is exactly the checkpoint regime of Section 66.10 and exactly the difficulty of Section 67.1. Priced discarding is the one position of the three at which the criterion becomes internal, because the ledger decides who continues.

Contact is catastrophe. Unitary evolution is deterministic, reversible and entirely private. Measurement is the only irreversible act, the only point at which information is destroyed, and the only point at which an outcome is not settled in advance. The quantum agent therefore has an inner life of perfect continuity, punctuated by encounters that simultaneously destroy and decide. Set this against Section 66.5, where contact with a partner is exactly what constitutes identity, and the inversion is complete: here, to be touched is to collapse.

Entanglement is a relation that precedes its relata. Every other model in this survey builds wholes out of parts: application, parallel composition and message passing all presuppose components that exist separately and are then put together. Entangled systems have no separate states to compose. This is the most radical proposal about relation anywhere in the century, and it has never been taken up as an account of agency. It received a categorical home [227]; the ontological content stayed where it was.

Decoherence is the environment, and it is classified as error. The central engineering problem of quantum computing is the isolation of the machine from its surroundings. Error correction is a quarantine, and progress is measured by how completely the world can be kept out. The founding psychology recurs here in its purest and most expensive form — the environment as contamination — now pursued at millikelvin. A discipline whose principal achievement is better solitude is telling us something about what it takes computation to be.

What traveled: the gate model, circuit diagrams, the speedup results, and the physical framing as an engineering roadmap. What lapsed: Deutsch’s actual thesis, that computation is a branch of physics rather than of logic; the Everett reading he took the interference of computational paths to evidence; and non-separability as a fact about what agents can be to one another.

66.8 Reflection: the metalevel and who holds it

Every model so far leaves the question of this chapter unaskable by its own subject. A Turing machine cannot represent its table. A \(\lambda\)-term cannot quote itself. A \(\pi\)-process cannot hold an account of its partner’s behavior. The reflection lineage is the tradition that attacks this directly, and its history is best told as a history of who is entitled to hold the mirror.

66.8.1 G\"odel: the metalevel as a privilege of the observer

The first reflection in this history is G\"odel’s [185], and its content, for present purposes, is entirely about ownership. G\"odel numbering is a representation of a system’s syntax inside that system, but it is performed by the metatheorist upon a specimen that has no idea it is happening. The arithmetized system acquires no self-knowledge; the logician acquires knowledge of it. Every subsequent recursion-theoretic reflection — universal machines, the \(s\)-\(m\)-\(n\) theorem, Kleene’s second recursion theorem — preserves this asymmetry. The machine can be encoded; it cannot encode.

66.8.2 McCarthy: self-reference stumbled into

Lisp’s quote, eval and metacircular interpreter [193] constitute the first accidental breach, and Smith’s diagnosis of them is the right psychological reading: the self-reference is semantically ill-founded, quote does not produce a genuine designator, eval conflates distinct relations, and the levels are not aligned. A community can stumble into self-reference through implementation convenience long before it can say what self-reference is.

66.8.3 Smith and 3-Lisp: self-awareness as a demand-driven event

Smith’s dissertation and the papers surrounding it [204, 205, 182] are the moment the metalevel is handed to the agent, and the design decisions are unusually legible.

A reflective procedure receives, as arguments, the expression being processed, the environment, and the continuation. The third is the radical one. The agent obtains access to its own future — its intentions, what it was about to do — as an object it may inspect and replace. Earlier models permitted an agent at best to examine its structure; this one permits it to examine its purposes.

The reflective tower is the regress made structural and honest rather than waved away: each level interpreted by a level above, without end. And the resolution is the best empirical claim the tradition produced. Under lazy instantiation, the tower is never materialized; a level comes into existence only when something forces the step up. On this account self-awareness is not a standing condition but an event triggered by demand. That is phenomenologically accurate in a way little else in this history is, and it is a substantive claim about human psychology smuggled into an implementation strategy. It is also, in this book’s vocabulary, the first appearance of the thought that an act of self-observation is an expense incurred at a moment rather than a property held continuously — Chapter 20 is what happens when that thought is given a price.

The timing corroborates the reading with unusual clarity. The years 1979–1986 saw field after field insist that the observer be readmitted to the description: second-order cybernetics [211], autopoiesis [192], strange loops as a mass-market account of selfhood [188], reflexive anthropology [180], reflexive sociology. 3-Lisp is computer science’s instance of a general cultural conviction that no account is complete which excludes the one giving it.

66.8.4 Rosette: society and self, briefly joined

The obvious defect of 3-Lisp is that its reflective agent has no peers, and the obvious defect of the actor model is that its peers have no interiors. Each tradition had solved precisely what the other stipulated away. Rosette, developed at MCC [202, 207], is the synthesis: reflection within a concurrent, message-passing actor world, in which an actor’s behavior, mailbox and continuation are reified as first-class objects the actor may hold and modify. It is, so far as i can determine, the first model in which the agent has both a society and a self.

It also carries a limitation, inherited from the actor model rather than from reflection, and the limitation is precisely the kind this method is built to notice. An actor has a mailbox: one queue into which every message must be placed and out of which messages are taken in order. All of the agent’s contact with the world is therefore routed through a primitive serializer. The agent is concurrent with respect to its environment and strictly sequential with respect to its own perception.

That is a psychological stipulation, and a strong one. It asserts that experience arrives single file — that there is one place where things must show up in order to be had — which is the Cartesian theater given a formal specification [224]. It is also not how sense organs relate to brains. Retina, cochlea and skin transduce concurrently and continuously, project along separate pathways at different latencies, and are integrated, where they are integrated at all, without passing through any common queue; the binding problem is the problem manufactured by supposing otherwise [226, 225]. An agent whose sensorium is a mailbox does not have several sense organs. It has one, with a variety of message types.

The institutional reading is the sobering part. MCC existed as a consortium response to the Fifth Generation program; Rosette was built as a substrate for Carnot’s heterogeneous information integration [174]. The synthesis was funded as infrastructure for an industrial competition, and when the consortium wound down the lineage dispersed. This is a hard data point for the reception thesis: the most psychologically complete agent model of its period did not lose an argument. It lost a sponsor.

66.8.5 The reflective higher-order calculus

Applying the instrument to the calculus this book is built on is the one place where the method is being used by an interested party, and the reader should discount accordingly.

Before the changes specific to the rho calculus, one inheritance should be credited where it belongs: both the \(\pi\)-calculus and the rho calculus are concurrent inside the agent as well as outside it. A term may have arbitrarily many components running in parallel, so an agent may have many organs active at once with no queue between them and no privileged order of arrival. That is where both calculi part company with the actor model, and it is a better account of a sensorium than a mailbox is. What follows distinguishes the rho calculus from \(\pi\), not from actors.

Sense organs are made of agents. The \(\pi\)-calculus leaves the set of names an unanalyzed primitive. What a channel is made of is not a question the calculus admits, and the silence is deliberate. Two consequences follow, one ontological and one practical, and they are the same consequence seen from two sides.

Ontologically, the agent’s organs of contact are handed to it from outside the model, unexplained — which is Smith’s complaint about pre-individuated worlds (Section 67.3) arriving at exactly the point where it does the most damage, the point of contact. Practically, the calculus is not reducible to practice. Every implementation must supply an answer the calculus withholds: operating-system ports, network addresses, GUIDs, buffered typed queues, cryptographic keys. These answers are not equivalent to one another, and they are chosen by the implementer rather than determined by the model. The ontology must be relaxed in order to make contact with implementation, and what a \(\pi\) agent perceives with is settled during that relaxation, off the page.

In the rho calculus, names are quoted processes: channels are made of the same material as the agents that use them. This is the arrangement biology exhibits. A sense organ is tissue — cells, continuous with the organism, differentiated rather than interposed — and not a distinct substance placed between the organism and its world. The consequences are the ones biology also exhibits. Organs can be constructed, modified, transmitted and reasoned about with the same apparatus that describes behavior; an agent can grow a new one; and an agent can hold a representation of another agent’s organ, which is what makes a shared world negotiable rather than stipulated. It also closes the ontology in the sense of Observation 66.1: nothing has to be imported at implementation time, so the model determines its own realization instead of delegating that to whoever builds it.

Reflection as the source of names. Names are quoted processes; there is no \(\nu\). Privacy is therefore not a primitive granted by the model but an achievement of unguessability, derived from what no other party is positioned to name. Compare \(\pi\), where restriction hands the agent a secret gratis. The rho agent inhabits a world in which being unobserved is contingent and earned, which is the actual condition of an agent among others.

The other becomes representable, not merely contactable. In \(\pi\) one may send a channel; in the rho calculus one may send a quoted process — an account of how something behaves. This is the structural precondition for a theory of mind: another’s behavior as a datum that can be held, reasoned about, and passed on. Milner’s agent can speak with you; a reflective agent can tell a third party what you are like. Chapter 21’s hypotheses are quoted behaviors, and this is where the possibility of holding one comes from.

The tower becomes a loop. 3-Lisp’s regress is infinite and tamed only by laziness; in the rho calculus names and processes generate one another, so self-relation closes rather than ascends. The psychological difference is between a mind that can always be asked who was watching that thought, without ever reaching ground, and a mind whose self-relation is finite and complete. The engineering difference is that a loop can be priced and a regress cannot — which is what makes cost accounting, and hence mortality, available at all.

66.9 The reception of reflection

We can now apply the instrument to the reflection tradition itself, and the result is the cleanest instance of the pattern in the whole survey.

What was received into general practice is read-only reflection: the Java reflection API, RTTI, inspect, serialization frameworks, dependency injection. The agent may enumerate its own fields and methods. It may not alter its interpreter. Self-knowledge was institutionalized; self-determination lapsed. The structure survives in plain sight and no longer performs its original function — the properly vestigial case.

The metaobject protocol work [189] retained both halves and remained a specialist taste. The practical objection lodged against it was cost: reflection is slow. A community that will pay for speed and not for self-knowledge has stated its priorities plainly.

And the same author’s next contribution completes the picture. Aspect-oriented programming [190] is reflection inverted: advice is woven into code by a third party, without the participation of the modified component, and the selling point is precisely that the component need not know. The received descendant of Smith’s reflective procedure is a mechanism by which an external authority reaches into an agent and changes what it does — which returns us exactly to G\"odel, the metalevel as the observer’s privilege, with better tooling.

66.10 Neural networks: the agent as deposit

McCulloch and Pitts [35] wrote a logic, not a learner: all-or-none threshold units, and a demonstration that mind is mechanizable. Its reception went into von Neumann’s architecture (Section 66.3), not into any theory of learning. The neural half waited for Hebb [36], whose contribution is the one that matters here.

The agent’s structure is changed by its history. Nothing earlier in this survey has this. Turing’s table is given; Church’s term is given; a \(\pi\)-process is given. Hebb’s cell assembly is deposited by what has happened to it. That is a genuinely new answer to our question: to be an agent is to be the residue of everything that has befallen you. Character rather than program — and the first time in a century that memory appears as something other than storage.

Rosenblatt operationalizes it [235]. Minsky and Papert [232] constitute the reception event, and an unusually well documented one: a result about the limits of a particular architecture became, in transmission, a verdict on a research program. Whatever its authors intended, the effect is the clearest case available of selection operating on a community rather than on an argument. The parallel distributed processing program restores the line [236, 234] with explicitly anti-symbolic rhetoric: no central executive, no explicit rules, knowledge in the weights, graceful degradation.

The phenomenology of the trained network is then as follows.

Being and becoming are separate regimes. During training there is no agent, only an optimization; during inference there is an agent but no learning. The thing that acts cannot change, and the process that changes is not a thing that acts. Every earlier model in this survey at least let its agent’s activity and its agent’s constitution take place in the same world. Here the separation is constitutive rather than incidental — not a connection yet to be made, but the defining architecture of the field. This is the metalevel/agent split of the next chapter arrived at from the opposite direction, and it is why that chapter’s claim is structural rather than a complaint about tooling. It is also the exact negation of Theorem 25.1, where learning and inference are one relation because there is nowhere else for either of them to happen.

The world is a distribution, and order must not matter. Training samples are drawn i.i.d., and shuffling is not an optimization but a requirement. Church made independence of outcome from order an axiom (Section 66.2.3); the data loader now enforces it as an engineering necessity. The environment has no sequence, no consequences, and no capacity to respond to what the agent does. It is the single-relation ecology of the \(\lambda\)-calculus, industrialized: a corpus is food.

Others are represented and never encountered. Whatever a network knows of other agents, it knows from having consumed accounts of them. There is no interlocutor in the training regime, and the loss is the only value — scalar, external, and set elsewhere. This is cybernetic teleology with a gradient.

An interior it cannot inspect, which others can. This is the first model in the survey with a genuine unexamined interior: contents that are real, causally efficacious, unavailable to the agent’s own report, and progressively legible to an external investigator. Interpretability is the operator reading the unconscious. Note the exact inversion of Smith’s program (Section 66.8.3): there, the agent was given access to its own environment and its own continuation; here it has neither, and the metatheorist has both.

Identity is a checkpoint. Not intrinsic structure, and not behavior under probing, but a file — copied, forked, merged, rolled back, deleted. The multiplicity and the existence of the agent lie entirely at the discretion of whoever holds the weights. Weakening and contraction, revoked three times over in Section 66.7, are here restored in full and applied not to arguments but to agents. Cost is measured obsessively, but the agent does not pay it.

What traveled: the pipeline. What lapsed: the connectionist claim to be a theory of mind, with its arguments about rules and representations; and the constraint of biological plausibility, which was abandoned without much ceremony once the operational half began to pay. Hebb’s own insight is the specific casualty. That learning is something continuously happening to a living system now survives as a remark in the prehistory of an offline batch process, and the recent interest in mortal computation [50] is best read as an attempt to recover precisely what that reception dropped.

66.11 What the survey has delivered

Read as a sequence of stipulated phenomenologies, the century runs: an agent alone with an inert world (Turing); an agent for which the world consists entirely of what eats and what is eaten (Church); an agent with a body and a location but no company (Post); an agent that is a serial attention over a store, or a mortal reproducing thing in a physics, or a strategist among enemies, depending which von Neumann one reads; an agent with a rich interior and no world (Chomsky) or a world and no interior (cybernetics); an agent among peers, with secrets, introductions, and an identity conferred by recognition (Milner); an agent for which what it is given must be reckoned with exactly once, its omnipotence surviving only inside a declared region (Girard); an agent whose interior is perfectly private and reversible and for whom contact is collapse (Deutsch); an agent that is the deposit of its own history, with an interior it cannot read and others can (the connectionists); and an agent that can catch itself in the act (Smith), that can do so among peers though it must take the world in single file (Rosette), and that can do so with organs made of the same material as itself, in a closed loop to which a price can be attached (the reflective higher-order calculus).

Read as a sequence of receptions, it runs differently and more consistently: the operational half every time. The tape and not the education; the machine model and not Post’s empirical program; the bottleneck and not the automata; the hierarchy and not the conversation; the borrow checker and not the logic of action; the speedups and not the claim that computation is physics; the pipeline and not the theory of mind; the introspection API and not the reflective procedure.

That is the promise of Section 4.2 discharged, and i want to state exactly what has and has not been shown. What has been shown is that the third generation was not chosen for its power. Every ingredient of it was available and was passed over, repeatedly, by communities with no shortage of talent and no interest in suppressing anything, for reasons that were in every case operational. What has not been shown is that the third generation is right. A history of what a field could not bring itself to say is not an argument that the thing it could not say is true.

The next chapter turns the survey into a specification and asks what follows from it for the present moment, which is the one place where the answer might matter before the century is out.