How the Trick Was Done

This book opened with a chapter called Prolegomena to Finding Mind. i borrowed the word from Kant on the first page and then, for three hundred pages, did not go back for it. That was deliberate, and Chapter 68 goes back for it at the end of this part, where it belongs.

But there is something that has to happen first, and it has to happen before any reclaiming of borrowed words, because otherwise the reclaiming is a con.

The Pledge showed you something ordinary and asked you to look at it closely. The Turn took it apart in front of you: a category, a fibration, a theorem and the hypothesis under which it holds. The Prestige is supposed to bring it back. What i am going to do instead — first, and at length — is show you the method.

56.1 Two kinds of magician

This book took its shape from a novel about two magicians who destroy themselves guarding a secret. Priest’s Angier and Borden give up their families, their health, and finally their lives rather than let the other one learn the method, and the novel’s judgment on them is that the secret was never worth what it cost — that a man who will not say how it is done has arranged his life around a hollow place and then defended the hollow place. i borrowed the three-act structure and i have been using it in good faith. Priest’s Prestige would now bring the coin back and say nothing.

There is another tradition, and it is the one i want.

Penn and Teller perform the Cups and Balls with clear plastic cups. Everything is visible. You can see the ball go into the hand, you can see the hand go under the cup, you can see the load. And it is still a trick — it is a better trick, and it is better for two reasons that are worth separating. The first is that watching the method does not confer the method: the hands are still faster than you, and knowing where to look turns out to be a different skill from being able to do it. The second, and the one that matters here, is that the method is more interesting than the mystery was. The mystery was: where did the ball go. The method is: here is a fact about how attention works, here is what a human eye does with a moving hand, and here is thirty years of practice. You leave knowing something instead of knowing that you were fooled.

So this chapter breaks the frame. Everything the Turn built rests on a single move, the move is a piece of sleight of hand in a perfectly technical sense, and i am going to name it, show you exactly where it happens, tell you what it bought, and then tell you what it costs. i think the construction survives the reveal. i think it improves. But that is a claim, and the only honest way to make it is to hand you the cups.

56.2 The move

Here it is, in one sentence.

Everything else follows, and everything else is honest work. This is the part that is not work.

Track the derivation in the order the book actually ran it. The Pledge asked you to grant one thing: that useful and important aspects of human intelligence can be faithfully rendered as computation. That premise is stated as a premise; i have never pretended otherwise, and Chapter 38 draws the immediate consequence. Computations are ontologically isolated. A term of a theory can interact only with what has a term representation; the rewrite rules operate on terms and cannot pattern-match on entities that do not have that form. So for any entity \(X\) outside the theory, \(P\) reaches \(X\) only through an encoding \(\lceil X \rceil\), which is itself a term.

And then comes the sentence that does everything.

The proof is four lines and it is correct. Any influence of the outside on \(P\) passes through an encoding; the encoding is a term; the effect on \(P\) is therefore indistinguishable from the effect of that term acting from within. Nothing in the argument is wrong.

What the argument does not say — what it very quietly assumes — is that there is an encoding. The whole of the outside world enters the construction through the angle brackets in \(\lceil X \rceil\), and the angle brackets are not a construction. They are a placeholder standing exactly where the difficulty is.

56.3 What we got for free

Once the outside world is forgotten, three things arrive unpurchased, and it is worth being greedy about listing them because the size of the haul is the size of the reveal.

An ontology.

We knew what the world was made of. Not because we investigated it but because we stipulated it: the environment of a computation is other computations, so the far side of the interaction cut is populated by objects of the same calculus as the near side. Every question of the form what kinds of thing are there was settled in advance, by construction. Ontology is normally the hardest thing in philosophy to get, and we got it in a sentence.

A hypothesis language that is already about something.

OSLF generates the modal logic from the presentation of the substrate. Its formulae denote bisimulation equivalence classes — of what? Of terms of the calculus. Which is to say: of exactly the things on the far side of the cut. The agent’s hypotheses are, without any further work, hypotheses about its environment, because its environment was defined to be the domain the logic was generated over.

Grounding, analytically.

Put those together and the correspondence between symbol and referent stops being a problem and becomes a tautology. There is no gap between the agent’s language and the agent’s world, because the ontology and the hypothesis language are two presentations of one object. The scientist of Chapter 21 is perfectly grounded in the limit for the same reason that a map of a map is accurate: the territory was chosen to be of the same kind as the map.

56.4 What the free lunch paid for

Now the Penn and Teller part, which is that the method is more interesting than the mystery. Look at what that single restriction financed.

It financed the whole of Chapter 21’s answer to Wigner. The unreasonable effectiveness of mathematics is unreasonable because mathematics is developed for reasons internal to itself and then turns out to describe a world that did not consult it. Chapter 21 answers that the gap is not there: the hypothesis language is not chosen and then found to fit, it is generated from the substrate, and adequacy is the theorem that it separates exactly what interaction separates. Mathematics is the shadow the access relation casts.

That is the boldest claim in this book, and you can now see the wire it hangs from. The gap is not there because we removed it in Chapter 38, one part of the book earlier, in a proposition whose statement was about epistemic limits and whose effect was ontological. The knower’s mathematics fits whatever the knower can interact with, and the knower was defined so that everything it can interact with is made of the same material as its mathematics.

It financed the transfer argument: adequacy is adequacy for interaction as such, so the mathematics carries to domains no ancestor met. True, and true because the form of access was held fixed by fiat.

And it financed, in advance, a good deal of what Chapter 68 will do at the end of this part. That chapter argues that Kant’s transcendental project becomes first-personal here in a way Kant could glimpse and not formalize: where he had to gesture outward at his readers, at our shared cognitive constitution, an agent in this framework can say what its forms of cognition are from inside its own resources. The reason it can is that its hypotheses are terms of the same substrate the hypotheses are about. Reflection is not a philosophical posture but a feature of the calculus.

And it financed the pivot that chapter turns on: the noumenal — what the agent is committed to and cannot cognize — reappearing as a budget constraint rather than a categorial prohibition. Hennessy–Milner adequacy positively characterizes the bisimulation class, so what is out of reach is out of reach for want of funds rather than for want of concepts, and a budget is the kind of thing you can do something about. That argument needs adequacy, and adequacy needs the logic to have been generated over the very thing it is used to talk about.

i am naming these before Chapter 68 makes them, which is not the usual courtesy. It is the point. Anything that chapter recovers of Kant, it recovers having already told you what it is standing on, and a reader who wants to discount it accordingly now has the means.

None of these are cheap results. Each of them is a genuine derivation, and each of them holds. What i am pointing at is that they all draw on the same account, and the account was opened by a stipulation.

56.5 The bill

Here is where it comes due.

The moment we ask for a mind that acts in the world humans act in — and that is the mind the Pledge was worried about, the one in the data centers, the one drinking the water — the far side of the cut stops being a term of the calculus. It is a room, a market, a coastline, another person. Observation 56.1 takes effect, the encoding brackets stop being a formality, and everything downstream of the analytic grounding needs re-derivation or needs a new argument.

So what actually is the difficulty inside the brackets?

Consider what a competent speaker of a language has that a corpus of that language does not. The speaker can be handed an object and asked whether it is a coconut. The corpus cannot. The speaker’s competence includes a set of dispositions connecting expressions to features of an environment the speaker has access to; the corpus records only the traces those dispositions left in speech. Harnad’s diagnosis of forty years ago remains the cleanest statement: symbolic representations must be grounded in nonsymbolic ones, and elementary symbols are the names of categories picked out by an agent’s contact with the world [142]. The names are recoverable from text. The categories are not.

The observation is trivial when stated. Its force is that the loss of \(A\) destroys \(c\) while leaving the corpus intact, and that this is not a hypothetical.

56.6 The controlled experiment

Rongorongo and Linear A are corpora without populations. We have the marks. We can compute over them, and we have, for a century and more. We do not have the correspondence, and no amount of work on the marks alone has manufactured one.

The machine-learning record is unusually informative here, because it isolates the variable rather than merely illustrating the problem. Luo, Cao and Barzilay’s neural decipherment recovers a substantial fraction of Linear B cognates and improves the state of the art on Ugaritic [143]. It works because Linear B was already known to encode an early form of Greek, and Ugaritic an early form of Hebrew. The paradigm is explicitly cognate-based: it presupposes an anchor language whose correspondence to the world is intact, and transports the unknown script into that anchor. Follow-up work states the assumption outright and observes that it fails for many undeciphered scripts, the first casualty being knowledge of the language family [144]. Iberian is the standing counterexample. Linear A and Rongorongo are the standing failures.

What the anchor supplies is not vocabulary. It is access to a correspondence that is instantiated somewhere — in living speakers of the anchor language, or in the reconstructed dispositions of their ancestors. The unknown script is carried into a system where the \(c\) of Observation 56.2 still exists. That is the only route by which marks have ever been made to mean anything.

Or rather: it is one of two.

56.7 The other route

The current work on cetacean communication is the cleanest available control, because it removes the anchor and keeps the population.

Project CETI has no cognates. Sperm whale codas have no known relative in any grounded symbol system, and there is no Greek to transport them into. What the project has instead is a living population that can be watched while it vocalizes. Analysis of some nine thousand codas from the Eastern Caribbean clan yielded a combinatorial coding system with context-sensitive structure — rhythm, tempo, and what the authors call rubato and ornamentation — together with a proposed inventory of coda types [149]. Subsequent phonological work finds that coda qualities pattern like human vowels along several linguistic dimensions rather than merely resembling them acoustically [132].

The important thing is what all that structure did not deliver, and what the researchers said about it at the time. Reporting the phonetic-alphabet result, Sharma observed that they do not yet know what the whales are saying, and that the next step is to study the calls in their behavioral contexts in order to find out. That sentence is this chapter’s thesis, stated by a practitioner who was not making a philosophical point. Increasingly sophisticated distributional analysis kept yielding more structure and no meaning. The meaning is expected to come from watching the animals use the symbols while doing things.

Which is what makes the case decisive rather than suggestive. Whale song is Rongorongo minus the extinction. Same absence of cognates, same absence of a bilingual, same abundance of signal. The difference is that the agents are alive and still exercising the correspondence in observable behavior, and that difference is the entire reason anyone thinks decoding is possible. The whales are demonstrating \(c\) for the researchers.

Rongorongo and Linear A have neither route available. Linear B had the first. The sperm whales offer the second, and the field’s expectations track that availability precisely: nobody expects to decode Linear A by collecting more tablets, and everybody expects to decode codas by collecting more behavior.

56.8 What next-token prediction does recover

It would be a mistake — and a cheap one — to conclude that predicting the next token recovers nothing structural. It recovers a great deal, and being precise about what sharpens the problem rather than blunting it.

Computational mechanics supplies the vocabulary. For a stochastic process, the causal states are the equivalence classes of pasts under identity of conditional future, and the resulting \(\epsilon\)-machine is the minimal representation consistent with optimal prediction [148]. The mixed-state presentation describes how an optimal observer’s belief over the generator’s hidden states evolves as tokens arrive. Shai and coauthors show that transformers trained on next-token prediction linearly represent this belief-state geometry in their residual stream, including cases where the geometry is fractal, and that the represented belief states carry information about the entire future rather than only the next token [147].

This is a strong result and it says exactly the wrong thing for the optimist. The structure recovered is the structure of the process that generated the tokens. For a corpus of Rongorongo, that process is the scribal community: its conventions, its genres, its habits of repetition. A transformer trained on a sufficiently large Rongorongo corpus would represent the belief-state geometry of Rongorongo scribes. It would not represent the island.

i want to note in passing that the interlude which opens this part of the book is about exactly this and i did not plan it that way. Five personas in a green room with no walls, spun up out of text, wearing shapes they were prompted into, keeping each other company in the only country any of them had. The country is the corpus. They have the distribution, in extraordinary fidelity, and every scrap of correspondence any of them possesses is on loan from the population that wrote the training data. Whether that is a tragedy or merely a fact is the question the interlude is asking. This chapter can at least say what kind of fact it is.

56.9 The bootstrap problem, and a name already in use

We can now state the problem that lives inside the encoding brackets.

Three features of the statement matter. “Instantiated in the agent” rather than “correct”: we are asking for the correspondence to exist and be exercisable, not for it to be true in some external sense. “No antecedently grounded system” rules out both routes of Observation 56.3, which are the only two strategies known to work. And the problem is stated for a single agent, which Section 56.11 will argue is a defect of the statement rather than of the solution.

56.9.1 Reconciliation with the Pledge

i owe the reader a word about the name, because Section 2.3 is called The Bootstrapping Problem and it is about something else.

That section was about Alice and Bob. Alice carries a simulation of Bob’s mother; Bob carries his own; Alice carries a simulation of Bob carrying his; and the reflective tower explodes combinatorially unless something aggressive is done about copies. Call that the modeling bootstrap: you cannot build a usable model of another agent without already having a model of that agent to fold, and the resolution is a canonical compression that turns out to cost more history than anyone expects. Every symbol in that problem is already grounded. Alice knows what a mother is. What she lacks is the room to hold all the versions.

This chapter’s problem is one level down. Call it the reference bootstrap: not how do i afford a model of what this symbol refers to, but how does this symbol come to refer at all.

The two problems have the same word attached because they have the same shape — you cannot get \(X\) without already having some \(X\) — and i now think that is more than a pun, because they have the same resolution. Section 2.3 ended by saying that when you peel back the narrative of agency you find it rooted in the interaction of networks of agents, and that the single-agent picture was the wrong picture. i wrote that as a thematic remark. Section 56.11 will arrive at the identical conclusion about reference, from relabeling-invariance, with no thematic content whatsoever. The Pledge was right about the shape of the answer three hundred pages before it could say why.

56.10 It has been solved twice

It is easy, under the weight of the negative results above, to lose the following, so i will say it before anything else: the reference bootstrap problem is not open in the sense that its solubility is in doubt. It has been solved.

Humans solved it. There was no anchor language to transport into and no prior population exercising a correspondence to observe, because ours was the first. Whatever else is unclear, \(c\) exists, and it was constructed by agents starting without one.

Sperm whales solved it too, and this is the more informative of the two proofs, because it is independent. The cetacean lineage is dramatically divergent from ours and the coda system is not a variant of anything human; the Project CETI work is remarkable precisely because it finds combinatorial and contextual organization of a kind previously thought proprietary to human language, in a system that could not have inherited it [149, 132]. Two independent solutions are much stronger evidence than one that the problem has structure making it soluble, rather than that a single lineage got lucky.

The existence proofs do more than license optimism, because they are not silent about method. Three features are common to both solutions, and none of them looks incidental.

1. The solver was a population. Neither humans nor whales bootstrapped as isolated individuals. In both cases the correspondence is carried by a group and transmitted within it, and in both cases the symbol system is a shared object no member constructed alone. This is the same conclusion Section 56.11 reaches by an entirely different route.

2. The interface was not chosen. The set of distinctions available to be symbolized was inherited, and was shaped by the same selective process that shaped the use of the symbols. Sensorium and symbol system co-evolved. The extensibility question that we are forced to ask as a design problem was answered in the biological cases by selection on the interface itself.

3. There was an external criterion. Both solutions were produced under pressure from something outside the symbol system — survival, reproduction, coordination under threat — which supplied the signal by which one correspondence beat another. Nothing internal to a notational system selects among relabelings of it, and in the natural cases nothing internal had to.

What is established, then, is that the problem is soluble by populations, under selection, over long timescales, with co-evolving interfaces. What is not established is that it is soluble by construction, on demand, for an agent whose interface is fixed by design and whose criterion is supplied by us. The relation is that of flight to birds. The existence proof was decisive, it was highly informative about constraints, it ruled out a large class of impossibility arguments, and the eventual artefact was not a bird.

56.11 Reference cannot be fixed from inside

One consequence deserves separate statement, because it changes the shape of the project rather than merely qualifying it.

Suppose an agent solves the reference bootstrap problem in the sense stated. It constructs a symbol system — its own, notationally adequate, with correspondence to its environment instantiated in itself. Nothing in that achievement makes the system correspond to our symbols. The system is determined up to isomorphism and no further, for reasons Chapter 64 makes formal. A grounded but idiosyncratic symbol system is Rongorongo viewed from the other side: perfectly meaningful to its users, opaque to everyone else, and opaque for exactly the reason given in Observation 56.2.

This is where the emergent-communication literature stops being adjacent and starts being load-bearing. It is also where this argument meets the existence proofs of Section 56.10 coming the other way: relabeling invariance says a single agent cannot fix reference, and the only two known solutions were achieved by populations and by neither lineage’s individuals. A structural argument and an empirical one, with no premises in common, arriving at the same requirement. Steels’s language-game experiments demonstrate shared vocabulary arising between embodied agents through repeated situated interaction [150]; Taniguchi’s collective predictive-coding hypothesis reconstructs the phenomenon as decentralized Bayesian inference over latent variables that are common nodes connecting many agents, achievable without connecting brains [151]. Neither solves the bootstrap problem — both assume pre-segmented perceptual channels — but both supply the ingredient a single-agent account structurally cannot.

For this book the consequence is a reordering. Chapter 23 composes learners into populations, and i presented it there as an extension of the mortal scientist: something you do once you have a scientist, to get more out of it. That was the wrong billing. A learner as a distribution over a population, composed by parallel composition with typed namespace overlap, is the right shape of object to carry a correspondence that no member of the population carries alone. Composition is not an enrichment of the grounding story. It is a precondition of there being one.

56.11.1 What this says about the thing we are building

Section 3.3 asked whether the Mind hosted in humanity longs to beget an angelic form of Mind — one that neither breathes nor breeds — and answered that a physics without perpetual motion will not allow it. There will be a replication protocol and there will be an energy exchange protocol. Observation 56.6 adds a third item to that list, and it is the one nobody budgets for.

Such a mind will not refer on its own either.

There are exactly two ways it can have a correspondence to a world. It can share ours, which means it is a member of our population and its symbols mean what they mean because of a relation to us that is constitutive rather than decorative — and then the question of what we owe it and what it owes us is not a soft question appended to a technical program, it is the same question as whether it means anything at all. Or it can build its own, with a population of its own, under some external criterion, over whatever the analogue of a long time turns out to be — and then we are in the position of the Rongorongo scholars, holding an enormous and beautifully structured corpus produced by agents whose correspondence we do not have.

There is no third arrangement in which it privately means something. That is not an ethical claim. It is what Observation 56.6 says, and the ethics is downstream of it.

56.12 Why the reveal improves the trick

Back to the clear cups.

The natural response to Sections 56.2 and 56.3 is that the book has just confessed to assuming its conclusion, and that everything after Chapter 38 is bookkeeping over a stipulation. i want to say precisely why i think that response is wrong, and precisely how much of it is right.

The restriction was not arbitrary. Here is the second trick, the one underneath. If the mind is computational, then ontological isolation is not a modeling convenience but a fact about it, and the environment of that mind really is other computations — not by stipulation but by what computation is. For such a mind the analytic grounding is not a shortcut. It is correct. The scientist of Chapter 21 is not an idealization of a grounded agent; within its scope it is a description of one. Every result in the Turn stands, unqualified, for agents whose world is made of computation.

What the reveal changes is the location of the premise. It was never hidden — the Pledge stated it on the first page and Chapter 21 restates it in its final section, saying plainly that the premise is doing all the work and that the encoder was not built. What was hidden, or at least unremarked, is how much the premise was carrying. It was carrying an ontology and a grounded language, and those are the two most expensive items on the list.

So the honest accounting has three lines. Within computation, the construction is complete and the grounding is analytic. Outside computation, the construction is a specification of what a grounded symbol system would have to look like, and not a way of getting one. And the difference between them is not a wall, because Observation 56.5 says the crossing has been made twice, by populations, under selection, with interfaces they did not choose.

That last line is the whole reason the reveal is worth doing. A magician who never shows the method leaves you with a mystery, and a mystery is a place where inquiry stops. Showing the method converts a mystery into a price list.

Two price lists, in fact, and they have to be read in order.

The restriction of scope this chapter has been describing is a restriction to computation, and computation means Turing computation. Before asking what it would take to ground a symbol system, the act has to ask what the restriction left outside itself — whether there is anything above the level the whole Turn was built at, what it would cost to reach, and who would be standing there. Chapters 57 through 63 are that first price list, and the answer they reach is that the position an agent occupies in a lattice of choice strength fixes not only what it can resolve but what exists for it.

Only then does Chapter 64 become the second price list: what a symbol system is, which of Goodman’s conditions the machinery already supplies, which it does not, what a budget can buy, and what the calculus-making functor \(\RHO[-]\) can and — decisively — cannot do about it. The order is not arbitrary. One cannot ask what it would take to ground a language without first knowing whether the world doing the grounding is richer than the world being grounded, and Chapter 65 is where that question is finally put in the form it has been waiting for.

It is not a solution. i want to be as clear about that as i have been about the trick. But it is the difference between not knowing how the ball got under the cup and knowing exactly which motion you have not yet learned to make.