Finding the Shape

4.1 The Shape of Mind

We now return you to your regularly scheduled reading. Let’s take stock of where we are in the campaign. Our discussion is organized in three main sections: the pledge, the turn, and the prestige. We are about a quarter of the way through the pledge. We have covered a lot of territory, noting that since this story is about a computationally based theory of mind we need to update the version of computation the reader is statistically likely to be carrying in their mind. Part of updating the version of computation is to address the relationship of autonomy (independently (co-)operating computations) and compositionality (making bigger computations out of smaller ones).

Another aspect of the story is that computations are ontologically isolated. They are effectively in their own little bubble. So, the story only needs to focus on what is it like to be a computation in an environment of other computations. From this perspective we might consider how a program begins to build a picture of its environment and itself. For example, how might it classify other computations and their behaviors? How is that classification related to experiments our plucky little program might perform to test hypotheses about its environment or itself?

This last set of questions opens up a surprising twist in the story. First, when we get into the mathematical account we will discover that there is an objective ontology for all the computations in a given model of computation. There is a real world where ontology and epistemology are perfectly aligned. Further, there is a true language whose semantics is perfectly aligned with this real world. These facts should be surprising as they fly in the face of in-grained cultural relativism. However, the vagaries of passing messages in conditions where errors may occur provides an intriguing obstacle to ferreting out that real world using only experiment-based probes.

What those obstacles look like and how they play out in a computation developing an ontology through an experiment-based epistemology is what we’re going to focus on in this section.

Waaaay back in April of ‘25 i was at an event in San Francisco hosted by Dr Ben Goertzel. Ben and i got into a discussion about how intelligence scales. For the discussion Ben took the position that there were no limits, apart from the laws of physics, on the scaling of intelligence. i was more skeptical of this position. My mind was turning to Geoffrey West’s scaling laws for natural and human-made systems and i began to wonder of something like these laws might apply. Our conversation in San Francisco was spirited and engaging, but inconclusive. So, i continued to mull over the question.

By November i had the glimmer of an answer. Yes, there are scaling limits and they have to do with population densities and the propagation of messages. Here’s how the argument goes. Imagine a massively multi-agent system, something like on the order of the number of quarks in the known universe. If you believe the LLMs, this is just a couple of orders of magnitude shy of a google. The propagation of a message from one agent to another is subject to corruption. The farther the message has to travel the more likely the message is to get corrupted. This puts limits on the size of a population that can come to agreement on things. It’s a kind of Tower of Babel or Whisper Game phenomenon.

So, what happens is that populations of agents will form clusters roughly the size of how big they can be and still faithfully communicate information amongst themselves. For the purpose of this argument, let’s say that each cluster has enough internal structure to form a “super” agent, an agent made of agents. When we are talking a google of agents even the population of “super” agents will still be too big for messages between “super” agents to be faithfully passed around the entire population. So, the process will repeat, forming clusters of clusters, “super super” agents. Indeed, with a population the size of the quarks in the known universe, and certain assumptions about the rate of message corruption, we see roughly the same sort of hierarchical clustering as we see in the visible universe: galaxies made of solar systems, solar systems made of stars and planets, planets hosting populations of organisms, organisms made out of populations of cells, cells made out of molecules, molecules made out of atoms, atoms made out of electrons, protons, and neutrons, and these made of quarks.

But the story doesn’t stop there! As we will see in the turn, where we formalize these intuitions, to each massively multi-agent system we can associate a latent topological space, a kind of abstraction of continuity and containment. When this latent topological space is compact, that is when it enjoys certain kinds of boundedness properties, then the agents can always come to agreement. So, the massively multi-agent system will be clustered into hierarchical clumps of compact subspaces of the whole system. These will “glom” onto each other until adding more breaks compactness.

What does this mean for an agent in the middle of all this attempting to make sense of its world? Imagine a few quadrillion quarks have assembled themselves into a structure that can support intelligence. This little community is extremely tiny by comparison to the community of a google of quarks. When it looks out at the rest of the community what will it see? It will see this hierarchically organized clustering of quark-agents. Using only experiment-based reasoning its ontology will be very far from the real ontology implied by the model of computation this community of agents was written in. Moreover, its ontology will be very misshapen. Why? Because it will mistake the clusters or “super …{} super” agents for “things” instead of the bisimulation equivalence classes, i.e. the collections of programs that are indistinguishable by experiment. If, by some stroke of luck, this agent bootstraps a language based on its ontology, again that language will be full of concepts and memes that have nothing whatsoever to do with the real world and the true language.

i don’t know about you, Dear Reader, but i find this fascinating. i was not expecting any of these results when i set out to find Mind. i can’t wait to see what happens next!

4.2 Which Computer?

There’s a promise i’ve made several times now and not yet kept. i keep saying that the version of computation most readers are carrying around is several releases behind, and that we need to update it before the rest of the story can be told. Fair enough. But an update to what? Let me at least name the destination, so that when we arrive in the Turn you’ll recognize the place.

There are, broadly, three generations on offer. The first is the one everybody learns: computation as a function. You put something in, you turn the crank, something comes out. Turing machines, the \(\lambda\)-calculus, your undergraduate programming course. It is a superb account of what a lone box does, and it has essentially nothing to say about two boxes talking to each other, which — as we established several sections ago — is what a computer actually is.

The second generation fixes that. Here computation is interaction: the primitive act is not applying a function to an argument but two processes synchronizing, and what a system is is characterized by what it can do in the presence of other systems. This is where bisimulation comes from, and where our can-do formulation of “anything Alice can do Bob can do” lives. Milner’s CCS is the classic; the \(\pi\)-calculus adds the ability to pass channels around, so that the shape of the network can change as it runs, which is the mobility we noticed in FPGAs and in the Internet itself.

The third generation is where this book lives, and the extra ingredient is reflection. In the rho calculus — the reflective higher order calculus, named partly as a pun, since rho comes after pi — a program can be quoted to yield a name, and a name can be dropped to yield back the program. Code and address are two views of one thing. This sounds like a small piece of syntax and it is not. It is what makes the fractal picture of the first chapter available: if a network of agents is itself the sort of thing that can be named and passed around and interacted with, then networks of networks are not a new kind of object requiring a new theory, they are the same object again.

And it is what lets the model do the thing i insisted on when we were complaining about Wheeler’s rods and clocks. If your measuring apparatus has to sit outside the model, your model is not a model of the world; it is a model of an experiment being conducted by somebody who isn’t in the world. To put the apparatus inside, the calculus has to be able to represent its own machinery as ordinary data — and to do it cheaply, in one step, without an interpreter standing by to decode. The \(\lambda\)-calculus can be made to encode such things, but expensively, and CCS essentially cannot manage it at all. Reflection makes it a single move.

So that’s the destination. Interaction rather than function; bisimulation rather than input-output equality; and reflection, so that the model can hold a picture of itself without stepping outside itself to do it. Everything in the Turn is built on those three, and if you keep them in mind the rest is commentary.