There is a moment that almost inevitably arrives after several months of working with modern language models. At first you treat them as a very powerful tool: you ask a question, get an answer, refine it, ask for a rework, check it. Then gradually you notice that the tool starts behaving nothing like an ordinary program. It holds the thread of a conversation, returns to earlier arguments, changes its point of view on request, can explain the same thing at different levels of complexity. Sometimes it even notices its own mistake — and sometimes, with exactly the same confidence, it argues for two things that contradict each other.
This last case is the most interesting.
An error by itself explains little. Any complex system makes mistakes. What matters far more is understanding why the model can build a convincing representation of some situation and then, continuing the same conversation, build another representation incompatible with the first. If inside there really exists some stable state that could be called a representation of meaning, why does it sometimes change so easily?
The question can be put even more sharply: where exactly in such a system does what we call meaning live?
Not in the philosophical sense. Not in an attempt to determine whether a machine "understands" a human.
In the architectural sense.
Because if we want to build the next generation of AI, at some point we will have to answer not only the question of how well a model continues text, but also the question of what kind of structure it builds along the way.
A machine that keeps moving forward
The basic idea of a language model is surprisingly simple. At each step it receives some context and estimates the probabilities of the next token. The chosen token becomes part of a new context, and the computation repeats.
If we strip away implementation details, we get an almost mechanical sequence:
context ↓ computation ↓ probability distribution ↓ next token ↓ new context ↓ next step
At the level of a single operation there is nothing here resembling human reasoning. But it is precisely from the repetition of this operation that a system emerges which can write programs, retell scientific papers, construct proofs and sustain long dialogues.
So it would be a mistake to say that a language model "just guesses the next word". To predict the next token well, it has to build a huge number of hidden dependencies. It must somehow distinguish objects, relations between them, context, style, causality, temporal order and many other things that are not directly contained in the next token.
From a very simple local operation, an extremely complex internal geometry gradually grows.
A simple rule does not mean a simple system.
But here an interesting question appears. If such a geometry really emerges inside, why does it exist in exactly the form in which it exists?
Between text and what text means
Imagine a person explaining how to get somewhere in an unfamiliar city.
They might say: leave the building, walk straight to the intersection, turn right, cross the road after two hundred metres, and keep going until you reach the square. If you follow the instructions exactly, you will most likely get where you need to go.
But now try asking a different question: what lies between two districts, can you get there by another route, what changes if the bridge is closed? A sequence of instructions is no longer enough.
You need a map.
A map is not a more detailed instruction. It represents the city differently. What matters in it is not so much the order of actions as the relations between objects. Thanks to this you can build a route that nobody described in advance.
Something similar happens with text.
The sentence:
"Peter took the key from the table, opened the door and entered the room"
exists as a sequence of words. But the situation it describes has a different structure. It contains a person, a key, a table, a door, a room and several relations between them.
┌── was on → table
│
Peter ── took ──→ key ──┘
Peter ── opened ──→ door ── leads to ──→ room
When a person answers the question "where was the key?" a minute later, they do not have to repeat the original sentence. They refer to some already formed representation of the situation.
A language model has no such simple separation.
This does not mean it has no internal representations. On the contrary, they are extremely rich. But they are distributed inside the model's overall computational dynamics, and it is not obvious whether there is a separate object there that could be called a stable semantic map — something that can be checked, changed, or used as a coordinate system for the next computation.
This is where the architectural problem appears.
Why not just make the model bigger
The strongest answer to all this sounds very simple: scaling still works.
And it is a serious argument. The growth of compute, data and model size has consistently produced new capabilities. It would be strange to declare the existing paradigm exhausted just because we would like a different architecture.
But there is a significant difference between two claims.
First: a larger model can possess new capabilities.
Second: increasing the model necessarily creates a new form of meaning representation.
The first we observe. The second still needs to be proven.
So it is more reasonable to speak not of "the end of the Transformer" but of a possible structural limit. If the same basic scheme keeps scaling without changing how state is organized, then at some point it may turn out that adding parameters solves quantitative problems but does not change the nature of the computation itself.
In our architecture this is exactly what remains a hypothesis, not an established fact.
It can be stated simply: perhaps there is a limit after which it becomes more advantageous for the system not just to enlarge its existing representation space, but to add a separate layer of stable structure to it.
And then a natural question arises: what should that structure be?
Poles instead of endless fog
Imagine the model's representation space as a vast landscape. In it there are regions where states close in meaning sit nearby, directions along which a representation can move depending on context, and many transitions between them.
The problem with such a space is that it is continuous.
That is both its strength and its weakness.
Continuity lets the model distinguish similar concepts very finely. But if we need to say that some concept must persist as a stable coordinate, a difficulty appears. We have a region of space, but no obligatory point the system must return to.
That is why the first layer of the proposed architecture introduces semantic poles.
A pole is not a word and not a ready-made answer. Rather, it is a stable point in representation space meant to denote some concept or constraint. The input state is first matched against such poles, and only then passed into continuous processing.
This is a small but principled change:
input
│
▼
┌─────────────────┐
│ pole lookup │
│ in the semantic │
│ map │
└────────┬────────┘
│
▼
stable
semantic anchor
│
▼
further processing
It is like a coordinate grid on a map. The grid does not tell us everything about the city, but it lets us determine where exactly we are.
And this is where one of the most unpleasant engineering problems appears.
How to train a discrete map
An ordinary neural network is well suited to continuous change. Parameters shift slightly, the error changes, the gradient shows the direction of further movement.
But a pole is a discrete object.
If the system has chosen pole No. 137, it cannot choose "137.3". It can choose 137 or 138.
This creates a gap between how the system makes decisions and how it is normally trained.
One way around the problem is the straight-through estimator. On the forward pass the system makes a real discrete choice, but during backpropagation it pushes an approximate gradient through the operation as if it had been continuous. This lets you train the system, although mathematically it does not make discretization continuous.
Another mechanism is commitment loss. If a hidden state has chosen a particular pole, it is additionally pulled toward that pole. Otherwise the map can gradually turn into a set of formal indices poorly connected to the states they are supposed to represent.
Finally, the poles themselves can be updated through the statistics of the states attracted to them, for example by moving average. Then the map does not jerk with every individual optimizer step but changes more slowly.
All of this sounds quite engineering-like.
And that is good.
Because now an experiment can be run.
The map has another problem: it can start to empty out
Suppose we have created a large number of poles. After long training, some of them are used constantly, while others are almost never activated.
This is a natural problem of any discrete map. If a certain region of data space appears rarely, the corresponding pole may gradually lose contact with real states.
It stays in the system's memory but stops doing work.
These are dead poles.
The proposed revival mechanism is fairly intuitive: if a pole has not been used for a long time, it can be moved to a region with a high density of representations that existing poles cover poorly. In this way the map tries to redistribute its limited number of coordinates to where they are actually needed.
But there is a fine line here.
If poles move too fast, the map will constantly forget old structure. If it is made too stable, it will stop adapting to new data.
This is almost a biological problem: a system must preserve its shape and at the same time be able to reconfigure.
And now another question arises: what if different kinds of meaning require completely different behaviour from such a map?
When a single map becomes too coarse
Consider two concepts.
The first is a technical term, for example a specific programming mechanism.
The second is a constraint that the system must not violate regardless of the task.
Both can be represented as concepts. But the requirements for them are completely different.
A technical term must interact freely with many neighbouring concepts. Its meaning can be refined by context, depend on the programming language, the library version or a particular piece of code. Flexibility is useful here.
A constraint works differently. If it really is a constraint, you do not want to reinterpret it every time together with the surrounding context.
If both types of information are placed in one space and forced to learn in the same way, we get a conflict. Rare but important constraints will compete with a huge amount of domain representations, while overly rigid constraints will start interfering with normal context work.
So the architecture separates two streams.
One works with stable constraints.
The other works with domain content.
This separation concerns the logic of Layer 1 specifically: first the system determines a stable semantic region and constraints, then passes the state further. Layer 2 does not replace this mechanism; it continues processing an already prepared representation.
In the constraining stream, a pole can act as a mask: it removes a forbidden direction, leaving the other components of the domain representation almost untouched.
The result is not so much a second neural network as a separate control loop.
We examined the question of who and how approves an agent's actions at the system boundary in detail in our earlier article "From Executor to Organization: Who Approves an Agent's Actions".
Continuity comes back anyway
If the whole architecture is made discrete, it will lose much of what makes modern models useful. Language is full of nuances, partial matches, contextual dependencies and weak links. You cannot decompose all of this into fixed cells in advance and expect the system to keep its former flexibility.
That is why, after a semantic anchor is chosen, Layer 2 returns the computation to a continuous space. There remains room for ordinary neural network dynamics — with its nonlinearity, attention and ability to distinguish fine shades.
The difference is only that this dynamics no longer has to be responsible for the whole architecture on its own.
It has coordinates relative to which it can work with local context.
Memory does not have to be hidden in weights
A similar story happens with memory.
A modern model can store huge amounts of information inside its parameters, but parametric knowledge is poorly suited to point edits. If a person needs to add a new fact, delete an old one or change a specific relation, it is not always reasonable to retrain the whole model for that.
It is much more natural to move the changeable part of knowledge into a separate structure.
This is how graph memory appears.
In it, entities, properties and relations exist explicitly. If a person changed jobs, you can change the corresponding relation. If a source turned out to be unreliable, you can mark it or delete it. If a new entity appears, you can add it.
This is a completely different way of working with knowledge.
But a graph brings with it a problem that does not exist on a pretty architectural diagram.
The processor loves sequences. The graph loves jumps.
When a system walks a graph, neighbouring nodes may be physically far apart in memory, so graph traversal quickly becomes a problem of latency and cache access.
That is enough for the first article. We will leave the details of how such memory can be physically organized for a separate discussion — and how biological memory becomes an engineering architecture is covered in our article "Beyond the Context Window — Engram Engineering for Productive Agent Memory".
Three different kinds of computation
Putting everything together gives a rather unusual construction.
The first layer must store stable semantic coordinates and constraints.
The second handles flexible neural processing.
The third stores changeable knowledge structure.
INPUT
│
▼
┌─────────────────────┐
│ Layer 1 │
│ semantic poles │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Layer 2 │
│ local neural │
│ context │
└──────────┬──────────┘
│
▼
┌─────────────────────┐
│ Layer 3 │
│ graph memory │
└──────────┬──────────┘
│
▼
OUTPUT
On paper this looks simple.
But behind this simplicity lies a fairly serious change in architectural logic. We no longer assume that a single computing environment must perform all functions equally well. For each part of the system you can look for its own mode of operation and even its own physical implementation.
And this is where photonics enters the story.
The dense matrix operations of Layer 2 fit well the type of computation for which photonic accelerators have long been considered. The deterministic logic of Layer 1 and memory work, by contrast, naturally remain on electronic systems.
This gives a hybrid:
SYSTEM
│
┌─────────┴─────────┐
│ │
digital part photonic part
│ │
Layer 1 + Layer 3 Layer 2
│ │
└─────────┬─────────┘
│
common loop
And here another story begins.
Because as soon as we stop requiring one type of computing substrate to do everything at once, it becomes possible to match the structure of a task with the structure of physics.
But the map does not mean understanding
At this point it is very easy to draw too big a conclusion.
You might say: we added semantic poles, external memory, separated the computational streams — so we finally got a system that understands.
No.
We have proven nothing of the kind.
We have not even proven that this particular architecture will be better than a well-scaled Transformer.
But we have done something more useful.
We turned a vague discussion of "understanding" into a set of engineering questions.
You can test how stable the poles are. You can measure how many of them become dead. You can compare the model against a continuous baseline. You can check whether quality survives after stream separation. You can measure the cost of graph memory and test how well changeable knowledge is actually edited without retraining the whole system.
Instead of the question "does the machine understand?" appears a question an engineer can take into their hands.
What structure does the machine build — and what can it do with that structure?
This is where the next part begins
Everything in this article so far has been a discussion of principle.
But a principle by itself is almost worthless if it cannot be turned into a working mechanism.
We can draw a layer of semantic poles on a diagram. We can call them coordinates of meaning. We can separate the domain and constraining streams.
But where does the pole itself come from?
Who decides that two states belong to the same concept? Why should one pole be close to another, and a third one not? What happens when a concept changes meaning depending on context? And how many poles does a system need that must work not with a single domain but with language as a whole?
This is where the beautiful architectural diagram ends.
Engineering begins.
In the next article we will examine how a semantic map can form, why it inevitably faces the problem of discrete learning, and what happens if the map starts to forget its own coordinates.
Series "Three-Layer Semantic Engine": part 1 of 3.
This article is the introduction: from next-token prediction to the architectural hypothesis of three layers. Part 2 will cover how the semantic map itself forms, how to train discrete poles, and what happens when the map starts to forget its own coordinates. Part 3 is about why a graph by itself is not yet memory: time, sources, trust and contradictions — and why the problem of contradictory knowledge turns into a problem of time.