In the first part we stopped at an uncomfortable question. If a language model builds representations of objects and relations inside itself, why do these representations not exist as a stable structure the system can address independently of the specific token sequence?
We proposed a possible answer: add a layer of semantic poles — stable points to which the system returns before moving on. For the details, see part one of the series.
But drawing a map is easy. Getting the system to build it on its own — and to make sure the map reflects the structure of the data instead of becoming yet another set of hidden vectors with a beautiful name — is much harder.
How should the model know that two states refer to the same concept? Who decides where the boundary between two concepts runs? What happens to a concept whose meaning depends on context? And how do you train a discrete map when almost all neural-network optimization is built around continuous gradients?
This is where the architectural scheme ends and engineering begins.
The map is not given in advance
The first temptation is to do it simply. Take a dictionary of concepts, assign each one its own vector, make the model pick the matching one.
cat → pole_001 dog → pole_002 car → pole_003
But such a system almost immediately hits a limitation. Human concepts do not form a fixed dictionary. One word denotes different things. Different words — practically one object. And one concept consists of sub-concepts that in one context must be treated as a single whole and in another distinguished.
Take the word "key". A physical door key. A key in cryptography. A key in a database. A musical key. A key as an explanation of some question. If you create one pole KEY in advance, it turns out too coarse. If you create a separate one for each meaning — who decides where one ends and another begins?
A semantic map cannot be a dictionary. It must emerge from the dynamics of representations themselves.
What a pole represents
A pole does not have to mean a word. Or even a specific object. It can correspond to a stable configuration of features that regularly appears in the data.
Imagine the model repeatedly encounters situations: a person intentionally changes the state of an object with a tool. The specific sentences are completely different, but a similar structure of relations lies behind them. If it is stable enough, the system can form a coordinate around it.
Such a pole means not the word "action" but a stable pattern of relations. This is fundamental. If a pole is tied to a word — we are building a better dictionary. If it is tied to a configuration of representations — a map appears that works independently of formulation.
But the nearest vector guarantees nothing
Formally, everything looks familiar. We have a hidden representation z and a set of trainable vectors p₁, p₂, …, p_K. The system selects the nearest one:
z → argmin ‖z − pᵢ‖² → pᵢ*
This is vector quantization. The idea is not new.
But we do not want simply to compress a representation. We want a semantic coordinate system. The difference seems small, and the consequences are huge. If the task is compression, poles just need to cover the input distribution well. If the task is a map of meaning, the poles must acquire a stable interpretable structure.
And here the problem appears. Imagine that after training we look at the poles and discover: one corresponds to "sentences from 12 to 17 tokens long containing a numeral". Another — "texts where a colon appears frequently". A third — "a certain statistical combination of syntactic features".
Technically, everything works. There is no semantic map, though.
We need to distinguish clustering from semantic organization. And for that we need a test.
How to check that the map is truly semantic
One criterion is stability under paraphrasing. Two very different expressions describing the same situation should activate close poles.
"The flight was delayed because of heavy snowfall."
"The flight had to be postponed because of a blizzard."
Lexically the sentences differ. The causal structure is similar. If the map reflects semantics, a change of wording should not destroy the coordinates.
Another criterion is contextual separation. The word "key" in "he put the key into the lock" and "the system uses a private key" should lead to different states, despite the identical character sequence.
This gives a double check: the same meaning is preserved when the form changes, and different meanings are separated when the form stays the same. This is more substantive than perplexity.
Table 1. How to check that the map is truly semantic
| Test | What we check | Expected map behavior | If the test fails |
|---|---|---|---|
| Paraphrasing | Stability of meaning under change of form | Close formulations activate the same or a close pole | The map encodes the form of the text, not the meaning |
| Contextual separation | Distinguishing homonyms and polysemous words | "Door key" and "cryptographic key" land in different regions | The map is too coarse or ignores context |
| Concept hierarchy | Relations between levels of abstraction | "Taxi" is related to "car" and "transport" but not reducible to them | The map is a flat codebook |
| Transfer | Use of the already formed structure | A new task uses existing poles instead of creating new ones for each formulation | Poles do not function as stable coordinates |
| Editing | Separating semantics from changeable knowledge | Changing a fact does not require rebuilding the map | The map stores facts instead of a structure of representations |
| Ablation of Layer 1 | The real usefulness of the map | Removing the semantic layer noticeably degrades at least some tasks | Poles perform a decorative function |
The boundary problem
Suppose two states are very close. Where does one concept end and another begin?
car car with a driver taxi business-class taxi transport public transport
This is not a set of independent categories. Here there is hierarchy, partial overlap, context. For one question "taxi" is a car. For another — a commercial service. For a third — an element of the transport system.
If we assign a separate pole to every level, the map becomes fragmented. If we merge them, we lose information. So a semantic map, most likely, cannot be a flat array of prototypes. It needs a structure of relations between the poles themselves.
"Cat" and "mouse" can be close in usage context, but one is not a kind of the other. "Car" and "engine" constantly co-occur but stand in a part–whole relation. Euclidean geometry alone is not enough. The map needs not only coordinates but also relations between them.
Discrete choice breaks training
So far everything looked relatively calm. But now the system needs to select one of K poles, and the selection operation is non-differentiable. If the choice changed from pole 137 to pole 138, there is no smooth transition between them.
A neural network is used to receiving the signal: "change the parameter a little in this direction." Discrete choice answers: "either this one or that one." The gradient at the jump is zero almost everywhere. Training breaks.
Approximate methods have to be used.
Straight-through estimator. On the forward pass the system makes a real discrete choice. On backpropagation it behaves as if the operation were continuous and passes the gradient through. This is not the exact gradient of the discrete operation — it is an engineering approximation that lets the optimizer move inside a system with discrete choice. By itself it does not guarantee that the formed codes are semantic. It only makes training possible.
Commitment loss. If a hidden state has chosen a pole, it is additionally pulled toward that point. Otherwise the model will learn to use poles formally, passing almost all information through the continuous channel. Then the discrete layer will be present on the diagram but will not participate in the computation. Architecturally the layer exists. Functionally it does not. This is one of the most dangerous scenarios.
Sliding update. The poles themselves are not moved by gradient — that is unstable. They slowly drift toward the average position of all the states that have been attracted to them. The map changes, following the statistics of the data, rather than jumping with every optimizer step.
But here the opposite problem appears too. You can force the model to trust the poles too much. Then two states that belong to one broad concept but differ in details will be rigidly collapsed to a single pole, and part of the distinctions will disappear. In language a small detail sometimes changes everything: "he did not take the money" and "he took the money". One particle completely changes the claim.
That is why a semantic map does not replace the continuous representation. It coordinates it. The pole fixes the anchor. The continuous state preserves nuances. This is exactly why in the first article we separated Layer 1 and Layer 2.
Dead poles
Suppose we have 65,536 poles. The data distribution is uneven. Some regions occur constantly, others extremely rarely. After some time:
pole 001 → 18,400,000 activations pole 002 → 12,700,000 ... pole 48192 → 3 pole 48193 → 0 pole 48194 → 1
Some poles have practically stopped participating. We can write "65 thousand semantic coordinates" on the diagram, but in fact only a few thousand are used. An illusion of capacity.
The solution is to periodically re-initialize unused poles, moving them into regions with high representation density and poor coverage. But simply placing them at a random point is not enough. We need to look for places where there is a lot of data and existing poles cope poorly.
And here a fundamental trade-off appears. Too-fast updates destroy stability. Too-slow updates make the map unable to adapt. The model trained for a long time on scientific texts and formed stable regions. Then it starts receiving medical texts. If the map is fully plastic, old structures collapse. If it is almost immutable, new concepts are squeezed into old coordinates.
The classic continual learning problem: how to learn something new without destroying something old. For a semantic map it is especially acute, because the map must be more stable than the current context. Otherwise it turns into just another fast-changing hidden layer.
Perhaps the map needs different tempos of life. A fast level reacts to the context of the current dialogue. A medium level adapts to local shifts. A slow level stores stable semantic coordinates. Different types of information require different update speeds.
Table 2. Engineering problems of semantic poles
| Problem | What happens | Possible mechanism | Main risk |
|---|---|---|---|
| Discrete choice | An ordinary gradient cannot be passed directly through the pole selection | Straight-through estimator | Approximate gradient can lead to unstable training |
| Collapse | Too many states are attracted to the same poles | Commitment loss, constraints on map usage | Loss of distinctions between concepts |
| Dead poles | Part of the coordinates almost never activates | Re-initialization by regions of poor coverage | Destruction of an already formed structure |
| Drift | Poles move too fast in pursuit of the data | Slow update, EMA | Loss of stable coordinates |
| Catastrophic forgetting | New data changes the old map | Different update rates, protection of stable poles | New knowledge displaces old knowledge |
| Boundary states | An object is simultaneously close to several concepts | Soft assignment | The map becomes almost continuous again |
| Context dependence | The same object changes meaning in different contexts | Context-conditioned routing | Routing complexity grows |
| Map scale | One level cannot represent general and specific concepts equally well | Hierarchy of poles | Growth of complexity and connections |
Context complicates the choice
Suppose the system sees: "the bank was closed." Without additional context it is impossible to tell whether this is about a financial institution or a riverside structure. A pole cannot be chosen by a single word. A sequence is needed:
bank + account + credit + interest → financial context bank + river + shore + water → another meaning
The choice of a pole depends on the state of the context. A semantic map does not replace attention — it must interact with it.
And here the question of geometry appears. The concept "car" occupies not a single point: sports, freight, electric, old, autonomous. All are connected. A single pole may be too coarse. A pole can be imagined as a local region — a center plus the shape of the distribution. But the more complex the structure of one pole, the fewer benefits discretization gives. At some point, instead of a simple map we will build yet another continuous latent space.
Soft choice and its limits
When a state is roughly equally far from two poles, hard choice forces the selection of one. But the context may be on the boundary. Instead of a single pole, a soft distribution can be used:
car ├── 0.65 → transport ├── 0.25 → personal transport └── 0.10 → commercial service
At high temperature the distribution is soft. As it decreases, it becomes more discrete. This gives a mechanism: during training the map is soft, and during operation it gradually sharpens.
But if every object simultaneously belongs to dozens of poles, discreteness loses its meaning. Instead of one coordinate — a distributed continuous state. A balance is needed: a sufficiently compact choice, but not so hard that it destroys boundary cases. A trainable coordinate system with controlled uncertainty.
The map must pay for itself
Suppose all experiments show: the poles are stable, few are dead, paraphrasing preserves coordinates, context separates homonyms. Can we say that we have obtained semantic memory?
No. One last test remains: can the model use the map for new computations?
If the poles merely fix what the model already computed without them, they are an additional visualization. But if the system uses a pole as an intermediate state for new inference — the situation changes. The map becomes not a description of the model but a part of its computational mechanism.
That is why the main experiment is not about clustering. You can build a beautiful visualization, color the points by category, see clusters. And all of it proves almost nothing. Much more substantive is the transfer experiment: train the model, form the map, change the formulations, add new combinations of known concepts, check whether the model uses the old poles, measure the quality. If the map is a stable coordinate system, a new task should use the existing structure rather than create a new pole for every sentence.
And one more test — editing. If the architecture truly separates a stable semantic coordinate from changeable knowledge, then changing a fact in graph memory does not require retraining the pole layer. Only the corresponding segment of memory changes. One of the reasons for a separate map is to make the structure of computation more stable than the factual content.
And finally, ablation. What happens if the map is removed? If after removing Layer 1 the quality barely changes — the map was decorative. This is perhaps the most important test.
In engineering, a new layer does not earn the right to exist because it looks beautiful. It must provide something that is substantially harder to obtain without it. Otherwise it is simpler to keep a well-scaled Transformer. The proposed architecture does not have to prove that the Transformer is bad. It has to prove that the additional structure is sometimes worth its price.
One map or several
Semantics has no single scale. There are very general concepts: object, event, time, cause. There are intermediate ones: transport, economics, programming. There are specific ones: a processor model, a particular API, a particular drug.
If everything is placed in one flat map, it becomes either too coarse or monstrously large. It is more natural to use several levels:
abstract poles
│
domain poles
│
local poles
One object simultaneously has a coordinate at several levels. A Tesla Model 3 is a car, an electric vehicle, transport, a physical object. This already looks more like a hierarchical semantic system than an ordinary codebook.
And if poles are connected to each other, a graph emerges. If the graph grows large, it turns into a separate structure. We return to Layer 3. Layer 1 answers: "what coordinate system are we in?" Layer 3: "what specific entities and connections exist?" Layer 2: "what is happening to them right now?" Three different modes of computation.
So far, this is a hypothesis
We do not know the optimal number of poles. We do not know whether they should be points, distributions, or something else. We do not know whether one map or a hierarchy is needed. We do not know how to form relations between poles. We do not know whether a truly useful semantic structure will emerge from these mechanisms.
But now we have a concrete object of research. Not "understanding". Not "machine consciousness". But a trainable discrete structure that must preserve stable coordinates of meaning and use them in subsequent computations. It can be implemented. And it can be broken. In engineering, that is good news.
The question harder than all the previous ones
Suppose we have learned to build a stable map. The poles do not die out, they survive paraphrasing, context separates meanings. Layer 2 uses the coordinates.
The question remains: what to do when the map contradicts memory?
The semantic layer says: X is part of Y. Graph memory contains: X is not part of Y. What wins? The model receives a text that contradicts old information. Change the pole? Change the graph? Keep both versions? Take the source into account? Add a temporal coordinate?
A semantic map by itself does not solve the knowledge problem. It creates a place where this problem can be formalized. Memory must not just store connections — it must distinguish fact, source, time, and degree of trust.
In the next part we will examine exactly this: why graph memory is insufficient by itself, how changeable memory works on top of a semantic map, and why the problem of contradictory knowledge ultimately turns into a problem of time.