JupiteX Get the app
Science & Technology16 Nov 2026 · about 7 min

Understanding Positional Embeddings in Modern LLMs

The brief

A token embedding represents a token as a vector of features, but by itself it does not specify where that token appears in a sequence. Positional embeddings or encodings provide that missing location information. It matters because changing word order can change a sentence’s meaning, even when the same words appear. The article illustrates this with “The red fox jumps over the gray fox” and “The gray fox jumps over the red fox.” The words are similar, but their order changes which fox jumps. A positional method gives the model information tied to token positions so its transformer can distinguish the arrangements. Axiom adds position vectors to token embeddings before the first transformer block; RoPE works inside attention instead. The article introduces both classic and contemporary methods, including RoPE, and says modern architectures may adapt or omit explicit positional encodings. Its excerpt does not detail those alternatives. The central design choice is where and how a model receives position signals, which affects how it represents sequence order.

01

What are positional embeddings or encodings, and what information do they add to a token’s embedding?

A token embedding represents a token as a vector of features, but by itself it does not specify where that token appears in a sequence. Positional embeddings or encodings provide that missing location information. It matters because changing word order can change a sentence’s meaning, even when the same words appear.

The article illustrates this with “The red fox jumps over the gray fox” and “The gray fox jumps over the red fox.” The words are similar, but their order changes which fox jumps. A positional method gives the model information tied to token positions so its transformer can distinguish the arrangements. Axiom adds position vectors to token embeddings before the first transformer block; RoPE works inside attention instead.

The article introduces both classic and contemporary methods, including RoPE, and says modern architectures may adapt or omit explicit positional encodings. Its excerpt does not detail those alternatives. The central design choice is where and how a model receives position signals, which affects how it represents sequence order.

02

In the article’s example sentence, how many token positions are there, and why can two occurrences of “fox” have the same token ID but different positions?

The example sentence, “The red fox jumps over the gray fox,” has eight token positions, numbered 0 through 7. The key distinction is between token identity and token location: an ID identifies which vocabulary item appears, while a position marks where that occurrence appears in the sequence.

In the article’s small vocabulary, “fox” maps to token ID 2. It appears twice in the example, at positions 2 and 7, so both occurrences have ID 2 but occupy different positions. The model can use positional information to tell these occurrences apart in context, even though the token itself is the same. The article treats each word as one token for simplicity and notes that real tokenizers may split words into smaller subword tokens.

This distinction is part of the broader design problem the article explores: token embeddings represent the input tokens, while positional methods add information about their sequence locations. It compares Axiom’s position vectors with RoPE inside attention, but this excerpt does not describe how either method handles repeated words in detail.

03

What can a model miss if it has no positional information, and how can a causal mask provide some order information on its own?

Without positional information, a model can struggle to distinguish sentences that contain the same tokens in different orders. The article’s fox example shows why that matters: switching “red” and “gray” changes which fox jumps. A model that misses the ordering distinction may miss an important part of the sentence’s meaning.

In standard causal self-attention, a mask prevents a token from attending to later tokens. This means each position sees a different prefix: the token at position 3 can use earlier tokens, while the token at position 6 can use a longer context. That difference can provide some implicit order information, even without an explicit position encoding. It does not directly label each token with its exact position.

The article focuses on explicit positional methods and notes that Axiom adds position vectors while MiniMax-M3 uses RoPE inside attention. The causal-mask explanation is established transformer knowledge, rather than a mechanism described in this excerpt. Together, masking and positional methods can shape how a model handles sequence order.

04

How does Axiom add learned absolute position vectors, and how does RoPE add positional information inside attention?

Axiom uses learned absolute position vectors: it adds a vector for each token’s position to that token’s embedding before the result enters the first transformer block. This gives the model position information at the input to the transformer. The article contrasts that with a more recent architecture, MiniMax-M3.

MiniMax-M3 uses RoPE inside attention in each transformer block. In standard RoPE, position-dependent rotations are applied to pairs of dimensions in the query and key vectors. The difference between the rotations at two positions makes their attention score depend on their relative separation. That mechanism is established background knowledge; the excerpt names RoPE but does not provide its equations.

The article frames these as different ways to place positional information: before the transformer in Axiom, or within attention in MiniMax-M3. It says modern architectures may adapt or omit explicit positional encodings and previews positional choices in 2026 open-weight models. The excerpt does not list those choices, but it establishes that position handling varies across architectures.

05

What does RoPE do to query and key vectors so that attention can reflect the relative positions of tokens?

RoPE, or rotary positional embedding, injects position into attention by rotating query and key vectors. It matters because attention compares these vectors, and the rotation lets that comparison reflect where the tokens are in the sequence. The article identifies RoPE as a relative positional method and says MiniMax-M3 applies it inside attention.

In the standard mechanism, pairs of query and key dimensions are rotated by position-dependent angles. When a query at one position is compared with a key at another, the resulting dot product depends on the difference between their rotation angles. That makes the attention interaction sensitive to relative position. This mechanism is established background knowledge; the provided excerpt does not spell out the rotation math or implementation.

The article presents RoPE as a contemporary alternative to Axiom’s learned absolute position vectors, which are added before the first transformer block. It also says modern architectures may adapt or omit explicit positional encodings. The excerpt previews choices in 2026 open-weight models but does not give their names or specific methods.

06

Why is self-attention without positional information insensitive to token order, and what roles do query, key, and value vectors play?

Self-attention without positional information has no direct signal telling it which token came first or how far apart two tokens are. If the inputs are reordered, the attention computation follows the tokens rather than understanding their sequence positions. This can make it insensitive to word order, even though the article’s fox sentences show that order can change meaning. This is a standard property of self-attention without positional signals.

In attention, each token is transformed into a query, key, and value vector. A query is compared with keys to produce attention weights; those weights determine how much information from the corresponding values is combined. Without position information, those comparisons are based on token content, not explicit locations. Positional methods can supply the missing sequence signal.

The article points to two different placements: Axiom adds position vectors to embeddings before its first transformer block, while MiniMax-M3 uses RoPE inside attention. It presents RoPE as a relative positional method, but this excerpt does not explain query, key, and value computation. The distinction matters because architectures can inject position at different stages.

07

Why can handling sequences longer than those seen during training be difficult for a model’s positional method?

A model’s positional method must represent locations it encounters at inference time. If a sequence is longer than those seen during training, the model may have to use positional information outside its familiar range. How well it handles that depends on the method; the provided excerpt raises positional methods broadly but does not discuss long-sequence limits.

For example, a learned absolute-position table has a vector for each supported position. If it was trained only for a fixed range, later positions may have no learned vector unless the model adds or extends one. RoPE uses position-dependent rotations instead, but performance beyond the training range is not guaranteed just because the calculation can be extended. These are established technical considerations, not details stated in the excerpt.

The article compares Axiom’s learned position vectors with RoPE inside attention, and it says modern architectures may adapt or omit explicit positional encodings. It previews positional choices in 2026 open-weight models, but the excerpt does not identify their context limits. Evaluating longer sequences therefore requires checking both a model’s positional design and its demonstrated behavior.

This brief was written by AI from the original reporting and checked by other models. Names, figures and quotes come from the source; read it for full context.

Read more in the JupiteX app

Pulse is free. New stories every 4 hours, each one broken into the questions that explain it.

Or read more news on the web