Long context changes the architecture problem.
At ordinary sequence lengths, a transformer can treat the prompt as a single field of tokens and compare every position with every other position. At millions of tokens, that strategy spends enormous compute rediscovering a simple fact that different parts of the past matter at different resolutions.
The latest tokens often need exact treatment. A symbol introduced hundreds of tokens ago may have to be reproduced precisely. Certain information is best carried forward as a compact state that changes with the sequence, while specific details are pulled back only when the current query needs them.
Quasar Long is built around that separation. Its context system combines exact local attention, compressed summary retrieval, recurrent matrix state, persistent memory slots, and token-addressed recall. Each mechanism operates at a different timescale, and the model learns how to blend them inside one decoder-only stack.
The result is a sparse 120-billion-parameter architecture with roughly 8 billion active parameters per token and a configured maximum context of five million tokens.
Benchmark preview
An internal benchmark preview places the current 120B checkpoint alongside several large reference models on MMLU, GSM8K, and BBH.
Until checkpoint identifiers and complete harness settings are published, these figures should be read as a progress snapshot rather than an independent reproducibility claim.
Long context is a memory problem
Quasar's context system is a learned memory hierarchy.
It relies on multiple mechanisms to handle different types of dependencies. Recent tokens remain exact, older blocks are turned into searchable summaries, recurrent state carries information forward, Raven stores selected details in memory slots, and Engram adds direct recall for familiar token patterns.
Each part handles a specific need:
- What happened just now? Exact local attention preserves the most recent 512 tokens.
- Which distant passage matters to this query? Compressed attention retrieves the 32 most relevant completed summaries.
- What has changed throughout the sequence? Gated linear attention carries a fixed-size recurrent matrix.
- What needs to be stored explicitly? Raven maintains 64 slots and selects 32 for each read.
- Have we seen this token pattern before? Engram maps bigram and trigram identities into learned features.
That split is the core design. Long context is handled by routing information across several forms of memory rather than extending one attention matrix.
Quasar at a glance
Quasar Long is a decoder-only sparse mixture-of-experts model. The main configuration uses a vocabulary of 157,184 tokens, a model width of 4,096, 16 physical blocks, 32 attention heads, and 128-dimensional heads.
Every physical block contains a sequence-mixing branch followed by sparse expert computation. The sequence branch changes according to a six-block cycle. Some blocks use compressed attention, some use recurrent state, some use persistent slot memory, and selected attention blocks also receive Engram features.
After the sequence branch, the MoE layer routes each token to two of 128 routed experts while also evaluating one always-on shared expert. The routed experts use SwiGLU transformations. Router logits are computed in FP32, selection can incorporate learned expert bias, and the final mixture weights come from the unbiased routing scores.
This layout separates two kinds of sparsity. The memory stack decides which parts of the sequence deserve resolution; the MoE layer decides which parameters should process the resulting representation.
Across 16 blocks, the cycle produces ten compressed-attention blocks, including two that receive Engram recall, three GLA blocks, and three Raven blocks. The pattern repeatedly returns to attention while inserting recurrent and writable memory between attention stages.
The stack can also support repeated passes through its physical layers. In the primary configuration the loop count is one, keeping the architecture direct. When repeated execution is used, persistent caches mutate only on the final pass, which preserves a clean distinction between internal refinement and sequence advancement.
One stack, multiple forms of memory
Each branch writes an update into the residual stream through a learned mixer. Rather than adding every branch output at full strength, Quasar learns how much of the current residual and the proposed update should participate.
For residual state r and branch update u, the mixer forms three bounded controls:
a = sigmoid(input_logit)
b = sigmoid(update_logit)
c = 2 × sigmoid(output_logit)
m = c × (a × r + b × u)
y = r + s × g × (m - r)Here, s is a residual scale and g is a learned gate. The form gives every branch a stable path back to the original residual while allowing the network to increase, reduce, or reshape the update.
That matters in a heterogeneous stack. An exact attention result, a recurrent read, and a slot-memory read have different numerical behavior. A shared learned mixer lets them enter the residual stream through the same controlled interface.
Exact attention where precision matters
Quasar preserves full causal multi-head attention over the newest 512 tokens. Queries, keys, and values are projected at full head resolution. Queries and keys receive per-head RMS normalization, and rotary position encoding is applied through the local region.
Beyond that region, retrieved summary keys remain unrotated. This short-RoPE, long-NoPE split gives exact local interactions a strong relative-position signal while allowing distant summaries to be compared primarily by content.
The local window is deliberately lossless. Code indentation, a recently introduced variable, a quotation boundary, or the exact wording of an instruction can be sensitive to individual tokens. Compressing that region too early would force the model to reconstruct detail that was still cheap to retain.
The window also bounds the dominant exact-attention term. Instead of allowing each new token to attend densely over a context of length T, its exact neighborhood stays fixed at 512 positions.
Compressing the distant past
Tokens that leave the local window enter a summary path called compressed sequence attention, or CSA.
During training, each completed block is pooled by a learned compressor. The model first predicts a scalar importance logit for every position in the block. A softmax turns those logits into pooling weights, and the weighted token representation passes through a value projection to produce the summary.
For token states x₁ … xₙ, the operation can be read as:
α = softmax(score(x₁ … xₙ))
summary = value_projection(Σ αᵢxᵢ)The summary is therefore content-dependent. A block can emphasize a definition, a decision, an entity, or a transition rather than reducing every token with the same fixed weight.
Each summary also receives a 64-dimensional index. The current query and all eligible summary indexes are L2-normalized, producing cosine-like retrieval scores. Quasar selects the best 32 summaries and attends only to them. Eligibility is strictly causal: a query can retrieve completed blocks from its past, never a block that overlaps its present or future.
The final attention result combines two views:
- Exact causal attention over the newest 512 tokens.
- Attention over the top 32 relevant summaries from older completed blocks.
A learned input gate controls their contribution before the output projection returns the result to the residual stream.
During incremental generation, the cache maintains the exact tail and an incomplete pending block. When that block finishes, it becomes a new summary and enters the searchable bank. In the current inference cache, completed CSA key/value blocks are summarized with arithmetic means, while the learned compressor above describes the differentiable no-cache training path. Cached HCA entries are means over groups of CSA summaries.
From CSA to hierarchical compressed attention
A flat summary bank works well until the bank itself becomes large. At multi-million-token lengths, even scoring every fine summary before a top-k selection adds noticeable work.
Hierarchical compressed attention, or HCA, introduces a coarser level. Fine CSA summaries are grouped into units spanning 1,024 source tokens. Retrieval can then operate over a smaller bank that represents broader regions of the sequence.
In the primary configuration, hierarchy activates automatically once context reaches 262,145 tokens. It can also be selected at designated repeated-stack steps when iterative execution is enabled.
The hierarchy preserves two important properties. First, the exact 512-token tail remains available, so increased context length does not lower local fidelity. Second, retrieval remains content-addressed. Older information becomes coarser, but it does not become a fixed recency-only cache.
This creates a natural resolution schedule: exact tokens nearby, learned summaries farther away, and coarser summaries across the longest spans.
Recurrent memory beyond attention
Retrieval is valuable when a query can identify a relevant region of the past. Some dependencies are better represented as an evolving state. Quasar uses gated linear attention, or GLA, for that role.
GLA projects each token into query, key, value, gate, and read vectors. Queries and keys pass through SiLU activations. A learned decay gate controls how much of the previous matrix state survives:
Sₜ = exp(gₜ) × Sₜ₋₁ + kₜ ⊗ vₜ
yₜ = qₜSₜThe readout is further modulated by a sigmoid read gate before its output projection. Because Sₜ has fixed dimensions, the branch advances in linear time with sequence length and carries a compact state across arbitrarily long prefixes.
This state behaves differently from a retrieved summary. A summary preserves a selected representation of a specific block. GLA continuously integrates evidence, making it well suited to quantities and patterns that evolve: topic state, repeated structure, accumulated constraints, or a latent progress signal.
Raven adds a complementary form of persistent memory. It maintains 64 learned slots and selects 32 for each token. A routed write candidate and write gate update the chosen slots with a normalized update rate. Reads use the routing weights to combine selected slot contents.
During parallel training, a blockwise exclusive prefix construction ensures that a position only reads memory written by earlier positions. During generation, the slots simply persist and update as new tokens arrive.
GLA offers dense continuous state; Raven offers a small writable workspace with selective addressing. Interleaving both with attention gives the model three distinct long-range operations: retrieve a passage, carry a state, or update a memory.
Token-addressed recall with Engram
Selected attention layers receive an additional recall path built from bigram and trigram hashes.
Quasar uses two n-gram orders with eight hash heads each, producing 16 table lookups per token. Each lookup returns a 16-dimensional vector. Concatenating the results gives 256 dimensions, which are projected to the model width and added to the residual stream with a small initial scale of 0.01.
This path turns short token patterns into direct addresses. It can surface phrase-level information without asking attention or recurrent state to rediscover that pattern from the active context.
The small initial residual scale lets the model introduce this signal gradually during training. Engram features augment the sequence representation, while attention still determines how current and distant contextual evidence interact.
Sparse computation after every sequence block
Memory efficiency alone does not make a large model economical. Quasar pairs the memory hierarchy with sparse expert computation.
For every token, the router evaluates scores for 128 routed experts, selects the top two, normalizes their weights, and applies a 1.5 output multiplier. One shared expert is always active, providing a stable dense path alongside specialization.
At the current scale, Quasar is reported at 120 billion total parameters with roughly 8 billion active per token. “Active” describes the routed execution footprint under the model's counting convention; it is not a direct measurement of FLOPs.
Expert parallelism distributes the 128 routed experts across four groups, while tensor parallelism splits expert intermediate computation across two partitions. The architecture therefore aligns model sparsity with distributed execution: memory branches bound sequence work, and expert routing bounds parameter work.
Persistent state during generation
Long-context generation becomes practical when the prefix has a durable representation. Quasar carries five kinds of state forward:
- The exact key/value cache for the newest 512 tokens.
- A pending buffer for the current incomplete summary block.
- Completed CSA and HCA summaries with their positions and indexes.
- Recurrent GLA matrices.
- Persistent Raven slots.
Prefill builds this state from the initial prompt. Every generated token then reads the current state, produces a representation, and updates the appropriate structures. Local key/value entries slide forward. A pending block eventually closes and appends a summary. GLA matrices and Raven slots advance every step.
The cache can also support paged prefix sharing, allowing common prompt prefixes to reuse their prepared state. The relevant unit of reuse is richer than a conventional key/value cache because it includes summaries and recurrent memory as well as the exact local tail.
The long-context cost model
Quasar replaces dense quadratic attention with several bounded or linear terms.
- Local attention costs approximately
O(T × 512). - GLA advances in
O(T)with fixed recurrent state. - Raven scales with tokens times the selected slot count.
- Engram performs a fixed number of table lookups per token.
- Summary count grows with
T / block_size, while retrieval attends to only 32 selected summaries. - HCA reduces the number of summary indexes scored at the longest contexts.
The summary search still grows with the size of its active bank before top-k selection, so the complete system has more structure than a single constant-memory claim. The hierarchy is what keeps that retrieval stage manageable as the context expands.
SILX's current internal preview reports the following throughput, per-GPU memory footprint, and estimated compute cost across four long-context operating points.
The practical gain comes from preserving precision selectively. Quasar spends exact pairwise attention on the region where exactness is most valuable and gives older evidence progressively more compact representations.
Training the memory hierarchy
The architecture is trained end to end with standard shifted-token cross-entropy. An auxiliary multi-token prediction objective adds a 0.1-weighted loss over four future offsets, covering targets two through five tokens ahead with a stride of two.
These objectives teach all memory paths through the same language-modeling signal. The compressor learns what a future query will need from a block. Summary indexes learn which blocks should be retrieved. GLA learns what should persist in continuous state. Raven learns when to write, where to write, and which slots to read. The routing layer learns which experts are useful for the resulting token representation.
That shared training signal is important: the hierarchy is not a collection of separate retrieval services around the model. It is part of the model's computation and is optimized with the model's predictions.
A model that chooses how to remember
The defining idea in Quasar Long is not simply a larger context limit. It is an architecture that chooses a representation for the past.
Recent evidence stays exact. Distant passages become learned, searchable summaries. Very long histories become hierarchical. Continuous dependencies flow through recurrent matrices. Selected information occupies persistent slots. Familiar token patterns receive direct learned addresses. Sparse experts transform the result without activating the whole parameter set.
Together, those mechanisms form a coordinated memory system: exact local state, retrieved summaries, recurrent matrices, persistent slots, and parametric n-gram lookup.
The five-million-token value is a configured architectural capacity. Model quality across that full span remains an empirical question for training and evaluation; the architecture defines how that experiment can be run without dense attention over the entire prefix.