Monodratic: a reproducible sparse attention mixer

Deyan Todorov
Misul Computing
Technical report, 10 July 2026

PDF · Code

Abstract. Monodratic is a stateless sparse attention mixer. Routed width 2 of 5 reaches 0.9909 mean accuracy on a three-seed associative-recall task. Natural-language quality, asymptotic linear construction, and fused-kernel speed are not claimed.

Mixer

The mixer receives a normalized residual stream, computes a sparse attention delta, and returns a tensor of the same shape. Residual updates, feed-forward layers, embeddings, output heads, cache policy, and inference scheduling remain outside the component.

After rotary position embedding, product routing writes keys into bounded causal posting lists and selects a fixed number of remote blocks. Local blocks are added separately. Attention over that set is ordinary causal softmax. No value summary replaces selected source values.

Monodratic data flow from normalized input to attention delta
Figure 1. Monodratic data flow. The router chooses source blocks. The attention operation over the chosen tokens remains exact. Accent outlines mark the owned steps.

Learned sparse route

The non-bypass experiment uses multi-query associative recall. Sequences contain six blocks. At the latest queries, five remote blocks are eligible and PHI selects two. The local-only control uses identical non-router weights and no remote route. The random control uses the same remote width and identical non-router weights. Three seeds.

Held-out associative-recall accuracy for local-only, random, and learned PHI routes
Figure 2. Learned routing solves the held-out associative-recall task. Open circles show three seeds. Filled diamonds show means. The learned route selects 2 of 5 eligible remote blocks. Every run has zero overflow and zero whole-call dense fallback.

Capacity

The capacity experiment uses SmolLM2-360M at 8,192 tokens. Before calibration, flattened post-RoPE query and key routing records 18 held-out overflow events at capacity 64. Spherical codebook calibration on the training split reduces held-out overflow to zero. The maximum assigned load after calibration is 11. At this length every remote block is eligible, so the run verifies conversion and capacity. It does not test sparse retrieval.

Maximum assigned load by layer before and after codebook calibration
Figure 3. Training-split codebook calibration removes held-out capacity failure. Points show the largest assigned load in each layer on one held-out 8,192-token sequence. Open marks are before calibration. Filled marks are after. Layer 24 exceeds capacity before calibration. After calibration the maximum is 11 and overflow is zero.

Finite-range scaling

The packed reference measurement includes index build and non-bypass search at 4K, 8K, 16K, and 32K tokens on four CPU threads. Each length uses one warm-up and three timed repetitions. Every length passes sampled scalar route and counter parity, causality checks, fixed candidate-slot checks, and zero-overflow checks. The fitted exponent is 1.004 over the measured range. The stable sort used during index construction remains O(N log N).

Packed routing time from 4K to 32K tokens versus a linear reference
Figure 4. Packed routing has near-linear finite-range timing. Points are medians of three timed runs on four CPU threads, in milliseconds. The dashed line is linear growth anchored at 4K.

Evidence

Quantitative evidence from the technical report. Scope is the mixer component.
MeasurementResult
Component tests429 passed
Associative-recall accuracy0.9909 mean, 0.9805 min
Margin over random routing0.4297 min
Margin over local-only attention0.7070 min
Target-block hit rate0.9362 mean
Selected-mask oracle error1.91 x 10^-6 max
Overflow0
Packed timing exponent1.004, 4K to 32K

Limits

The associative-recall task is synthetic and short. The 8,192-token capacity run uses an all-eligible selected set. The packed implementation is portable PyTorch rather than a fused kernel. The reported evidence does not support a general language-model quality or deployment-speed claim. Cache behavior is outside the mixer contract.