Interlace: a shared architecture for language and iterative computation
Abstract. Interlace combines bounded rational feed-forward features, content-selected attention, and gated delta memory in a backbone that reuses its layers across successive computation steps. A single 29,814,640-parameter checkpoint supports text generation, synchronous streams, and experimental Flow-style refinement.
Architecture
Interlace is one trainable backbone. Bounded rational features provide nonlinear feed-forward transformations. Content-selected attention connects each row to selected earlier rows. Gated delta memory carries and updates a recurrent state. Shared depth applies the same backbone repeatedly, with learned loop identifiers. Synchronous streams represent inputs, intermediate states, and outputs in separate roles.
The text interface predicts the next token. The stream interface advances all three roles together and learns intermediate states on prefix addition modulo ten. Flow revises a candidate sequence over repeated passes. All three interfaces share the released model’s 29,814,640 parameters.
Results
The architecture comparison uses BF16 for both models, random initialization, seed 1001, identical data order, optimizer and task mixture, and 16,384 updates. Interlace has 29,814,640 parameters; the dense control has 29,921,280. Both use the same task interfaces and process 4,194,304 language target tokens.
| Measurement | Interlace BF16 | Dense BF16 |
|---|---|---|
| WikiText-2 test loss, nats/token | 4.7339 | 5.2516 |
| Time to 5.5-nat validation-monitor target | 276.52 s | 806.91 s |
| Stream answer accuracy, held-out lengths 2–6 | 100.00% | 100.00% |
| Stream answer accuracy, longer lengths 8–10 | 80.47% | 4.69% |
| Total training execution | 22.03 min | 22.26 min |
These are single-seed results under a common training recipe. Relative to the dense control, Interlace records 9.86% lower WikiText-2 test loss and 65.7% less training execution time to the 5.5-nat validation-monitor target.
Released checkpoint
The released checkpoint was trained from random initialization and contains all three interfaces in one parameter set. Packed weights occupy 30.76 MB, with FP8 weights and projection operands and wider working tensors. It is distinct from the BF16 architecture comparison above. Its WikiText-2 test loss is 4.7602 nats/token, 0.56% above the BF16 checkpoint of the same architecture. Stream answer accuracy is 100.00% on held-out lengths 2–6 and 92.97% on longer lengths 8–10.
Limits
The architecture comparison is a single seed under one training recipe. Time to target includes compilation, data handling, validation, checkpoints, and retries. The paper and system card report the full method, the inference-latency tradeoff, Flow results, and evaluation scope. Natural-language quality at larger scale is not claimed here.