A small language model trained on Tenstorrent hardware.

It hears "faster than light" and reaches for lightning — the nearest thing in a corpus that contains a great deal of weather and very little physics. Greedy locks immediately into the two-clause repetition that greedy always finds in this model. At 0.8 it collapses into pure affirmation. At 1.0 it produces the most interesting line of the three, which is oracular in shape and empty in content, and that is roughly where this model lives.

Nothing in the output notices that the request was a request. Nothing declines the premise, because nothing in 392M tokens of Gutenberg and TinyStories has ever declined anything.

Source: episod-log.md, 2026-08-16, model artifacts/hf-tt-tnt-1024a. These are ad-hoc samples from scripts/evaluate.py --try, which writes to a scratch directory outside docs/ and outside git precisely so nobody mistakes them for a measurement. The comparable numbers come from two digest-pinned frozen prompt sets, described below.

1. Corpus and tokenizer

   A nine-source, licence-audited blend built to a 400M-token budget, and a 32,000-token byte-level BPE tokenizer trained on that blend rather than inherited from someone else's vocabulary. Both are CPU work.

2. Trained from random init on Blackhole, with ttml

   tt-train's own trainer, its optimizer, its checkpointing, and its Llama implementation. The architecture is vendored as a verbatim copy of tt-train's nanollama3.yaml so it cannot change under a tt-metal upgrade without a test failing.

3. Converted to Hugging Face format, and checked numerically

   The conversion is verified against an independently derived pure-NumPy reimplementation of ttml's forward pass, written from the C++ source rather than from the converter, so the two paths reach logits by different routes. They agree to a maximum absolute logit difference of about 6e-6. This mattered: an earlier conversion loaded cleanly, tied its weights correctly and generated fluent prose while computing the wrong function, because of a RoPE row-layout mismatch worth 1.3 nats. Only the numerical comparison caught it.

4. Packaged with tt-model

   A manifest declaring the runtime, the mesh, the entry point and the ttnn version floor, plus a serving adapter that travels with the bundle.

5. Served through the Tenstorrent vLLM plugin

   tt-model serve episod/tt-tnt. The plugin logged the bundle's adapter registering out of the extra-models directory, the mesh opening at (1, 1), and /v1/models reporting the manifest's declared context on the wire. The procedure — and the traps that make it more than one command — is written up in docs/serving-with-tt-kernel.md.

Parameters (1024a)

122,962,944

Layers / heads / KV

8 / 16 / 4

Hidden · SwiGLU

1024 · 2816

Context

512

Vocabulary

32,000

Steps

10,764

Train tokens

352,729,403

Epochs

1

Parameter count read from artifacts/hf-tt-tnt-1024a/model.safetensors; shapes from its config.json; steps and token count from the run's own train.log. Llama-3 style throughout: RoPE (θ = 500000), RMSNorm, SwiGLU, grouped-query attention, tied embeddings, bfloat16 weights. AdamW at a constant 3e-4, weight decay 0.01, stochastic rounding on, seed 5489.

Those two share-alike rows are 45.96% of the blend, under two copyleft terms that cannot both be satisfied by one concatenated file. So the corpus is not redistributed anywhere. What is published, as episod/tt-tnt-corpus, is everything needed to rebuild it byte-identically and nothing that would redistribute it: the source registry with pinned dataset revisions, the fetch, prepare, measure and blend scripts, the generated licensing table, and the provenance manifest carrying the blend's SHA-256.

A published copy is a copy, and copies drift. scripts/publish_corpus_recipe.py defaults to --verify: it downloads what is currently on the Hub, diffs it byte for byte against what the working tree would publish right now, prints which files differ and exits non-zero. That mode exists because the manual copy had already drifted twice, once carrying a hand-transcribed figure that claimed a 0.46% gap where the real one was about 1.71%.

Whether weights trained on share-alike data are a "Data Derivative" under CDLA-Sharing-1.0 or an "Adaptation" under CC-BY-SA-3.0 is not settled, and this project does not assert that they are not. Anyone publishing weights trained with this code should reach their own conclusion rather than inheriting one.

Both patches describe themselves in-file as shims, not fixes, each with the upstream change that would make it a deletable no-op. docs/serving-with-tt-kernel.md is the how-to for the whole path: what the manifest fields do, the command that brings a server up, how to confirm you are serving the weights you think you are, and the traps between — a launch command that depends on the working directory, an environment variable that must be left unset, a newer ttnn wheel that cannot compile a kernel on this host, and two independent caches that go stale behind a name that did not change. What genuinely cannot be fixed from this side is written up in docs/upstream-tt-metal-asks.md, each ask with a reproduction and an explicit note on whether it blocks anything here. None of them currently does: the workarounds cost nothing measurable, and one is filed purely because the failure mode is expensive to debug and the next person to hit it deserves the analysis. One entry has been corrected in place after it turned out the thing it said could only be fixed upstream could in fact be fixed here — that story is in §05.

1.41× — redundant explicit attention mask

ttml's trainer always passed an explicit attention mask, and the fused SDPA kernel picks its mask mode from whether a mask object was passed rather than from what is in it. So every step paid for the arbitrary-mask path: roughly double the attention FLOPs, with load balancing disabled. tt-metal could not be edited. But ttml ships two Llama implementations, and while the C++ one binds the mask as non-optional, the pure-Python one reaches an SDPA binding that already declares its mask optional. No rebuild, no monkeypatch, no tt-metal edit.

Checked before believing it: with the mask still passed, the two implementations cost 521.7 vs 521.9 s/1000 steps at the 384 shape — within 0.4%. The entire difference is the mask, not the language. Causality was verified directly: perturbing token t leaves every earlier logit bit-identical on both paths.

3.98× — four chips instead of one

Every run before this used one chip of four on a TT-QuietBox 2. The blocker was that a DDP-only run must open a [1,4] mesh while the stock descriptor declares the physical [2,2] wiring — and that mismatch is not rejected. The mesh opens, the model trains at full speed, step 1 completes, and step 2 hangs forever in the first gradient all-reduce. The fix needed no tt-metal change: a vendored descriptor declaring the logical shape.

At the smaller 384 shape the mask fix is worth more on its own: 503.3 → 356.7 s/1000, 1.41×. The two levers are independent, and they compose.

Four-chip checkpoint saving

The training step re-marks each replicated parameter's topology as sharded while the data stays genuinely replicated, so the saver wrote all four replicas concatenated — 1,475,602,288 bytes against 737,824,624 for the same single-chip run. The gather was not the bug; it faithfully honoured wrong metadata. For a while the saver simply refused to write such a file rather than produce a plausible-looking corrupt one, and the upstream ask recorded this as something that could not be fixed here.

That expectation did not hold. The topology marking turns out to be writable from Python, so it can be corrected by any holder of the tensor: the saver now re-marks each parameter as replicated immediately before a write and restores the original marking immediately after. No data moves, and a four-chip save costs what a one-chip save costs. The file is back to 737,824,624 bytes, and after 50 data-parallel steps every tensor in it is bitwise equal — maximum absolute difference 0.0 — to replica 0 read independently through a path that does not consult the placements being corrected. The numerical parity gate has now run against a data-parallel checkpoint for the first time, at 2.56e-6.

The upstream defect is still real and still worth fixing at source: every consumer of a data-parallel parameter's topology is currently told something false and has to know to disbelieve it. What changed is only that this project is no longer waiting on it.

A seed-noise floor

Two runs, identical in every respect except the random seed. Everything they differ by is noise, by construction, and that difference is the floor under which no claim in this project is allowed to stand. Measured over 22 matched checkpoints, the seed alone moves validation loss with a standard deviation of 0.1944 nats, and it does not even hold a consistent sign: 8 of 22 checkpoints negative, 14 positive, two-sided p = 0.286.

Every reported difference is printed beside its ratio to that floor. Anything within about 1.2× of it is labelled NOT INTERPRETABLE regardless of what its confidence interval says — the interval does not get a vote, because the floor already demonstrated that intervals can be cleared by nothing at all. The threshold is not a round number chosen for comfort; it is set just above the largest false positive this project has actually observed.

There are two ways to fool yourself here, and both are gated. A delta can clear its interval while sitting inside the noise floor. Or it can sit far outside the floor while failing to clear its own minimum detectable difference — a large ratio over a tiny denominator — which is labelled below paired detection instead. Both gates have to pass before a difference gets called better or worse.

The four labels above are this page's entire colour system. They are the verdicts scripts/evaluate.py can stamp on a measured difference. The next section borrows the same four for the external benchmarks' chance gate — the same shape of rule against a different null — and says so where it does it. Nothing else on the page borrows those hues.

The reference column is measured, not quoted

GPT-2 small was run through the same harness: same lm_eval version, same task definitions, same machine, same day. Quoting its published figures instead would have compared our scores under lm-eval's detokenizers against numbers produced by different code, and it would have concealed the single most useful thing the reference run turned up. It paid for its hour of CPU three times.

It validated the entire setup with one number. Our measured GPT-2 WikiText word perplexity is 37.3695 against the paper's published 37.50 — 0.3% apart. The harness, the detokenizer, and the claim that lm-eval's wikitext task really reports WikiText-103 test perplexity are all confirmed by that agreement. That last one is not assumed either: the script loads both dataset configs at run time and confirms their test splits are byte-identical (1,288,493 characters), printing UNVERIFIED on any failure rather than a false confirmation.

It showed where quoting would have misled. The same cross-check disagrees by 29.2% on LAMBADA accuracy — 0.3256 measured against 0.4599 published — because the paper filters stopwords on generation while lm-eval scores the exact continuation by loglikelihood. A published figure and a measured one are different kinds of object. Both are printed, with the gap.

And it showed that three of the eight tasks are dead instruments at this scale, for either model. GPT-2 small is itself at chance on WinoGrande and below chance on ARC-Challenge and MMLU. Read against quoted figures, our three matching rows would have looked like our failures; they are the benchmarks' own floors. A later model that moves WinoGrande or MMLU has done something GPT-2 small could not.

What the table says

The model has learned English, not just this corpus. That is the claim no instrument in the repository could reach, because every other one is scored against the same corpus the model was trained on. ARC-Easy at +6.4 standard errors and PIQA at +4.2 — on data nobody here chose, scored by code nobody here wrote, against a null nobody here controls — are what establish it.

That is not a triumph. It is what a 123M-parameter model that has read 352.6M tokens should do, and the distance to the reference class is one number: WikiText-103 perplexity 222.66 against GPT-2 small's 37.37, at 1.2% fewer parameters (122,962,944 against 124,439,808) and 113× less training data (352,714,752 tokens seen, against WebText's ~40 billion). Roughly six times the perplexity for roughly a hundred-and-thirteenth of the data is an ordinary place to sit on the scaling curve. A reader who sees 222 and concludes "broken" is reading the parameter row instead of the token row.

Full reports, with all 15 metric rows, the per-task truncation audit, the pinned harness versions and the exact command line: docs/measurements/external-tt-tnt-1024a.md, external-tt-tnt-v3.md, and the reference run external-gpt2-small.md. lm-eval is not a dependency of this repository and did not become one: it lives in a throwaway virtualenv under scratch/ and the script shells out to it. No device was opened.

It reads model.embed_tokens.weight straight out of model.safetensors and nothing else — no forward pass, no device, CPU only. Characteristic tokens are chosen by log-odds ratio with an informative Dirichlet prior, z-scored (Monroe et al. 2008), 150 per source, winner-take-all so the nine label sets are disjoint. Raw frequency would say nothing, since every source's commonest token is the same function word.

Take a token characteristic of one of the nine sources, look at its ten nearest neighbours by cosine in the embedding space, and 54.6% of them are characteristic of the same source. Chance is 11.1%. The label-permutation floor — the identical computation with the source labels shuffled, which holds the geometry and the class sizes and destroys only the correspondence — is 0.1103 ± 0.0031. The observed value sits 138.8 of those spreads above it. A frequency-only control, built from log corpus count and embedding norm and containing no directional information at all, reaches 0.1350.

A linear probe agrees, and it is the number that speaks to whether a layout is possible at all, because a grid needs source identity to be linearly legible: multinomial logistic regression recovers which of nine sources a token belongs to, from its embedding alone, 77.98% ± 2.31% of the time — against 10.07% with the labels permuted and 20.99% from the frequency-only control. Linear on purpose; a stronger classifier would answer an easier question.

All of the above is the content condition: the same measurement with the 500 globally most frequent tokens excluded. If the geography were a frequency artefact it would collapse there. It strengthens — purity 0.4984 → 0.5458, probe 0.7373 → 0.7798.

Every source is far above chance, but the ordering splits cleanly and the split is the part that matters for a layout. procedural, flavour, poetry and wikipedia_simple occupy well-separated regions. folklore, weird and gutenberg_children share one — they are each other's largest off-diagonal neighbours, folklore and weird trading 0.14 and 0.17 of each other's neighbourhoods. Asking that region for three independent directions would return three draws from the same neighbourhood, which is exactly the failure the proposal exists to avoid.

Source: scripts/probe_embedding_geography.py, written to docs/measurements/embedding-geography-tt-tnt-1024a.{md,json}. Model artifacts/hf-tt-tnt-1024a, embedding matrix only. CPU, no device opened, no forward pass run.

Read without the control, the decay row is a finding: its interval excludes zero. With the control, decay minus held is −0.0122 ± 0.0095, |t| = 1.29 — not distinguishable from run-to-run noise. The prior was a 0.1–0.3 nat gain. The observed effect is an order of magnitude smaller and unattributable. The schedule really did run — the learning rate is traced falling through the log — so this is the useful form of a negative result, an intervention that applied and did nothing, as distinct from one that silently failed to apply.

The second control was external and blunter: run the whole configuration again, changing nothing but the random seed. The seed reproduced the result. Same four signals, same direction, comparable magnitudes, the same closing line — better on four, worse on none. The finding was withdrawn, and the threshold rule described above exists because of it.

The same table, read twice

This is the clearest demonstration of why the floor has to exist. Below are two comparisons on the same 45-prompt frozen set, scored by the same code. On the left, a 5.6× increase in parameters, with everything else — batch size, context, seed, corpus, optimizer, step count — held. On the right, a different pair of checkpoints entirely: one configuration run twice with nothing changed but the random seed.

Better on four, worse on none — twice, on the same four signals. One of those columns is an experiment. The other is a coin. Without the floor there is no way to tell them apart, and a report that showed only the left-hand column would read as a clean win.

The arithmetic is visible in the table itself. The right-hand column is the floor — it is where the floor's numbers come from. So the capacity column's register margin, −0.077, divided by the seed column's −0.075, is 1.03×: NOT INTERPRETABLE, whatever its confidence interval says. The project's own record of which model is current says so in as many words: the register advantage was substantially context, not capacity, and the clean capacity leg is a null by this project's standing rule.

What capacity did buy is real and it is on a different axis: at a matched 512-token window the larger model's validation loss is lower by 0.2994 nats on average and is lower at 22 of 22 checkpoints (sign test p ≈ 4.8e-7), where the seed-only control wanders across sign at 8 of 22. Final matched-window validation loss 2.9281 against 3.1938. Lower loss; no demonstrated change in how it writes.

Two consequences, both measured rather than assumed. Generation could never terminate naturally, so it always ran to whatever token limit the caller set. And a position-wise probe showed per-token loss flat from position ~64 onwards — with boundaries unmarked, distant context genuinely is unpredictable, so a model that ignores it is behaving correctly. The mid-generation topic collapses in the sample sheets are the same fact from the other side: the model faithfully reproducing the unmarked document transitions it was trained on.

The fixed blend terminates every document with a line holding exactly </s>. There are 798,771 of them, one per ~500 tokens, 0.200% of the corpus.

The zeros are not "rarely". They are never, by construction. On the larger 45-prompt set the sampled termination rate goes from exactly 0.000 ± 0.000 to 0.051 ± 0.012 — one of the few differences in this project's history large enough that no noise floor is needed to believe it. The two checkpoints either side of this table differ in context length as well as in separators, so most behavioural differences between them are not cleanly attributable. Termination is the exception: the earlier model had never seen the token it was supposed to stop on.

It is a partial fix. Two thirds of completions still run to the limit, and the model is much readier to stop on short-document material than on the book-length sources — which is exactly what the corpus taught it, since separator density ranges from one per ~210 tokens in the short sources to one per ~80,000 in the books.

Register, and what moves it

The corpus was blended to pull the model's voice away from children's stories and towards its nineteenth-century naturalist and folklore sources. Three deliberate interventions were aimed at that, and the project's own note records the outcome plainly: a decoding change, a stratified validation split, and a learning-rate decay tail were all tested, and none of them moved the model.

Then something did, and it was not the thing being tested. Filling in the 2×2 of capacity and context showed the register signal moving 3.51× the seed floor between the 22M model and the 123M one — of which almost none is capacity. The clean leg, where two checkpoints share batch size, context, seed, corpus, optimizer and step count and differ only in architecture, moves it 1.03× the floor: NOT INTERPRETABLE, the same magnitude as re-rolling the seed. What moved was the context leg, at 2.48× the floor and 71% of the total — and it moved in the direction nobody expected, register improving as the training window got shorter. That leg varies batch size along with the window, to hold tokens-per-step fixed, so it is not clean either. The honest summary: register is not immovable, capacity is not what moves it, and what does move it has not been isolated.

The voice the blend was aiming at has still not arrived. TinyStories remains the single most common nearest source for the current model's completions at 22.3% — ahead of the eight others, though below its 31% share of the corpus.

A number worth seeing, on the way past: the same models score materially differently on the two frozen prompt sets — nearest-source-is-TinyStories runs 0.45–0.53 on the 15-prompt set and 0.22–0.36 on the 45-prompt one. Same models, same code, different prompts. That is why the sets are never pooled and why both sides of a comparison are always scored on the same one.

On-device generation, and the defect that used to be here

This section previously recorded that the same weights served through the vLLM plugin produced worse output on device than on CPU — greedy decoding collapsing into a repetition loop, local repetition at 1.55× the CPU rate, verdict not usable. That defect was in the vLLM layer, not the model.

Running the same weights, prompts and greedy decoding through tt_transformers directly, with no vLLM, gave a median agreement of 12 tokens with the CPU reference at a local-repeat rate of 0.000, against 4 tokens and 0.222 through vLLM. That located the fault. The plugin was 12 commits behind, including a fix for a sampler returning a stale token when no forward pass was pending — which produces exactly the observed repetition. On plugin c127c17 the local-repeat rate is 0.031 against a CPU reference of 0.000, and free-running decode reads as ordinary prose.

It is back, on a different model and a newer stack (2026-08-19). Serving tt-tnt-1024-dialogue on tt-metal v0.77.0, the local-repeat rate is 0.1125 against a CPU reference of 0.0125 on the same weights and prompts, with one prompt at 0.80 — “The cat. The cat. The cat.” The same bisection localises it the same way, and more sharply than before: driving tt_transformers directly, greedy, no vLLM, gives 0.0000 — cleaner than the CPU reference itself. The prompt that loops worst under vLLM generates “stars. The cat was happy to see the star…” on the same silicon.

Two candidates are already excluded. The plugin, by A/B: serving the same model on c127c17 and bd150c7 gives byte-identical results — every per-prompt rate, every agreement count, every divergence index — which also establishes that the serving path is deterministic across restarts. And multi-block paging, by reducing block_size 64 → 512 so a 512-token context is one block with no page-table traversal: it does not help.

What that refutes precisely is traversal, not paged attention itself — one block still runs the paged code path. Separating those needs the direct arm to page, which currently throws inside paged_fill_cache; that is a harness limitation and not evidence. Left standing: the paged attention op at one block, the scheduler, and the upstream vLLM version itself, about which the plugin A/B says nothing, because it varied the plugin and not vLLM. Open.

Because the answer depends on which path is taken and not on the weights, the fault is somewhere in the decode / KV-cache path. Prefill looks sound — 20 of 21 prompts agree with CPU on the first generated token. A long series of hypotheses has been tested and refuted: the weights, the conversion, the grid patch, MLP quantization, prefix caching, async decode, on-device sampling, KV-cache dtype, and the possibility that it was merely an artifact of greedy decoding. The defect has also survived a change of tokenizer, corpus, corpus revision, context length, weights and tt-metal build. The cause is not known.

Worth noting for anyone gating a serving path: the tt_transformers PCC check passed at 0.9940–0.9998 throughout, because it exercises prefill far harder than long decode.

And the ordinary limitations of a small model

• It is a base completion model. No instruction tuning, no chat template. Give it the opening of a passage; do not ask it questions.
• One epoch over one corpus. It has seen its training data once, not memorized it.
• Under greedy decoding it finds repetition attractors that sampling largely avoids — which is the honest way to read the greedy loops, not an excuse for them.
• It has no knowledge to speak of. Its idea of physics is weather.