State in a Stateless Graph: LLMs and Transformer Models in ONNX
ONNX graphs are pure functions. A language model is not: generating text one token at a time only stays affordable because each step remembers the keys and values computed for every token before it — the KV cache. Those two facts do not fit together, and every toolchain that runs a transformer from ONNX has had to decide what to do about it. There is no standard answer, and ONNX itself acknowledges the gap. [1] [2]
This article follows the problem through one concrete case: a 367,000-parameter transformer written in Turbo C for DOS, aimed at an AURIX TC4x microcontroller. The model is small enough that nothing is hidden by scale, and small enough that the cost of getting the state wrong is visible in bytes rather than gigabytes. It does not run on that board yet — the state question is what had to be answered before it could, and the last section describes what we built to answer it.
A transformer on a floppy disk
In November 2026 Hackaday covered TINY 2.1, a question-answering transformer written in Turbo C 2.0 that runs on a 486DX or Pentium under plain DOS and fits, with its interface, on a single period-appropriate floppy. [3] It is a real decoder, not a lookup table: 367,000 parameters, 4 layers, 96 wide, reading and writing one letter at a time, stored as 8-bit weights with one scale per row, 380 KB in all. It needs about 540 KB of free conventional memory and was trained on roughly 3,500 facts, each asked in many wordings. Outside those facts it makes things up confidently. [4]
A model that runs in 540 KB on a 486 is interesting for a quite different reason: it is small enough to make an embedded target a sensible question rather than an absurd one. If TINY 2.1 fits on a 486, why not on an AURIX TC4x? That is what prompted the exercise this article is about.
Getting there means generating the C rather than hand-writing it, and that decides the first step for us. The emmtrix Edge AI Compiler workflow starts from a trained model in ONNX or PyTorch and turns it into portable C, so feeding this model to our own toolchain means having it as ONNX first.
The original project has none. It is a C program and a training kit, which is all a 486 ever needed — the transformer is hand-written C, not an exported graph. So the ONNX export had to be written before anything else could happen, and writing it is where the trouble starts.
Two ways to export the same network
The fork used here adds an exporter that reads the 8-bit weight file directly and builds the graph node by node, and six ONNX models checked in beside it. [5] Two of them matter, and the difference between them is the whole subject of this article.
| Model | Graph | Interface |
|---|---|---|
tiny2.onnx
|
the whole sequence at once, [batch, seq] dynamic
|
one input, one output |
tiny2_step.onnx
|
one letter per call, KV cache in and out | ten inputs, nine outputs |
The first is the obvious export and the easier one to look at in a viewer. It takes letter ids and returns logits for every position. To generate text with it you call it once per letter with the whole prefix, and it recomputes the attention over that entire prefix every time. For a 128-character answer that is 128 forward passes over an ever-growing input: correct, and quadratic.
The second is what you actually want on a microcontroller. It computes exactly one letter, given the state left over from the letter before. It also has fully static shapes, which is what a C code generator needs — no dynamic allocation, every buffer sized at compile time. [5]
The price is in the interface. Of its ten inputs, two are the work — tok[1] and pos[1] — and eight are the cache: past_k_0..3 and past_v_0..3, each of shape [4,128,24], which is 4 heads × 128 context positions × 24 values per head, for each of the 4 layers. Eight of the nine outputs are the same tensors, updated. The logits are one output among nine; the rest is bookkeeping.
That is not a quirk of this export. It is the only shape the problem can take in ONNX.
Why the state has nowhere else to go
ONNX is a dataflow graph of pure functions. The specification has no variables, no assignment, and no mechanism for a tensor to outlive a call. The ONNX documentation says so while describing the very pattern used for caches: "As a reminder, the ONNX representation is still a functional representation, with ops that are pure functions." [1]
A program has somewhere to keep a cache — TINY.C simply holds it in memory between calls, which is why the Hackaday article never mentions a KV cache at all. There is nothing to mention; a C program keeping an array across function calls is not a design decision worth remarking on. A graph has no such place. The only way to express "remember this" is to make it an output and take it back as an input on the next call, and that is what every ONNX export of a decoder does.
So the state does not disappear, it moves into the signature. And the cost moves with it: a caller must now hand in 8 × 4 × 128 × 24 floats and take eight of the same back, every letter. That is 98,304 float32 values, 384 KB — passed in and passed out again, 768 KB of traffic per character, for a model whose entire weight file is 380 KB. The cache is the same size as the network.
On a desktop that is wasted bandwidth and nothing worse. On the intended target it decides the design before a single line is deployed, and it is worth being specific about how badly 384 KB does not fit, because "too big for fast memory" is the kind of phrase that invites a shrug.
The TC4x PPU has two levels of local memory: VMEM, 128 KB of L1 vector memory beside the vector DSP, and CSM, 512 KB of L2 cluster shared memory, which the manual describes as usable for instructions as well as data. [6] The cache does not fit in VMEM at all — it is three times the size of that entire level. It fits inside CSM arithmetically, 384 KB of 512, but CSM is the memory that also has to hold code, and committing three quarters of it to a single tensor is not a trade anyone makes willingly. What remains is the LMU, ten banks of 512 KB of on-chip SRAM reached across the interconnect rather than sitting inside the cluster. [6] That is the "large slow memory" the implementation note names — still on-chip SRAM, not external DRAM. [7]
Whether moving 768 KB per character through it is fatal or merely wasteful is a different question, and this article does not answer it. The PPU has a DMA unit, the STU, whose documented job is transfers between VMEM, CSM and memory outside the PPU, [6] so the copying can in principle overlap with computation instead of stalling the core. Settling that needs a measurement on the board, and none has been made.
How the ecosystem answers this
The obvious optimisation is to let the output tensor and the input tensor be the same memory, so the cache is updated in place and never copied. Everyone does this. Nobody does it the same way, and ONNX does not specify it.
What ONNX provides
Since opset 24 ONNX has operators that make an in-place cache update expressible: "ONNX opset-24 has introduced new features to facilitate the representation of in-place KV cache updates." [1] The pattern pairs a fixed-size cache with TensorScatter, which writes the new keys and values into the buffer at a given index, and Attention, whose optional nonpad_kv_seqlen input says how much of the fixed buffer is valid. [8] [9]
And expressible is not the same as used. The export in this case writes the cache with plain arithmetic — kept = past × keep, added = fresh × here, new = kept + added, a masked add that leaves every position but one untouched. It is exactly what TensorScatter computes with write_indices = [pos], and the model is opset 25, so the operator was available and simply was not reached for. The graph contains Mul and Add and no scatter of any kind. [5]
That is worth noticing, because it makes the problem worse rather than better. With TensorScatter there is at least an operator in the graph whose specification talks about caches and in-place updates, so a reader or a tool has a hint. With a masked add there is nothing at all: eight tensors go in, eight come out, and the arithmetic in between is indistinguishable from any other multiply-add. Whatever marks these tensors as a cache has to come from outside the file either way — but in this form, not even a hint survives.
Expressible is also not the same as specified. The same page is explicit about who decides whether anything is actually done in place:
the input/output aliasing is entirely up to backend implementations.[1]
And it states the condition a backend must satisfy: "the backend will need to ensure that the input to TensorScatter is not subsequently reused by other ops." Only then "is it safe to reuse the memory allocated to the past_k/v input of the op for the present_k/v output." [1] The document calls the arrangement "a useful common pattern", not a requirement. The graph still says nothing about which output feeds which input.
What the gap looks like from inside ONNX
This is a known gap, not an oversight that nobody noticed. An open umbrella issue on the ONNX repository, opened in August 2026, puts it plainly: ONNX "can expose state as graph inputs/outputs but lacks a structured model-level declaration" for how a runtime should carry those tensors across calls. [2]
It sketches what such a declaration might contain — an illustrative StateBindingProto with fields for input_value_name, output_value_name, a role, an update policy, initialisation, reorder and rollback policies — and is careful to add that this "does not add hidden mutable graph state": "the ONNX graph stays a pure function with explicit state inputs and outputs." [2]
It remains an idea. The issue is a tracking issue with no linked pull request and no milestone, and the name is described as illustrative rather than proposed. The gap is acknowledged; it is not closed.
What the runtimes do instead
Because the model file cannot say which output feeds which input, every toolchain supplies that information out of band, and the mechanisms differ in a way that matters.
ONNX Runtime GenAI declares it in a configuration file, by name pattern — the documentation's own term. model.decoder.inputs.past_key_names is a "Name pattern for past key tensors", model.decoder.outputs.present_key_names the matching one for the present tensors, and the example config uses past_key_values.%d.key against present.%d.key with 32 layers. The documentation adds that "the names and patterns here must match the actual ONNX model graph." Sharing itself is a search option: with past_present_share_buffer set, "the past and present KV cache buffers point to the same memory block"; without it, the present buffers "are re-allocated before every forward pass of the model and copied to the past KV cache buffers." [10] [11]
OpenVINO has actual state in its IR. A MakeStateful pass "changes the structure of the model by replacing the user-defined pairs of Parameter and Results with the Assign and ReadValue operations", where ReadValue "reads information from an internal memory buffer" and Assign "writes data to this buffer". The pairs are given explicitly, as tensor names or as operation objects. A second pass, LowLatency2, instead finds the pairs automatically — but only for models containing TensorIterator and Loop. [12]
Hugging Face Optimum settles it by convention. Its exporter builds the inputs as past_key_values.{i}.key and .value and the outputs as present.{i}.key and .value, one pair per layer. That the cache grows rather than occupying a fixed buffer is visible in the declared axes: the sequence axis of the input is past_sequence_length, and of the output past_sequence_length + sequence_length. [13]
The pattern across all of them: the pairing is real and necessary, and none of it is in the ONNX file. OpenVINO is the instructive exception — its MakeStateful pass takes the declaration and rewrites the model so that the state becomes part of the IR, which is precisely the step ONNX has no equivalent for. Everywhere else the knowledge lives beside the model, in a config file, a command-line flag or a naming convention, and a model handed to a different tool arrives without it.
What we built
The ONNX-to-C step of that workflow is emx-onnx-cgen, and this is where the export arrives. The point of producing C at all is that it can be read, compiled with an existing toolchain and optimised for the target, instead of the model being deployed through a large inference runtime — which is precisely why the cache cannot be a runtime's private business. It has to become a buffer somebody can see and place.
The step graph handed to it has eight cache tensors on each side, and generating the obvious code — copy in, compute, copy out — would have put 768 KB per letter through memory to move a cache that never needed to leave it.
The feature that addresses it short-circuits a graph input with a graph output, so the state lives in one buffer that persists across calls. [7] Three decisions in it are worth stating, because each is a refusal.
The pairs are never inferred from names. They are given as --state-pair INPUT=OUTPUT on the command line, or as the model metadata key emx_onnx_cgen.state_pairs. This is the opposite of the ORT GenAI approach above, deliberately: a name pattern that matches the wrong tensor produces a model that computes confidently wrong answers, and nothing in the output looks unusual. The cost of being explicit is one flag per pair; the cost of guessing is paid later and silently.
That the cache update is a masked add rather than a TensorScatter makes no difference to this, which is the point: the check is about the dataflow, not about recognising a blessed operator. Both forms are among the cases it is tested against. [7]
There is no fallback to a copy. Before anything is generated, the lowered model is checked: the two sides must have the same static shape and dtype, and no node may read the old state after the node that writes the new one. The writing node itself may read it only if its kernel is safe in place. A violation raises an error naming the pair, the node, the shapes and a hint — it does not quietly emit a copy instead. That condition is the one the ONNX documentation puts on backends, enforced rather than assumed. [1] [7]
The buffer belongs to the caller. The generated code declares a model_state struct and takes a pointer to it; it does not define a global. On the AURIX the cache has to go into large slow memory, [7] and only the caller can place it there — with a section attribute, a pragma, or a linker script. A generated global would have made that placement the generator's decision, and the generator does not know the memory map. [7]
For TensorScatter the saving is visible in the emitted C: when the cache input and the output are the same buffer, the kernel drops the past_cache parameter and the loop that copied the whole cache, and writes only the rows that changed. [7]
The arithmetic of that is worth doing. A step writes one position of 128, which is 4 heads × 24 values = 384 bytes per tensor and 3 KB across all eight — against 768 KB copied in and out without the aliasing. The ratio is 256, which is no coincidence: it is twice the context length, once for the copy in and once for the copy out. The longer the context a model supports, the worse the copying gets, while the write stays one row.
Verifying that it is still the same model
Aliasing a buffer is the kind of change that produces plausible wrong answers, so the test does not compare the generated code against itself. A verify mode runs the original model N times with explicit inputs and outputs, feeding each step's state outputs back in as the next step's inputs, and compares every output of every step against the aliased version running through the shared buffer. [7] The question it answers is the right one: does short-circuiting the buffer change the arithmetic?
On tiny2_step.onnx the answer needs stating carefully, because the comparison does not land inside the default tolerance: the logits come out around 9,000 ULP apart at values near 1e-3. Running the same check without state pairs gives 12,000 ULP. The deviation is therefore float32 accumulation order, which was there before and is unrelated — but only running both tells you that. A check that had simply reported "fails" would have been read as the aliasing breaking the model. [7]
What this does not solve
The pairs still have to be declared by hand, once per model, exactly because ONNX offers nowhere to put them. The model file remains silent about its own state, so the declaration lives beside the model rather than in it — which is the situation the ONNX umbrella issue describes, and it will stay that way until something like it is adopted. [2]
It is also not a deployment story yet. TINY 2.1 does not run on an AURIX at the time of writing. What has been solved is the part that blocked everything else: the model can now be expressed as C whose state lives in one buffer the caller places, instead of C that copies a 384 KB cache through slow memory twice per character. Everything that comes after — fitting the rest of the model, the PPU's own constraints, measuring it on real silicon — is still ahead, and this article deliberately claims none of it.
Nor is it a general LLM story. TINY 2.1 is 367K parameters with a 128-token context and a fixed cache; the shapes are static and small enough that the whole question reduces to one buffer and one write per step. A model with a growing cache, paged attention, beam search or a quantized cache raises problems this does not address, and the toolchains above make different trade-offs precisely because they target those.
What the small case does show is where the difficulty actually lies. It is not in the attention mathematics, which is a few matrix multiplications. It is in the fact that a dataflow graph has no concept of memory that outlives a call, and that every system built on such a graph has to reintroduce one from the outside.
See also
- emmtrix Edge AI Compiler — the workflow from a trained model to portable C, of which the ONNX-to-C step is one part
- emmtrix ONNX-to-C Code Generator — the generator itself, its design goals and its non-goals
- Numerical Precision in ONNX and AI Inference — what ONNX does and does not pin down about arithmetic, the same question as this article asks about state
- Infineon AURIX TC4x — the target the exercise was aimed at
References
- ↑ 1.0 1.1 1.2 1.3 1.4 1.5 In-place KV Cache for Attention — ONNX documentation — onnx.ai — https://onnx.ai/onnx/technical/kv_cache.html
- ↑ 2.0 2.1 2.2 2.3 [Umbrella] ONNX gaps for storage-faithful and efficient GGUF/LLM interchange · Issue #8370 · onnx/onnx — github.com — https://github.com/onnx/onnx/issues/8370
- ↑ Not-So-Large Language Model Fits On A Floppy Disk — hackaday.com — https://hackaday.com/2026/11/18/not-so-large-language-model-fits-on-a-floppy-disk/
- ↑ TINY 2.1 — a transformer for DOS, the original project by WollyMonjac — github.com — https://github.com/WollyMonjac/Tiny-2.1
- ↑ 5.0 5.1 5.2 The ONNX models and their accuracy,
training/MODELS.mdon theworkbranch of the fork, which is where the ONNX export lives; the original project contains no ONNX — github.com — https://github.com/strimo378/Tiny-2.1/blob/work/training/MODELS.md - ↑ 6.0 6.1 6.2 Infineon AURIX TC4Dx User Manual v1.00, chapter 14 (Parallel Processing Unit); the memory sizes are Table 740, "Product specific parameters", section 14.14.1, and the LMU banks are in the address map.
- ↑ 7.0 7.1 7.2 7.3 7.4 7.5 7.6 7.7 7.8 codegen: state pairs share one persistent buffer (KV cache) · Pull request #804 · emmtrix/emx-onnx-cgen — github.com — https://github.com/emmtrix/emx-onnx-cgen/pull/804
- ↑ TensorScatter — ONNX operator specification — onnx.ai — https://onnx.ai/onnx/operators/onnx__TensorScatter.html
- ↑ Attention — ONNX operator specification — onnx.ai — https://onnx.ai/onnx/operators/onnx__Attention.html
- ↑ Configuration reference — ONNX Runtime GenAI — onnxruntime.ai — https://onnxruntime.ai/docs/genai/reference/config.html
- ↑ Obtaining a stateful OpenVINO model — docs.openvino.ai — https://docs.openvino.ai/2024/openvino-workflow/running-inference/stateful-models/obtaining-stateful-openvino-model.html
- ↑ optimum-onnx,
optimum/exporters/onnx/base.py— github.com — https://github.com/huggingface/optimum-onnx/blob/main/optimum/exporters/onnx/base.py