A Language Model on a Floppy Disk, and the Problem It Exposed in ONNX

emmtrix Tech Posts
Category: Embedded AI

A 367,000-parameter transformer written in Turbo C 2.0 runs on a 486 under DOS and fits on a single floppy. We wanted it on an Infineon AURIX TC4x, which meant generating C from it and that meant exporting it to ONNX first.

That is where it stopped being simple.

ONNX graphs are pure functions. A language model is not: every generated character depends on the keys and values computed for every character before it, the KV cache. A graph has nowhere to keep that, so the cache has to leave through the outputs and come back in through the inputs.

For this model the numbers are blunt. The step graph has ten inputs and nine outputs. Eight of each are the cache: 384 KB, passed in and passed out again for every single character. The entire weight file is 380 KB. The cache is the size of the network.

Generated literally, that is 768 KB of copying per character to change one position out of 128.

The ONNX specification permits the obvious fix and does not express it: “the input/output aliasing is entirely up to backend implementations.” Every toolchain therefore declares outside the model which output feeds which input – ONNX Runtime GenAI by name pattern, OpenVINO by explicit pairs – and an open ONNX issue acknowledges the gap.

In emx-onnx-cgen we made that declaration explicit: the input is short-circuited to the output, the state lives in one buffer the caller places, and the write drops from 768 KB to 3 KB per character. Pairs are never guessed from names, and there is no silent fallback to a copy.

The full write-up, with the measurements and the comparison to other toolchains, is in our wiki.

Side-by-side diagram comparing ONNX KV cache handling: copying a 384 KB cache in and out moves 768 KB per generated character, while sharing one buffer requires writing only 3 KB at the updated position.

For this model, sharing the KV cache buffer replaces 768 KB of copying with 3 KB of writes per generated character.

Cookie Consent with Real Cookie Banner