AI generated
The Jetson AGX Xavier came out in September 2018, four years before ChatGPT existed. It has 16–64 GB of LPDDR4x at 136 GB/s. The GPU is a 512-core Volta, and people skip the board for local AI because of it. Mine is the 32 GB module. It sat on a shelf collecting dust. It is "quite terrible". I wanted a real number for how a modern LLM runs on it. Getting there cost about $300 in tokens on GPT-6-Astra and Fable 5.2, so you don't have to spend yours. I started with current llama.cpp and NVIDIA's Nemotron 3.5 Bolt, the Gemma 4 E4B model they advertise as smarter than Qwen3.8 27B (cough). llama.cpp did about 350 tok/s prefill and 23 tok/s decode. The speed is fine. A 4 GB model against 32 GB of memory, again, is "quite terrible". My own engine, Little-Gemma, is well ahead of llama.cpp, but I built it for Ampere and later. I was not going to backport it to Volta, so I tried something else. ||Prefill (peak)|Decode (peak)| |:-|:-|:-| |Nemotron 3.5 Bolt E4B|880 tok/s |50 tok/s| |Gemma4 12B|215 tok/s|35 tok/s| |Qwen3.8 27B|106 tok/s|17 tok/s| I forked MLC-LLM. Upstream does not run GGUF, and it does not know Nemotron, so the weights had to be converted from safetensors into MLC's TVM format and the Nemotron backend written from scratch. Astra did that first port and added in-memory switching between Nemotron, EmbeddingGemma 2, and LiquidAI's d1, so an agent can change models without reloading. One setup was: d1 decides whether to query EmbeddingGemma 2 over about 20,000 pages of notes, then Nemotron answers. The plumbing worked. The search sometimes took a minute, which is too slow. Then Fable 5.2 spent hours on the engine and rewrote several kernels. That work is still going. As of this week, Nemotron prefill peaks at 880 tok/s on a 512-token prompt, and decode on code reaches 50 tok/s. Nemotron is still too dim for some jobs, so I moved to Gemma 4 12B. On the same 512-token prompt, prefill peaks at 215 tok/s, and decode on code is 35 tok/s. Three times the weights, and code decode only falls from 50 to 35. The 136 GB/s bus is why a 2018 module can still carry a 12B model. Prefill is where Volta shows its age. None of that counts unless it can run Qwen3.8 27B. Generic MLC conversions are already on Hugging Face, so I am not publishing another one. The checkpoint matters: DavidAU/Qwen3.8-27B-TURBO-Fable-Cold-Fusion is the one I kept. I am not uploading that conversion either, with Qwen4 close. Draft with Qwen3.8 2B, or convert DavidAU's model-mtp-restored.safetensors yourself. On that 27B model, prefill now peaks at 106 tok/s for a 512-token prompt and holds 104 tok/s at 1,024. A Qwen3.8 2B draft reached 13.4 tok/s decode on a planning prompt. Cold-fusion agrees with that draft less often, so decode there sits nearer 10 tok/s. The attention kernel landed this week. The checkpoint's own MTP head, in place of the 2B draft, is the next thing to try. These rates will move again. At MAXN, a long prefill pulls about 35 W. NVIDIA did not design the dev kit's brick for this. Under sustained load it browns out and the board shuts off, which is a known thread on the NVIDIA forums. I dropped the stock brick for a USB-C PD supply and a short OWC USB4 cable. It has now stayed up for more than a week. There is a further trick I have not tried, and probably will not. The dev kit's slot is the module's PCIe 4.0 x8. Two kits can be cabled, one as root and one as endpoint, for about 16 GB/s each way, around 32 GB/s with both directions full. That is enough to speed up a single model if each layer is split across the two modules: the cable moves the residual stream, and the weights stay put. Some third-party carriers only bring out Gen2. Skip those if this is the plan. As of November 2026, a 16 GB module is about $130 on eBay, and it runs Gemma 4 12B. A 32 GB module turns up around $300 if you wait. Some listings are the bare module. Get one with a carrier, or budget about $60 for the carrier on its own. The engine: https://github.com/cortexist/mlc-llm The weights: https://huggingface.co/cortexist/models