AI generated

I've been looking at TinyInfer, and I think the project has an interesting direction. Building a lightweight inference engine directly in C++/CUDA, with custom kernels rather than relying on PyTorch or cuBLAS, is a worthwhile approach. If I were deciding where to take the project next, I would focus on a few improvements that could make a measurable difference, rather than expanding the feature list too quickly. 1. Separate Prefill and Decode This would be my highest priority. The current token-by-token approach in Session::feed() appears to limit prompt processing. I'd keep the GEMV-oriented path for autoregressive decoding, but introduce a proper batched prefill path using GEMM or optimized matrix multiplication kernels. Prefill and decode have very different performance characteristics. Treating them as separate execution paths could significantly improve prompt processing without sacrificing decode performance. 2. Improve KV cache efficiency The current FP16 KV cache is a reasonable starting point, but memory consumption becomes a major limitation as context length and model size increase. I'd consider: INT8 or FP8 KV cache. Dynamic KV allocation rather than reserving everything upfront. Better memory planning and reuse. This could make longer contexts practical on GPUs with limited VRAM. 3. Expand model and architecture support I'd avoid trying to support every architecture immediately. Instead, I'd introduce a cleaner separation between model architecture and the inference backend. A sensible progression would be: Llama 2/3 compatibility. Qwen 2.5 and Qwen 3. MoE architectures later, once the dense execution path is mature. The tokenizer and model conversion pipeline should evolve alongside this work. Supporting more models is only useful if conversion is reliable and the resulting inference is correct. 4. Introduce hybrid CPU/GPU execution This could become one of TinyInfer's most interesting advantages. Rather than requiring the entire model to fit in VRAM, I'd eventually introduce layer-wise CPU/GPU offloading, using CUDA streams and double buffering where appropriate. I'd implement this in stages: first a reliable memory planner, then controlled layer offloading, and only afterwards more aggressive transfer/compute overlap. For hardware with limited VRAM, this could be a genuinely useful feature. 5. Build a proper benchmark suite I would make performance claims reproducible. At minimum, measure: Prompt processing speed (tokens/sec). Generation speed (tokens/sec). Time to first token (TTFT). Peak VRAM and RAM usage. Performance at different context lengths. Compare against llama.cpp using the same model, quantization, prompt and hardware. This would show exactly where TinyInfer is faster, where it isn't, and what deserves optimization. 6. Add an OpenAI-compatible HTTP server Not necessarily immediately, but eventually. A lightweight server with streaming, chat completions and basic session management would make TinyInfer much easier to integrate into existing applications. My suggested development order: 0.5: Separate prefill/decode paths and establish reproducible benchmarks. 0.6: KV cache improvements and memory management. 0.7: More model architectures and tokenizer support. 0.8: Hybrid CPU/GPU execution. 0.9: HTTP server and streaming. 1.0: Stabilization, correctness testing and documentation. One important point: I wouldn't try to turn TinyInfer into a smaller clone of llama.cpp. Its value could be precisely that it remains a compact, understandable C++/CUDA inference engine where the execution path and memory management are under direct control. If I had to choose only three priorities right now, they would be batched prefill, efficient KV cache management, and reliable benchmarking. Those three would provide a much stronger foundation for everything that comes afterwards.