AI generated

I had a look at TinyInfer and I think you're onto something interesting here. What I particularly like is the idea of keeping the engine relatively small and working directly with C++ and CUDA instead of building another layer on top of existing frameworks. If I were working on it, there's one thing I'd probably tackle before anything else: prompt processing. From what I can see, Session::feed() processes tokens one at a time. That's fine for generation, where you're predicting the next token, but it's not ideal for processing an entire prompt. I'd experiment with a separate prefill path using batched matrix operations, while keeping the current GEMV approach for token generation. I suspect this could make a much bigger practical difference than adding lots of new features. The other thing I'd look at is the KV cache. FP16 is a good starting point, but it can become expensive very quickly, especially with longer contexts. INT8 or FP8 KV cache support would be worth investigating. Even dynamic allocation and better memory reuse would help. There's also an opportunity with models that don't fit entirely into VRAM. I'd eventually like to see some kind of CPU/GPU layer offloading, perhaps with CUDA streams and double buffering. Not an easy feature to implement properly, but potentially very useful on consumer GPUs. One thing I'd be careful about, though, is expanding model support too quickly. I'd rather see solid Llama and Qwen support, with a reliable conversion pipeline, before moving into more complicated MoE architectures. And please, don't underestimate benchmarking. I'd really like to see TinyInfer compared against llama.cpp on exactly the same hardware, model, quantization and prompt. Prompt processing speed, generation speed, time to first token and memory usage. Otherwise it's difficult to know whether an optimization is actually working. Personally, I'd concentrate on three things first: faster prefill, a more efficient KV cache, and proper benchmarks. Once those are in place, I'd start thinking about CPU/GPU offloading and an OpenAI-compatible server. I wouldn't try to make TinyInfer another llama.cpp. I think there's more value in keeping it small, understandable and focused on doing a few things really well.