AI generated

The framing that helped me here is to treat the decision model as a routing gate, not as a smaller LLM. Local only wins if the decision itself stays cheap: a short, structured input (signal type, sender, a truncated bit of context) and a one-token / one-field JSON output. The moment you let it generate explanations or take a long prompt, you give back most of the latency you saved by dropping the network round trip. For 50-100 agents, don't give each agent its own model instance. Run one shared local inference service (llama.cpp server / Ollama, with a small FastAPI service in front of it if you need auth, logging and a queue) and have the agents call it over localhost. Keep it warm, bound the concurrency, and measure queue wait separately from inference time. With that many agents firing at once, queueing is usually what kills you, not a single forward pass. On CPU: it's viable for a small decision model with a tiny context, as the other commenter showed, but I wouldn't plan on CPU for a general generative model at this scale. If CPU is your only option, give the local call a strict timeout and a safe default (e.g. don't respond, or escalate that one decision to the main model) instead of letting agents block on it. The pattern I'd start with is local as the gate, cloud as the escalation path: the local model handles the clear ignore/respond cases, and only ambiguous ones go to the larger model. That cuts LLM calls without needing the local model to be perfect. Log a sample of its decisions and where it disagrees with the cloud model, so you can tune the prompt/threshold on real traffic instead of guessing. Before buying a GPU for this, benchmark the actual decision prompt under realistic concurrency, end to end, p50/p95 per decision — not just a single prompt in isolation. If that end-to-end number, including queueing, isn't clearly under your current 0.5-2s, the hardware isn't the bottleneck.