Classifier receipt

LokalBot is an open-source Mac app I've been working on for the past 6 months, it runs several useful local models. These models help the app record your meetings, transcribe them, write the notes, search everything, and autocomplete what you type. It's GPLv3, and everything runs on your device by default. The five models (Mac defaults, 9.6 GB on disk) | Job | Model | Runs on | File | |---|---|---|---:| | Speech to text | Qwen3-ASR 1.7B | MLX, 8-bit | 2.47 GB | | Who spoke when | Nemotron 3 | Core ML | 0.20 GB | | Notes, chat, agents | Qwen3.5 4B | llama.cpp, Q4_K_M | 2.74 GB | | Autocomplete | Gemma 4 E2B base | llama.cpp, Q6_K | 3.85 GB | | Search by meaning | EmbeddingGemma 2 | llama.cpp, Q8_0 | 0.31 GB | How they take turns - Recording a call loads nothing. The app only writes two audio tracks. Transcription waits until you hang up, and autocomplete pauses while you record. - After the call, the speaker model and the speech model load, do their job, and unload once they've been idle for 120 seconds. One surprise here: MLX keeps freed GPU buffers in a pool, so dropping the model wasn't enough. After one meeting the app was still holding 4.3 GB while idle. Every MLX engine now clears that pool when it unloads. - Qwen3.5 4B runs in the bundled llama-server (Metal, 32K context), which starts when something asks for it. Each request takes a lease, and ten minutes after the last lease the model unloads. The embedding model has its own server and the same rule. - Gemma 4 E2B runs inside the app's process through libllama, for latency. It has no idle timer. It stays until macOS reports memory pressure. - A broker keeps the loaded weights under a budget of half the machine's RAM and evicts the least recently used model first. A lease pins a model while a request is running on it. - Requests that have to wait for a server are served in order: what you're watching first, then agents, then meeting notes, then background work like digests and indexing. Numbers, all on one M4 Max with 48 GB - Notes for a 26-minute meeting: 33 s with the model warm, 60 s including the load, at about 85 tok/s. - Autocomplete: 56 ms median and 102 ms p95 per keystroke. - Embedding a query: 9 ms. - Speaker model: 14.6% DER on 8 AMI recordings, at about 500× real time. Limits - I've only measured on that one machine. I have no numbers from a 16 GB Mac, or from an M1, M2 or M3. There is a Lightweight preset for less memory (Qwen3-ASR 0.6B, Qwen3.5 0.8B, and LFM2.5 1.2B for autocomplete). - Eviction is plain LRU plus pins. Priorities order the queue; they don't choose who gets evicted yet. - If a model can't fit under the budget, everything unpinned is evicted and the load goes ahead anyway. - The video simplifies one thing. It shows the notes model arriving after the speech models have left; in the app, the notes start while the speech models are still counting down their two minutes. - The narration is my own voice, cloned and generated with ElevenLabs, so it's the one part of this post that didn't run locally. - Windows and Linux builds are in alpha. Code, benchmarks and the scripts behind these numbers: https://github.com/stevyhacker/lokalbot If you run it on a 16 GB Mac, I'd love to see what Settings → Diagnostics shows is loaded right after a meeting. And if you've built a scheduler for several local models, I'd like to know what you evict first.

Model stylome-1.2-large · recorded 2026-11-10T12:03:51.687369