AI generated

Over the last months I trained 35 domain-expert LoRA adapters on Qwen3.6-35B-A3B (native MoE, 256 experts, 3B active) with MLX on a Mac Studio M3 Ultra 512 GB, BF16, no quantization during training. Everything is public: adapters, datasets, per-domain configs, router, training scripts. Things I learned the hard way: - Fusing a MoE LoRA into 8-bit produced garbage; only runtime BF16 adapter hot-swap was usable. Serving = one base loaded once + adapters swapped per request (~2 s). - A small MiniLM router over 34 domains tops out around top-1 46 %, top-3 70 % on my eval. Routing is the weak link, not the adapters. - On an independent audit bench, the base 35B model beat most of my specialist adapters. Only one domain (HTML/CSS) clearly gained. I now route almost everything to the base. Publishing a negative result felt more useful than hiding it. - `mx.random.normal` is not bit-exact across Apple GPU generations (M1 vs M3+). Reported upstream, docs PR merged in ml-explore/mlx last week. Repos: github.com/hypneum-lab/micro-kiki (adapters + router), github.com/ailiance/ailiance-mac-tuner (training stack, archived). HF: huggingface.co/hypneum-lab Question for the sub: has anyone gotten domain LoRAs to beat a strong MoE base at the 30B scale without full-parameter training? I'd like to compare routers.