AI generated
Hi Hy4 team, congratulations on the release. I work on small and medium sparse MoE models and quantization. I have a 35B-A3B model (256 experts, top-8, about 3B active) quantized to roughly 2.1 bpw ternary, and I am trying to recover capability with 4-bit on selected expert tensors plus low-rank correction branches. Your 12B-active, 512-expert LatentMoE design is the closest reference I have. I have five questions, roughly in order of how much they would change what I do next. - Your IQ2-AH build uses IQ2_XXS on gate/up and Q4_K on down. On my 35B, the opposite works better: Q4 gate/up plus ternary down gives PPL 2.34, while Q4 down plus ternary gate/up gives 2.68, and all-ternary gives 2.93. Is down generally more sensitive in Hy4, or is that specific to LatentMoE and the 1536-d expert latent? What signal do you use to decide which expert projection gets more bits? Relatedly, is per-expert mixed precision worth the complexity in a large fine-grained MoE, or is per-tensor allocation usually enough? - Do you train the router when recovering a quantized MoE? I froze the router and saw routing drift and a tool-use regression, and preserving the original routers did not fix it. Is router co-adaptation essential after quantization, and how do you keep it stable? - For a frozen, quantized body, would you use low-rank branches at the MoE and attention outputs, full-body LoRA/QLoRA, or QAT on the quantized weights? Where do adapters belong in a sparse MoE? My rank-512 branch LoRA at the block outputs was not enough, and I would rather not repeat that on a larger run. - As active parameters drop from 12B toward 3B, what degrades first, and is there a point where a ternary or 2-bit MoE stops being viable? If you had a 3B-active model, what would you spend the next bit or the next unit of compute on: more active capacity, Engram, higher bits, or latent experts? How much does Engram contribute at 12B active, and would it be worth adding to a 3B-active model? Would shared n-gram tables with per-layer projections, as in Chorus, be a better fit than per-layer tables? Thanks, and congratulations again on the release.