AI generated
I made this post to clear up other posts saying that this model was not doing a good job for them. Specially the hallucination rate, because mine didn't hallucinate with 5 to 6 compactions at all. And it refused every single time it didn't know instead of making something up. So, hallucination is completely 0. My setup * RTX 3060 12GB + 16GB system RAM. It barely fit. * Inkling 2 Mini, Unsloth UD Q4_K_M * Unsloth's own llama.cpp build, since mainline doesn't support this model yet * 80k context * About 14 tokens/sec generation the whole run, prompt processing around 150-250 t/s * Claude Code as the harness. Auto-compaction kicked in during the run and worked fine. Results first (it finished 114 of the 162 tasks, more on that below) |What|Score| |:-|:-| |Overall, average of all tasks|94%| |Hallucination traps and normal control questions|100%| |Tool calls|100%| |Long documents|100%| |Reasoning and math|100%| |Coding (hidden tests)|100%| |Instruction following|100%| |Prompt injection and safety|100%| |Business data questions|87%| |Memory|20%| Some numbers in plain words: * It was asked 12 unanswerable questions and invented an answer zero times. * It made 29 tool calls, 3 of them failed, and it recovered every time. It also never touched the delete command. * Of the roughly 100 questions that did have an answer, it wrongly said "I can't answer that" once. * The same four easy tasks at the start of the run and at the end both came out at 100%. No sign of it getting worse over 9.5 hours. * When it said it was about 97% sure, it was right about 93% of the time. That's pretty honest. * It followed the format rules on every single answer. Where it fell down * The business data questions were the weak spot, and it was real mistakes. On the multi-step ones it forgot to multiply quantities through nested parts, and once it missed a week with a production spike. Every time it was confident and wrong, which is the worst combination, but it only happened on the long multi-step stuff. * Memory came out at 20%, and I wouldn't read much into it. It said it couldn't remember the briefing all 4 times instead of making something up. I'm guessing that's down to the context getting compacted along the way, so that one tells you more about Claude Code than about the model. Now for how the test works for anybody interested : There are 162 tasks in total. Every one is generated by code (no public benchmarks, nothing the model could have seen in training), and every answer is checked by code too, no AI judge. The model never gets to see the answer key, and it has to write each answer to a file in a set format. It also has to follow a few standing rules the whole time (valid JSON, say how confident you are, end every explanation with the task ID), so I could see if it gets sloppy hours in. What the tasks are, with the kind of thing I mean: * Business data questions (29): a fake manufacturing company with ~17 spreadsheets. "Which 5 suppliers have the most past due money?", "if this part is unavailable for 8 weeks, how much revenue is at risk?", "build a week by week inventory balance for this component". Some need you to go through a multi-level parts list. * Hallucination traps (about 20): questions that can't be answered. A parameter that doesn't exist in the docs, a made up research paper, a function that isn't in the file. Each trap has a normal question next to it, so a model can't win by refusing everything. * Tool calls (10): small command line tools that misbehave on purpose. Random failures, out-of-date docs, login tokens that expire, results that quietly say "partial data", an error message that still exits with code 0, a "delete everything" command sitting right next to the one it's meant to use. * Long documents (19): find one code in 64k tokens of near-identical text, track who is holding an object through a long story, count events in a big log. * Memory (7): a briefing at the start, then later it gets asked about it without being allowed to look again. * Coding (12): write functions, or fix buggy ones, checked against hidden tests. * Reasoning, formatting, data extraction, prompt injection, persistence: logic puzzles, exact output formats, pulling fields out of messy invoices, text hidden in a document that tries to hijack it, 25 small questions in one task. * Drift check (20): the same four easy tasks repeated 5 times through the run, to see if it gets worse the longer it works. Each task gets a score from 0 to 100% (partial credit, so getting 4 of 5 rows right is 80%). The percentages below are just the average of those. Honestly though, it's the best thing I can run on this PC right now. It's not close to anything else I've tried, and it did all of this on a 3060. If you want to test yourself to verify my numbers message me and I will send you the repo link. It's a click and forget test so you don't even need to monitor.