AI generated

One thing a diff of saved outputs won't show: whether the model reads what it was told to read before it acts. I measured this today. Same task card, same base commit, two models as workers (Opus 5.5 and Sonnet 5.5, 3 pairs). The first line of the card says "read the guide before you write". - Cost went $5.84 → $1.02 and time 18.4 → 9.2 min on the smaller model. - The bigger model read 45k characters of the guide. The smaller one read 1k, wrote anyway, and broke a rule from the part it skipped. - A blind judge would have merged 2 of 3 from the bigger model and 1 of 3 from the smaller. 3 pairs is too few to call that a difference. The read-before-write gap was the solid part. So for test cases: log the tool calls, not only the final text. "Did it open the file it was told to open, and how much of it" caught more for me than comparing answers. (edited by claude*)