AI generated
Confident
Academic benchmarks are almost useless for this, they test trivia and reasoning puzzles, not whether a model can summarise your inbox without hallucinating. Your 95 percent idea is on the right track, I would build it around everyday tasks instead, write an email, summarise a long doc, answer from a PDF, plan a trip, and score pass or fail, not a percentage. The threshold matters less than the task list being honest about what normal people actually do. Tie it to cost too, a model that passes at 2 cents beats one that passes at 20 cents, and that is the real decision for most people. If you publish the task set openly, others can run the same models on their own hardware and compare, that is what would make it stick.