AI generated

i genuinely don't get the hate boner for aa on this sub. like seriously, what's the issue? they're running benchmarks, publishing the results, and unless someone can actually demonstrate that they're running those benchmarks improperly, i don't see what the controversy is supposed to be. and the intelligence score you're seeing is just a weighted score based on the benchmarks themselves. it's not some arbitrary number they're pulling out of their asses. obviously, the choice of benchmarks and how they're weighted will influence the final score. that's literally how composite scores work. surely we're not getting mad at aa because they've decided to put more weight on terminal-bench 4 and frontiercode 2 instead of t2.1 and livecodebench? when pretty much every 300b+ model is scoring 80-95 on a particular benchmarks, maybe that benchmark just isn't particularly useful for distinguishing between frontier models anymore. including it in the overall score anyway would just dilute the differences that other benchmarks are actually showing. and people on this sub literally criticized prismml for doing the exact opposite of what aa is doing for their bonsai lineup. they kept including benchmarks where the differences between models were getting increasingly diluted, and then pointed to the relatively small drop in the overall average as evidence that quantization barely mattered. meanwhile, "important" benchmarks showed substantially larger drops. you don't have to agree with aa's methodology. you can absolutely argue that certain benchmarks are inappropriate, that their weighting is wrong, or that their evaluation setup doesn't reflect real-world performance. those are all perfectly reasonable criticisms. but actually make that argument. because simply getting upset that a different selection of benchmarks produces a different ranking isn't a criticism of benchmark integrity. and expecting every benchmark aggregator to use whichever metrics happen to produce the rankings you personally prefer defeats the entire purpose of having independent evaluations in the first place.