AI generated
After the harness leaderboard thread I wanted to see what the harness alone changes when the model stays the same. Model: M3.1 Flash. It's open weights, but at about 428B total I'm calling it through MiniMax's API, effort pinned to high. Same prompt in every harness: four bug reports filed against pytest this week, fix all four with a regression test each. The bugs: * pytest --fixtures dies with a TypeError when two conftests import the same fixture * a non-mark in pytest.param(marks=...) raises an internal AttributeError instead of a TypeError * approx() raises InvalidOperation when the actual value is a Decimal NaN * rm_rf leaves the whole basetemp behind when an empty dir sits under a read-only parent |Harness|Wall clock|Requests|Tokens|Cached input|Cost|New tests| |:-|:-|:-|:-|:-|:-|:-| |OpenCode 2.2.7|23m 38s|159|13.9M|98.8%|$0.27|10| |Pi 1.5.2|18m 34s|134|9.1M|98.5%|$0.19|8| |mini-swe-agent 2.4.6|21m 16s|136|10.4M|98.5%|$0.22|9| All three fixed all four. I re-checked each one in a clean checkout with its own new venv: every report's repro now behaves the way the report asks, and the full suite (about 4,700 tests) shows nothing new compared to an untouched main, apart from the added tests passing. The first request is where the harnesses differ most: 8.6k tokens for OpenCode 2, 3.7k for Pi, 3.1k for mini-swe-agent, task prompt included. It didn't decide the bill though. With 98%+ of input coming back as cache reads billed at 1.5 cents per million, cost followed how many turns each harness took and how much context it carried, which is why OpenCode's 159 requests came to about 8 cents more than Pi's 134. mini-swe-agent only gives the model a bash shell and still landed in the middle on both time and cost. One run per harness, so treat it as one data point. If you run open-weight models in more than one harness, which one ends up cheapest per task for you, and do you see the same request-count pattern?