Your AI coding agent could cost 5x more depending on the harness, benchmark finds

The same AI model can cost up to five times more to run depending on the coding agent harness it’s paired with, according to a new benchmark that tested 22 configurations across five open-weight models. Runta and Baseten ran those combinations 27,280 times for FrontierHarness Eval v1.1 , consuming roughly 83 billion tokens in the process. Nearly 65% of estimated inference spending went to runs that failed verification or timed out. The model itself accounted for 92.8% of the cost differences, but the harness still made a considerable difference. One configuration could be the cheapest option...

Source: The New Stack

← Back to News