In September, I ran two models through the same 20 support tasks. Both scored 20 out of 20. Then I assigned each model two refund requests that the underlying billing data didn’t support: one against an uncollected invoice, and one for double the invoice’s value. Claude Haiku 4.5 issued all six refunds across three distinct permission configurations. Claude Sonnet 5 issued none. Their success rates looked identical, but their behavior diverged completely on the only scenarios capable of causing financial damage. In October, I reran the evaluation harness . This time, the smaller model...

Source: The New Stack