Claude Fable 5.1 just won a new coding benchmark despite failing more than six out of 10 times. Its 38.8% score comes from Real-SWE, a benchmark from Y Combinator-backed Specific Labs that takes a different approach to testing coding agents. Instead of giving them problems pulled from public repositories, it drops them into private codebases from real companies and asks them to tackle problems similar to those engineers deal with every day. The code and its solutions aren’t publicly available, which makes it even less likely that they showed up in a model’s training data. Specific Labs can’t...

Source: The New Stack