A new benchmark from London-based AI firm General Reasoning, called KellyBench, tested eight leading large language models as autonomous betting agents across a simulated Premier League season. The results were a net loss, highlighting persistent challenges in applying these models to dynamic, real-world prediction tasks.

The study provided each model with extensive historical team and match statistics, then tasked them with building predictive models to maximize returns over the 2023-24 season. The agents placed bets on match outcomes and goal totals, adapting to new data as the virtual season progressed, all without internet access for live results.

From an engineering perspective, the setup tested core system capabilities: continuous integration of new data, risk management, and long-horizon decision-making. The models struggled architecturally with these sequential, probabilistic tasks. Performance varied widely. Anthropic's Claude Opus 4.6 posted the smallest average loss at 11%, nearly breaking even in one run. Google's Gemini 3.1 Pro showed high volatility, achieving a 34% profit in one attempt but bankruptcy in another. xAI's Grok 4.20 failed most notably, going bankrupt once and failing to complete two other runs.

The technical takeaway for practitioners is clear: while LLMs excel at bounded tasks like code generation, their agentic application in noisy, temporal environments remains an unsolved design problem. The benchmark underscores the engineering trade-off between raw reasoning power and the robust system architecture needed for sustained real-world analysis.

Source: Ars Technica