A new benchmark from Microsoft Research quantifies a problem many data engineers have suspected: handing long editing workflows to frontier LLMs quietly degrades the documents they're supposed to improve. The DELEGATE-52 benchmark, detailed in arXiv:2604.15597, tests models including Gemini 3.1 Pro, Claude 4.6 Opus, and GPT 5.4 across 52 domains from crystallography to Python code. After 20 round-trip editing instructions, the average corruption rate across 19 models reaches 50 percent; frontier models average 25 percent. Only Python code cleared the 98 percent fidelity bar for most models. Every other domain saw performance collapse.

The architecture of the benchmark is instructive. Each test starts with authentic seed documents, then applies chains of reversible edits—forward changes followed by inverse operations—designed to measure semantic preservation. Domain-specific parsers score output by structural elements like ledger entries or code functions, catching errors that generic metrics like BERTScore miss. Critical single errors account for roughly 80 percent of total degradation. Weaker models tend to delete content; frontier models corrupt it by altering numbers, reordering steps, or inserting subtle factual mistakes that compound across interactions.

Scaling reveals a grim pattern. GPT 5.4 retains 91.4 percent fidelity on 1,000-token files after 20 interactions but falls to 59.9 percent on 10,000-token ones. Degradation continues past 50 or 100 interactions without plateauing. Distractor files—related but irrelevant documents common in real desktops—compound the harm. Agentic tool use (file reading, code execution) added overhead without fixing the core issue, dropping performance an extra 6 percent on average.

For data and ML engineers, the engineering trade-off is clear: current models are unreliable delegates for multi-step document workflows. Deploy them for narrow, short-horizon tasks where errors stay containable, or invest in human oversight loops, versioning layers, and validation layers that treat every AI edit as provisional. The latter raises costs and slows the productivity gains leaders seek. DELEGATE-52 is publicly available for teams to track progress. The data says trust remains premature.

Source: Webpronews