On BrowseComp-Plus, GPT-4.1 answered 93.49% of the questions correctly when it was given the supporting documents directly. When an agent used the same model but relied on BM25 to search for evidence to pass to the model, accuracy fell to 14.58% . That difference matters because a system that gets the answer right 93% of the time is far more useful than one that gets it right only 15% of the time. An agent can look excellent in a controlled demo. The corpus is clean, the task is familiar, and the relevant material is easy to find. Production is messier. Documents may be duplicated, outdated...

Source: The New Stack