Building voice agents that function reliably in production remains one of machine learning's persistent challenges. While foundational models have matured, evaluating their performance in real conversations lags behind. This week, researchers at ServiceNow unveiled EVA, a new end-to-end framework designed to stress-test conversational voice agents under realistic conditions.

Unlike prior benchmarks that isolate speech-to-text or dialogue management, EVA executes full bot-to-bot simulations over live audio. It generates two distinct scores: EVA-A for task accuracy and EVA-X for conversational experience. The initial release includes 50 airline scenarios, challenging agents to handle rebooking, cancellations, and voucher issuance without requiring human annotation for validation.

Results from testing 20 cascade and audio-native systems reveal an uncomfortable reality for engineering teams. There is a consistent tradeoff between accuracy and experience. Models optimized strictly for task completion often deliver robotic, frustrating interactions, while conversational fluency sometimes comes at the cost of factual reliability. Named entity transcription emerged as a primary failure point; mishearing a single confirmation code can derail an entire workflow.

EVA also highlights consistency issues often missed in single-run tests. Many systems show a significant gap between peak performance and reliable execution across multiple trials, measured via pass@k metrics. The framework is now available on GitHub, offering ML teams a way to measure not just if an agent finishes the job, but how it feels to use one. As voice interfaces become standard in enterprise workflows, tools like EVA suggest that optimizing for human experience is no longer optional—it is an engineering requirement. The team plans to expand coverage to multilingual support and prosodic quality assessment in future iterations.

Source: Hugging Face Blog