Decision models have exploded onto the scene recently, starting with TypeSafe’s Jev, followed by Perplexity’s Decider, Liquid’s d1, Cloudflare’s Clef, and OpenAI’s Decisions API1. These models promise fast, inexpensive...
Anthropic released new eval tooling for Claude Code. Their claude-api plugin now includes a new build_eval and hill-climb command that helps you build evals, check the graders, and improve your application against them....
We compared 100 human annotated traces against automated eval systems. Here's what we found.
For the past 3 years, AI evals have been my professional focus.1 The most common objection I hear to evals is “our product is hard to eval”. This objection is a product smell. Artifacts that are hard for you to verify...
img.img-fluid { border: 1px solid rgba(255, 255, 255, 0.25); border-radius: 4px; } Is the heyday of the data scientist over? The Harvard Business Review once called it “The Sexiest Job of the 21st Century.”1 In tech,...
Today, I’m publishing evals-skills, a set of skills for AI product evals1. They guard against common mistakes I’ve seen helping 50+ companies and teaching 4,000+ students in our course. Why Skills for Evals Coding...
Programmers love to proclaim they’ve found the best tool. Paul Graham called Lisp his “secret weapon.” DHH described Ruby as “a magical glove that just fit my brain perfectly.” Pieter Levels ships million-dollar...
This document curates the most common questions Shreya and I received while teaching 700+ engineers & PMs AI Evals. Warning: These are sharp opinions about what works in most cases. They are not universal truths....
Over the past year, I’ve focused heavily on AI Evals, both in my consulting work and teaching. A question I get constantly is, “What’s the best tool for evals?”. I’ve always resisted answering directly for two reasons....