How I AI: Ankur Goyal's Playbook for Agent-Driven Benchmarking and AI Evals

AI agents are starting to handle the kind of engineering work teams used to avoid because it was too tedious, too risky, or too expensive to benchmark properly. In this episode of How I AI, Braintrust CEO Ankur Goyal walks through two concrete examples: week-long infrastructure experiments that test database performance changes at scale, and an eval workflow that continuously improves an AI documentation assistant.

The infrastructure setup is intentionally rigorous. The agent gets a measurable performance target, production-like workloads, and enough compute to exhaustively test combinations that a human team would rarely have time to explore. The product workflow applies the same idea to AI behavior: start with real user questions, generate answers, score them across a dataset, then refine the evaluator itself using expert feedback.

Ankur’s framing of evals is especially useful because it shifts the conversation away from prompts and toward outcomes. An eval defines what good looks like without prescribing the implementation. In practice, that makes it feel closer to a modern PRD: examples, constraints, scoring criteria, and repeatable tests that survive prompt changes, model swaps, and product iteration.

Using coding agents to run exhaustive infrastructure benchmarks See How to Use AI Coding Agents for Exhaustive Infrastructure Benchmarking.

Braintrust uses coding agents for engineering problems that benefit from relentless experimentation. The agent is not asked to "improve the database" in the abstract. It gets a concrete performance problem, access to realistic workloads, and a benchmark loop that can run for days while continuously testing alternatives.

"Now that models are so good at actually writing code, one of the best things that we can do is create really hard evals. And I'm not talking about like AI evals. I mean things like why is this query so slow? And if you create the right tests and success criteria for a model, then it can be really creative and it can work on this stuff in the background and actually try to improve a bunch of things."

The important shift is that the human engineer defines the outcome and the constraints, while the agent handles the exhaustive search process.

Define the bottleneck and the success criteria

One example involved a slow search path across billions of traces spanning a 90 day window. Users were effectively searching for a needle in a haystack across huge datasets, and Braintrust needed lower query latency without guessing blindly at optimizations. The benchmark defined the measurable outcome, but the agent had flexibility in how to achieve it.

Run the experiments in a real environment

The experiments run inside coding-agent sessions with access to the repository, benchmark tooling, representative data, and enough compute to execute real workloads. That distinction matters. The system is not generating speculative recommendations in a document. It is actually running the benchmarks, comparing implementations, and measuring the tradeoffs.

• Data: The benchmark workload needs to resemble production closely enough that the results are trustworthy. Ankur described using production-like datasets and, in some cases, carefully controlled access to data stored in object storage.
• Compute and sessions: He runs multiple concurrent agent sessions in tmux and offloads heavier experiments to remote machines such as EC2 when local hardware is insufficient for large-scale benchmarking.

Search the full performance matrix

For the slow-query investigation, the agent explored several layers of the storage and execution stack simultaneously instead of testing one idea at a time:

1. Column formats: It compared the existing Tivy column store against multiple open-source alternatives to measure differences in storage and retrieval behavior.
2. Execution engines: It paired storage formats with different execution engines so the team could evaluate the full interaction matrix rather than optimizing one isolated variable.
3. Index choices: After roughly a week of continuous experiments, the system identified bloom filters as surprisingly effective for this workload despite the team initially dismissing them.

Ankur’s argument is not that agents magically produce perfect infrastructure code. The advantage is persistence and coverage. A human team usually prioritizes a handful of benchmarks, debates the rest, and moves on. An agent can keep running experiments, uncover second-order effects, and test assumptions that would otherwise be skipped because they are too time consuming. The quality of the outcome still depends on benchmark design, realistic workloads, and strong human review.

Treating evals like executable product specifications

For AI products, Ankur applies the same philosophy. Instead of obsessing over prompts in isolation, he treats evals as the operational definition of product quality. An eval combines examples, desired behaviors, and scoring logic so the team can rerun the same tests after changing prompts, retrieval systems, or models.

In this episode of How I AI, he demonstrated the process live using a Braintrust documentation assistant.

The target behavior

The assistant needed to answer Braintrust documentation questions accurately, concisely, and with useful implementation guidance.

Build a dataset from real questions

Ankur started with actual documentation questions people ask, including setup instructions and product behavior questions. He emphasized that the dataset is primarily a set of representative inputs. The answers are not necessarily hand-labeled ground truth upfront.

Create a simple baseline system

The initial version was intentionally lightweight: a straightforward prompt, GPT-4.5 mini as the model in the demo, and a retrieval connection to the Braintrust documentation through an MCP-style retrieval server. Part of the exercise was seeing how far a minimal implementation could get before layering on additional complexity.

Generate the evaluator with AI

Instead of manually writing every evaluation rule, Ankur asked an AI model to convert his quality criteria into a reusable evaluator prompt. He specified things he cared about, including concise code snippets, consistency in language choice, and overall clarity.

Hey, can you come up with a good scoring function for these outputs? I care about having concise code snippets only using one language, and avoiding em-dashes.

Braintrust used Claude to inspect the task definition, the dataset questions, and the generated answers, then synthesize a detailed scoring rubric. That evaluator could then apply the same criteria consistently across the entire dataset instead of relying on scattered manual reviews.

Review aggregate patterns, not just examples

The resulting run produced scores for dimensions such as accuracy, concision, and code quality. Ankur said he increasingly prefers aggregate analysis over isolated spot checks because teams otherwise end up playing whack-a-mole with prompts. Looking across the full dataset makes regressions and patterns visible, while still allowing reviewers to inspect individual outputs when something looks wrong.

The David loop: scaling taste without removing the expert

Ankur was careful to separate measurable quality from human judgment. He described an ongoing feedback loop with Braintrust designer David, whose product instincts and aesthetic judgment are difficult to encode directly into a first-pass rubric. You can follow the full implementation in How to Scale Expert Judgment in AI Systems with a Human Feedback Loop. See How to Scale Expert Judgment in AI Systems with a Human Feedback Loop.

The workflow is iterative. The team first improves the system quantitatively through evals, then brings a smaller set of outputs back to David for a qualitative review and a final vibe check.

1. 1. Run the eval suite and improve the measurable criteria across the dataset.
2. 2. Ask David to review the outputs and identify what still feels off, awkward, or low quality.
3. 3. Convert recurring feedback into new evaluation criteria, examples, or scoring guidance so the same mistake is less likely to repeat.

The important point is that David is not replaced by the system. His judgment becomes more scalable. Instead of manually reviewing everything, his standards shape the evaluator over time, allowing the team to apply his taste across a much larger surface area while still routing new or ambiguous cases back through human review.

What actually connects these workflows

The infrastructure experiments and the documentation evals are structurally similar. In both cases, the team defines a measurable outcome, gives the agent access to a representative environment, runs enough trials to surface patterns, and reviews the outputs critically instead of trusting a single run.

That rigor depends on strong foundations. Infrastructure agents need dependable benchmarks, CI systems, realistic workloads, and safe execution environments. Product evals need representative datasets and scoring criteria that actually correlate with user experience rather than arbitrary metrics.

The recurring theme throughout this episode of How I AI is that the highest-leverage work increasingly happens at the specification layer. The hard part is not manually implementing every step. It is defining the problem, constraints, and evidence clearly enough that an agent can explore solutions autonomously.

The most transferable part of Ankur’s approach is not the specific tooling. It is the insistence on measurable loops. This works especially well for persistent technical bottlenecks, retrieval quality problems, or repetitive product-review cycles where humans currently rely on intuition and spot checks. The agent can search broadly and tirelessly, but human judgment still matters at the boundaries: choosing representative workloads, deciding whether metrics reflect reality, simplifying overly complex systems, and knowing when a result merely scores well versus actually feels right to users.