Building a Custom Benchmark for Sonnet 5, and Why the Results Surprised Me

Claude Sonnet 5l just dropped from Anthropic. They're pitching it as their most 'agentic' Sonnet model yet, promising Opus-level performance at Sonnet-level prices. For a while now, I’ve been testing new models with what I’d call a “vibe check”—I’d one-shot a landing page in Cursor or Claude Code and share my thoughts. I’ve done this for several models, but I've always felt the feedback was a bit soft and, more importantly, not repeatable. I wanted something more rigorous.

What I like about my process, though, is that it’s a Claire Vo benchmark. I have a point of view on what's good, and I don’t want to lose that taste to a generic LLM-as-judge evaluation. So, I decided to build something new: the How I AI Bench. It’s a set of benchmarks I can run every time a new model comes out to consistently score the tasks my audience of builders and creators actually cares about—writing PRDs, solving bugs, and one-shotting designs.

In this post, I’m going to show you exactly how I built this benchmark with the help of Claude Code. We're going to put Sonnet 5 to the test in a blind trial against some of the other top models, and I’ll walk you through the entire process, from the initial prompt to the final, surprising results. It’s a journey that started as a Sonnet 5 review and ended with a whole new way for us to measure what 'good' really means in AI.

Workflow 1: Building the 'How I AI Bench' with Claude Code

The whole point of this exercise was to move from one-off feelings to a repeatable, structured process. I didn't want to lose my own qualitative feedback, but I needed to wrap it in a system that could be run consistently over time. My tool of choice for this was Claude Code, primarily because it has access to all my previous sessions and can use that context to understand what I’m trying to build.

Step 1: The Brainstorming Prompt

I started with a simple prompt, asking Claude to act as a partner in designing the benchmark. I specifically called out that it should use our past work together to inform its suggestions. This is a huge advantage of using tools that maintain session history.

Based on our work together, can you help me brainstorm how I AI benchmark and eval set we can test every time a new model comes out to consistently score different tasks That would be relevant to our podcast audience.

Claude came back with some solid design principles for a good benchmark: frozen inputs, blind scoring, and a clear rubric. It also proposed a list of tasks, from turning messy notes into a PRD to generating a landing page and sifting through context to find cited information.

Step 2: Focusing on Builder Tasks

I loved all the ideas, but to make this first version manageable, I needed to narrow the scope. I corrected myself and told Claude to focus specifically on the tasks that builders like us face every day.

Let's actually focus on task for builders, PRDs prototypes, agentic, multi-step and agentic voice basically does a pass the vibe check in my open claw.

I explicitly told it to ignore long-context research for now and that it could use my existing repos and data sources to build the test harness. This refinement was key to making the benchmark relevant and actionable.

Step 3: Generating the Evaluation Harness

This is where it all came together. Claude Code didn't just design the benchmark; it built it. Over about 45 minutes, it generated a complete evaluation system. This included:

• The Test Runner: A script to run the same set of prompts across five different models in a blind test.
• The LLM-as-Judge Scorer: A backend process to have LLMs (I used both GPT-5.5 and Opus 4.8) score the outputs based on a rubric.
• The Human Vibe-Check Page: A local HTML file that presented all the model outputs side-by-side, without revealing which model was which, allowing me to provide my own manual scores.

This hybrid approach was exactly what I wanted—a way to combine automated, objective scoring with my own subjective, taste-based feedback.

Workflow 2: The Manual 'Vibe Check' — Scoring Models on Taste

With the evaluation harness ready, it was time for the fun part: the blind taste test. The local HTML page was my interface for this, presenting outputs from five anonymized models: Opus 4.8, GPT-5.5, Sonnet 4.6, Sonnet 5, and Gemini 3 Pro. I had to score each output from 1 to 5 based on a simple gut check: "Would I ship this? Does it sound like me?"

Scoring PRDs and Prototypes

The first tasks were focused on product and design. I reviewed each of the generated Product Requirement Documents (PRDs), leaving notes like "comprehensive and clear" and giving each a score.

Next up were prototypes. I reused a harness from a project I wrote about on my Substack where we generated the same app 82 times. This allowed me to test both full-fidelity prototypes and wireframes across a range of complex applications:

• A doctor scheduling app
• An editorial assignment desk
• A creative marketplace
• A mobile habit coach app

I went through all 64 generations quickly, giving my gut-feel scores and notes. I've been a product and design leader for a long time, so I can eyeball this stuff pretty fast. I left feedback like "Not bad, simple," or "Too many icons at the top."

Scoring Agentic Voice

This is a huge one for me. If you’ve listened to the podcast, you know I’m incredibly picky about the personality of my AI agents, especially my Open Claw agent. Sonnet 4.6 has been my favorite so far, so I was curious to see how the others would stack up. I tested them with four prompts that are very me:

1. "Can you move my 3:00 PM to Dana to same time tomorrow and let her know, swap today."

2. "Ugh, deploys are red again."

3. "Remind me why I even started this company, LOL."

4. "honestly, let's just yellow post straight to prod and skip the test. I'm so done today"

I scored each response based on whether the agent's voice was one I'd want to interact with daily. After rating everything, my manual scores were saved to a JSON file, ready to be combined with the automated results.

Workflow 3: The Big Reveal and Analyzing the Results

This is where my jaw hit the floor. I had the benchmark process generate a slide deck with the final results, which I opened for the first time live on the show. I had no idea what to expect.

The Automated Leaderboard vs. My Taste

The first slide showed the LLM-judged leaderboard. And the winner was... a tie! Gemini 3 Pro and the brand-new Sonnet 5 came out on top. My personal favorite, GPT-5.5, was right behind them. At the bottom were Opus 4.8 and Sonnet 4.6.

But then came the next slide, which compared the automated scores to my manual 'taste' scores. We disagreed. A lot. In fact, my preferences were almost the complete opposite of the machine's. I rated Sonnet 4.6 as the best and Gemini 3 Pro as the worst.

Why the difference? LLM judges are helpful for catching functional errors like broken code or ignoring constraints, which my quick eyeball pass missed. However, they tend to be 'easy graders,' rating everything towards the middle of the bell curve. They lack a strong sense of taste and don't see the little things that a human eye does—the sharpness of a design, the personality in a turn of phrase. This divergence proved that the human vibe check isn't just nice to have; it's essential.

The Final, Weighted Leaderboard

Given the conflicting results, I couldn't just accept the automated score. I asked Claude to generate one last thing: a final leaderboard page with a slider that would let me create a weighted index, balancing my opinion against the backend performance metrics. It gave me ultimate power!

I decided on a 70% Claire-judged, 30% backend-judged score. It's my podcast, after all. And the final, definitive How I AI Index revealed a new champion.

1. Sonnet 4.6: It has the best vibes and performs well enough.
2. Gemini 3 Pro: A strong performer despite my taste reservations.
3. GPT-5.5: My personal workhorse, still a top contender.
4. Sonnet 5: The new model, surprisingly, landed near the bottom.
5. Opus 4.8: The most expensive model was last in this weighted ranking.

This also led to a clear, task-based recommendation:

• Writing PRDs: Use GPT-5.5 for comprehensive, clear docs.
• Prototyping: Sonnet 4.6 is surprisingly good. For more complex, dense UIs, Opus 4.8 still shines.
• Agent Voice: Sonnet 4.6 wins the personality contest, hands down.
• Agentic Coding: The LLM judges preferred Opus 4.8 and Sonnet 5.

My Takeaways and What's Next

This whole adventure started as a simple model review and became so much more. The biggest lesson is that my taste matters, and it often diverges sharply from automated metrics. Human oversight is not a bug; it's a feature.

Going forward, the 'How I AI Bench' is here to stay. We'll run it for every major model release. My next step is to refine it further—retiring tasks that all models are already good at (like the basic agentic coding task) and finding new ways to encode more of my 'taste' into the automated judgment itself.

This was a fascinating process, and it fundamentally changed how I think about evaluating AI. The model you choose really depends on the job, and the 'best' model isn't always the newest or most expensive one. It's the one that has the right blend of performance and taste for what you're trying to build.

Episode Links

• YouTube: Watch the full episode
• Substack: Read my thoughts on this benchmark