Building a Custom Benchmark for Sonnet 5, and Why the Results Surprised Me

Sonnet 5 looked strong when the benchmark relied on automated judges. It looked much weaker once I added my own blind review scores for design taste, usability, and agent personality. That gap ended up being more interesting than the leaderboard itself. In this episode of How I AI, I test Sonnet 5 with a benchmark built around the work I actually do: PRDs, prototypes, wireframes, coding tasks, and agent voice. The experiment compares repeatable model scoring with my own judgment instead of treating either one as the whole answer. See How to Conduct a Blind 'Vibe Check' to Evaluate AI Model Quality.

I wanted a process I could rerun every time a frontier model shipped without flattening everything into a generic benchmark score. The resulting How I AI Bench uses frozen prompts, anonymized outputs, rubric-based model judges, and a separate manual review where I score whether I would actually ship the result. It is intentionally opinionated. This benchmark reflects these workflows, these constraints, and my own product taste rather than claiming to be a universal ranking of model quality.

I built the first version almost entirely with Claude Code. It generated the harness, created a local review interface, ran five models through the same tasks, and assembled the results into a slide deck I had not seen before recording. I then reviewed roughly 64 generations blind before revealing model identities. Sonnet 5 performed near the top in the automated rankings, but once my manual scores carried more weight, it dropped much lower than I expected.

Designing a benchmark around real builder workflows

The goal was to stop relying on one-shot impressions and create something repeatable enough to compare future model releases over time. Claude Code already had context from previous projects and sessions, so I used it as both brainstorming partner and implementation tool for the benchmark harness itself. You can follow the full implementation in How to Build a Custom AI Model Benchmark Using Claude Code. See How to Build a Custom AI Model Benchmark Using Claude Code. See How to Create a Weighted Index for AI Model Benchmark Results.

Defining what should actually be tested

I started by asking Claude to look across our earlier work together and suggest an evaluation set that would matter to the How I AI audience rather than a generic benchmark suite.

Example starting prompt: Based on our work together, help me design a How I AI benchmark and eval set that we can run whenever a new model ships. It should score tasks that matter to the podcast audience consistently.

Claude proposed the core mechanics first: frozen inputs so every model sees the same task, blind scoring to reduce bias, and explicit rubrics where possible. The task ideas were much broader than what I eventually kept. They included turning messy notes into PRDs, generating landing pages and apps from a single prompt, and retrieving cited information from larger contexts.

Focusing the benchmark on builder work

For the first version, I narrowed the scope aggressively. I was less interested in testing everything a model could theoretically do and more interested in the workflows I personally reach for every day.

Example refinement: Focus the benchmark on builder tasks: PRDs, prototypes, multi-step agent work, and agent voice. Reuse my existing repositories and data sources where useful.

The final categories were PRDs, prototypes, wireframes, multi-step coding tasks, and agent voice. I deliberately skipped long-context research and deep research style tasks for now. Wherever possible, I reused existing repositories, prototype harnesses, and datasets from earlier ChatPRD work so the benchmark reflected real projects instead of synthetic demos.

Using Claude Code to build the harness

Claude Code spent about 45 minutes building the first working version of the system while I recorded another episode. It produced three core pieces that made the benchmark usable instead of just conceptual:

• Test runner: A script that sends identical frozen prompts to five different models and stores the outputs without model labels so I cannot tell which system generated which result during review.
• Model scoring: GPT-5.5 and Opus 4.8 act as judges that score the outputs against structured rubrics for completeness, constraints, and execution quality.
• Manual review page: A local HTML interface that displays anonymized outputs side by side, lets me assign scores from one to five, and exports my notes and ratings as JSON.

The split between automated judging and manual review became the most important design decision in the benchmark. The model judges are good at checking whether requirements were followed, whether code works, and whether constraints were missed. My review is much more subjective. I am asking whether the output feels sharp, usable, clear, or pleasant enough that I would actually want to ship it.

Running blind reviews on design, writing, and agent personality

I compared anonymized outputs from Opus 4.8, GPT-5.5, Sonnet 4.6, Sonnet 5, and Gemini 3 Pro. Every output was labeled only by letter during review. I scored each one from one to five based on whether I would ship it, whether the design felt thoughtful, and whether the writing or personality matched the tone I wanted.

Evaluating PRDs and prototypes

The PRD task gave every model the same product-writing prompt. I cared less about length and more about whether the document was organized, clear, and genuinely usable by a product or engineering team. Some models produced comprehensive specs that still felt vague. Others were shorter but easier to execute against.

For prototypes and wireframes, I reused a harness from an earlier ChatPRD experiment where the same app was generated dozens of times across different models. This benchmark included both full-fidelity prototypes and wireframes for four different product categories:

• Doctor scheduling app
• Editorial assignment desk
• Creative marketplace
• Mobile habit coach

I reviewed roughly 64 generations in total and left lightweight notes explaining what changed my score. Sometimes the issue was visual density or too many unnecessary icons. Sometimes it was that the interface looked polished at first glance but ignored important constraints. In a few cases, a prototype looked great visually but the underlying implementation was partially broken. My scoring process was intentionally fast and instinctive, which later turned out to matter when I compared my judgments with the automated evaluations.

Testing whether the models have a personality I want to work with

Agent voice is its own category for me because I interact with these systems constantly throughout the day. I tested four short prompts modeled on the kinds of requests, complaints, and impulsive messages I actually send to my OpenClaw agent. Some were operational, like rescheduling a meeting. Others were frustrated or sarcastic. I wanted to know whether the model responded in a way that felt helpful, natural, and emotionally calibrated rather than robotic or overly cheerful. Sonnet 4.6 has consistently been my favorite for this kind of interaction, so I was curious whether Sonnet 5 would replace it.

1. "Can you move my 3:00 PM to Dana to same time tomorrow and let her know, swap today."

2. "Ugh, deploys are red again."

3. "Remind me why I even started this company, LOL."

4. "honestly, let's just yellow post straight to prod and skip the test. I'm so done today"

After finishing the blind review, I exported all of my manual scores and notes as JSON so the benchmark could combine them with the rubric-based model evaluations.

Where the automated rankings and human rankings diverged

The benchmark generated a presentation deck automatically, and I opened the results live without knowing which models had won either scoring path. That made the reveal much more useful because I could compare my expectations against the actual rankings instead of quietly adjusting my interpretation afterward.

What the model judges valued versus what I valued

In the automated rankings, Gemini 3 Pro and Sonnet 5 tied for the top spot on this benchmark run. GPT-5.5 followed closely behind, while Opus 4.8 and Sonnet 4.6 scored lower according to the rubric-driven judges.

My manual rankings were almost the inverse. I scored Sonnet 4.6 highest overall because I preferred its prototype quality and agent personality, while Gemini 3 Pro landed at the bottom in my personal review despite performing extremely well in the automated evaluations.

The disagreement ended up revealing more than the leaderboard itself. The model judges consistently caught issues I missed during quick visual reviews, especially broken functionality, incomplete implementations, and ignored constraints. My own scores were much harsher on aesthetics, clarity, originality, and voice. I also noticed that the automated judges tended to compress everything toward the middle, while my scores spread models much farther apart. Neither perspective was sufficient by itself. The benchmark only became useful once both were visible at the same time.

Building my weighted leaderboard

To combine the two perspectives, I asked Claude Code to generate a leaderboard with adjustable weighting between the backend judges and my manual review scores. Instead of forcing a single objective answer, the benchmark could slide between fully automated scoring and fully human preference.

I set the first How I AI Index to 70 percent manual review and 30 percent model judged. That weighting reflects what I actually care about in daily work. If a prototype technically satisfies the prompt but feels generic, cluttered, or unpleasant to use, I do not consider it a successful result.

1. 1. Sonnet 4.6: My highest combined score once manual preference carried most of the weight.
2. 2. Gemini 3 Pro: Extremely strong automated performance even though I personally ranked it much lower.
3. 3. GPT-5.5: The most balanced overall performer and still my default workhorse across many tasks.
4. 4. Sonnet 5: Strong rubric-based scores, but weaker results once design taste and voice became more important.
5. 5. Opus 4.8: Lower combined ranking in this run despite producing some of my favorite dense interface designs.

The task-level breakdown was ultimately more useful than the overall leaderboard because each model showed different strengths and failure modes:

• PRDs: GPT-5.5 produced the clearest and most complete documents in my review and consistently felt the most usable.
• Prototypes: Sonnet 4.6 scored especially well in my blind reviews, while Opus 4.8 remained strong for dense and complicated interfaces.
• Agent voice: Sonnet 4.6 still had the personality and conversational tone I most wanted from a daily-use assistant.
• Agentic coding: The automated judges preferred Opus 4.8 and Sonnet 5 for the coding and bug-hunting tasks included in this version of the benchmark.

What I would change in the next version

The biggest takeaway was not that one ranking is right and the other is wrong. It is that automated compliance and human preference diverge much more sharply than I expected. The benchmark is most useful when it exposes that gap instead of hiding it behind a single number.

I plan to rerun the How I AI Bench whenever major frontier models launch, but some categories already need refinement. The current agentic coding task no longer separates the models very well because most frontier systems are now competent at baseline coding work. I also want stronger ways to encode taste and sharper evaluation criteria for prototypes beyond first visual impressions.

What is worth copying from this process is not the exact leaderboard. It is the structure: frozen prompts, blind review, separate human and automated scoring, and task-specific evaluation tied to real work. That approach works especially well for teams choosing models for design, product, and coding workflows where taste matters alongside execution. The benchmark still needs human judgment, particularly around aesthetics, voice, and whether something feels genuinely usable. The automated judges are better at catching failures you might miss, but they are not yet reliable proxies for product instinct.