How I AI: Hamel Husain's Guide to AI Quality with Error Analysis and Claude Workflows

If you're tired of "vibe checking" your AI product, this episode of How I AI is for you. AI consultant and educator Hamel Husain demonstrates a systematic method: read real user traces, label the earliest meaningful failures, count recurring patterns, and only then build evaluations for the problems you observed.

Hamel demonstrates the process with Nurture Boss, a virtual leasing assistant for property managers. The team had a working prototype, but they were stuck in a loop of prompt changes, unable to tell whether one fix improved quality overall or just created a new failure elsewhere.

Hamel also opens his own operating system: specialized Claude Projects, a private GitHub monorepo for business context, and a Gemini workflow that turns presentations into annotated notes.

The quality method comes first. It replaces broad, unactionable scores and gut feelings with a short, prioritized list of observed failures, test cases drawn directly from those failures, and automated judges that have been validated against human labels.

Find the real problems with error analysis

A single prompt edit is not evidence of improvement. Hamel begins with production traces so the product roadmap reflects what users actually attempted and where the system actually broke.

"It has an immense quality. It's so powerful that some of my clients are so happy with just this process that they're like, that's great, Hamel, we're done. And I'm like, no, wait. We can do more."

Step 1: Inspect real user traces

A trace captures the full sequence behind an interaction, including user messages, model responses, retrieval, and tool calls. For a conversational product, that sequence reveals failures that a simple final-output score can miss.

Nurture Boss used observability tools like Braintrust and Arize Phoenix to inspect and annotate traces. Hamel emphasized that the specific viewer matters less than making real interactions fast to read and label.

Real users do not write like test fixtures. They send fragments, misspellings, shorthand, and ambiguous requests. One Nurture Boss trace began with:

Hello there. What's up to four month rent?

The assistant misinterpreted the message as a question about rent specials instead of asking for clarification. That first misunderstanding shaped everything that followed.

Step 2: Label the earliest meaningful failure

Hamel uses an error-analysis technique called open coding, which he describes as journaling what went wrong. The reviewer reads each trace and writes a short, plain-language note.

1. Start with a random sample of about 100 traces.
2. Stop at the most upstream error: the first failure likely to cause later problems in the sequence.
3. Write one plain-language sentence. For the ambiguous rent message, the note could be: "Should have asked follow-up questions because user intent was unclear."

This initial pass is manual by design. For Nurture Boss, a few focused hours of review produced enough evidence to identify the largest patterns of failure.

Step 3: Group and count the errors

A hundred short notes are far easier to reason about than a hundred full conversations. The next task is to turn them into a usable failure taxonomy.

• Export the annotations from the trace viewer or custom review interface.
• Ask a model to propose categories for the notes.
• Review the categories yourself, correct the grouping, and separate recurring product problems from isolated cases.

Then, you count the examples in each category. Frequency does not capture every kind of severity, but it gives the team a grounded, data-driven starting point for prioritization.

The Nurture Boss review surfaced several recurring problems:

• Transfer and handoff failures when the assistant tried to route a user to a person.
• Tour-scheduling failures in which a rescheduling request created another tour instead of changing the existing one.
• Follow-up failures when the assistant did not respond appropriately to a user’s question, along with some cases of incorrect information.

The team now had observed examples, raw counts, and a direct connection between each failure category and the underlying user traces.

Step 4: Write evaluations for observed failures

Error analysis determines which evaluations are worth building and maintaining, and it supplies the initial examples for testing them. Hamel separates objective checks from judgments that require human interpretation.

1. Code-based evaluations handle deterministic checks. If an internal user ID must never appear in a response, code can test whether the output contains one.
2. LLM judges can handle more subjective questions, such as whether a handoff failed. The judge should evaluate one defined problem rather than produce a general quality score.

Step 5: Validate every model-based judge

Vague scores like "helpfulness: 4.2" are hard to act on because the number does not identify a concrete failure. A higher score also means little if the judge itself has not been checked against human decisions.

Hamel’s guidance, supported by research like "Who Validates the Validators?", has three parts:

1. Use a binary output for one specific question, such as whether a required handoff succeeded.
2. Hand-label a set of examples to establish the expected answers.
3. Measure the judge’s agreement with those labels. If the automated score contradicts the experience of users and reviewers, fix the judge before relying on its dashboard.

Step 6: Fix the system and repeat the analysis

The evidence points to the right intervention. One Nurture Boss failure was caused by a system prompt that did not include the current date, so the assistant could not resolve "tomorrow." Other failures might come from retrieval, tool behavior, prompt design, or system architecture. Fine-tuning is only one option and usually not the first one to try.

Hamel’s personal OS for consulting and content

Hamel applies the same emphasis on explicit, reusable context to his own work. He uses Claude Projects for specialized jobs, while a private GitHub monorepo keeps all his related source material available across different tools.

Step 1: Create specialized assistants for defined jobs

Each of Hamel's Claude Projects receives a curated set of documents, examples, and instructions for one area of his business.

• Consulting proposals: Hamel provides examples of successful proposals and a client-call transcript. The project creates an almost complete draft that takes him about a minute to edit.
• Course support: the project uses the course book, a detailed FAQ, transcripts, and Discord discussions to help instructors answer questions and create standalone learning material.
• Legal review: a separate project helps Hamel examine documents, with Hamel retaining responsibility for consequential decisions.
• Copywriting: the instructions reflect his preference for short, direct writing, including:

"Do not add filler words. Don't repeat yourself. Get to the point."

Step 2: Keep related knowledge in a single repository

The shared foundation for his work is a private GitHub monorepo containing his blog, project files, notes, prompts, data sources, and supporting tools.

A set of repository instructions tells coding agents what the collection contains and where to look for writing, proposals, or course materials. This structure lets Hamel point Claude Code, OpenHands, or another tool at the same source of truth, avoiding vendor lock-in with a single model provider.

Step 3: Turn presentations into annotated notes

Hamel’s Gemini workflow ingests a YouTube presentation, pulls the transcript and video, captures screenshots of the slides, and writes a summary beneath each one. It gives his students a quick way to review the substance of a long presentation, but he notes that it remains derivative material that should always be checked against the original recording.

Error analysis first, evaluations second

The sequence is the most important part of this process. First, read real traces. Second, label the first important failure. Third, group and count the notes. Only then should you write tests for the problems that actually recur. Starting with a generic evaluation dashboard skips the essential evidence needed to define what is worth judging.

This method is worth copying for any team that feels stuck just "vibe checking" their AI product. That first manual pass is the most critical part, creating a roadmap, test cases, and hand labels all at once. It still requires focused human judgment from a subject matter expert, often the product manager, to identify the most important upstream failures. But once those pieces exist, the team can automate monitoring with confidence, knowing exactly what their quality scores mean.