How I AI: Hamel Husain's Guide to Debugging AI Products & Writing Evals

The most important thing for improving AI products is looking at the data, but that is also the hardest part when outputs are non-deterministic and user inputs are messy. In this episode of How I AI, Hamel Husain explains how he debugs products with error analysis, writes evaluations tied to observed failures, and checks automated judges against human labels.

AI products are hard to debug because model outputs vary, user inputs are messy, and a single prompt change can improve one behavior while degrading another. A team needs a reliable way to tell whether a change actually addressed the failure it was meant to fix, without just "vibe checking" the results.

Instead of starting with complex tools, Hamel starts with real conversations, human review, and simple counts. His process is to inspect the data, describe the first observable error in each user interaction, group similar errors into categories, and then prioritize the categories that appear most often.

Hamel first demonstrates product error analysis and targeted evals, then shows how Claude, Gemini, and a private GitHub monorepo provide reusable context for his consulting and education work.

Finding and fixing product failures with systematic error analysis

Hamel’s workflow makes failures visible and gives the team concrete evidence for deciding what to fix first, replacing vague impressions with a prioritized list of problems.

Step 1: Log and examine real user traces

Hamel begins with traces from real product use. Depending on the product’s instrumentation, a trace may include user messages, model responses, tool calls, retrieval events, and other system context. Real inputs reveal the ambiguity, typos, and conversational patterns that polished test cases miss. Because traces can contain sensitive information, teams need appropriate notice, access controls, redaction, retention rules, and secure review tools.

• Tools: Platforms like Braintrust or Arize are designed for logging and visualizing these AI traces. You can also build your own logging infrastructure.
• Process: Sample interactions from the deployed system, following the product’s data-use rules. Synthetic examples can help before launch, but as Hamel notes, they often fail to reproduce the full, messy distribution of real-world behavior.
• Example: Hamel demonstrated the method with Nurture Boss, an AI assistant for property managers. One user asked, "Hello there, what’s up to four month rent?" This kind of ambiguous wording is exactly what a polished, internally generated test set would likely omit.

Step 2: Perform manual error analysis

Hamel manually reviews a sample of traces and writes a short note about each observed failure. This initial human review creates the error vocabulary the team will use for categorization and automated evaluation later. The process is so powerful that some of his clients feel like their main problem is solved at this stage, simply because they finally know what to do.

The method is sometimes called open coding or journaling. Hamel stops at the first error in a sequence because an upstream failure may explain several downstream symptoms. This is a prioritization heuristic, not a definitive root cause analysis.

• Process: Randomly sample about 100 traces, which might take a few hours. For each trace, read until you find the first incorrect, ambiguous, or high-friction part of the experience, then write a concise note about it.
• Focusing on the earliest visible failure keeps the first pass manageable. The team can investigate causality more deeply once the most common error categories are clear.

Example note: The assistant should have asked a follow-up question because the user’s intent was unclear.

Step 3: Create a custom annotation system

To speed up the review process, Hamel recommends building a lightweight annotation system. It can be as simple as an internal tool or a custom view in an observability platform. The important part is reducing friction so product managers or subject matter experts can move through traces and label issues quickly. For Nurture Boss, the team "vibe coded" a simple app to navigate traces from different channels like text messages and email.

• Tools: While platforms like Braintrust and Phoenix offer annotation features, a custom app can be tailored to your specific needs, channels (text message, email, chatbot), and metadata.
• A useful annotation tool minimizes clicks, keeps notes readable, and lets reviewers move quickly between the trace and its associated metadata.

Step 4: Categorize and prioritize errors by frequency

After collecting notes on individual failures, Hamel uses an LLM and human review to cluster similar issues. He suggests you can even paste the notes into a chatbot and ask it to bucket them into categories. He then counts how often each category occurs to decide where deeper investigation will have the most impact.

• Process: Aggregate all your notes. Use an LLM or manual review to group similar notes into error categories (e.g., "transfer and handoff issues," "tour scheduling issues," "incorrect information"). Count how many times each category appears.
• For Nurture Boss, the top categories included failed handoffs to a human agent and attempts to schedule a new property tour when a user wanted to reschedule an existing one. These counts created a ranked list of observed problems, which is a starting point for the product roadmap.
• "Counting is powerful." Frequency does not measure severity by itself, but it replaces vague impressions with a concrete starting point for prioritization.

Step 5: Write targeted evaluations

Once you know your biggest failure modes, you can write evals that test for those specific problems. Hamel breaks them into two broad categories:

• Code-based evals: Use deterministic checks when the expected behavior can be expressed precisely. One example is testing that internal identifiers from system context, like a UUID from a tool call, never appear in a user-facing response. Use synthetic secrets in test fixtures rather than exposing real credentials or customer data.
• LLM judges: Use a model when the failure requires semantic judgment, such as determining whether a handoff to a human was appropriate. The judge still needs a narrow, specific question and must be validated against human labels.
• Binary outcomes: Ask the judge one specific yes-or-no or pass-or-fail question. Hamel cautions against arbitrary aggregate scores, such as a helpfulness rating of 4.2 versus 4.7, because they don't provide a clear signal for "helpfulness."
• Validation: Hand-label a sample of data, compare the LLM judge’s ratings with those labels, and examine any disagreements. A high score on a dashboard is meaningless if the judge itself hasn't been checked. If the evals show everything is great but the team's perception is that the product is broken, you lose trust.
• Hamel cites research from the paper "Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences" when explaining why judge instructions improve through iteration. People are often bad at writing specifications upfront, but by reviewing errors and disagreements, a team can make its evaluation criteria more explicit.

Step 6: Iterate and improve with prompt engineering or fine-tuning

Once the evals represent observed failures, the team can test fixes. A fix might be as simple as adding the current date to the system prompt to handle relative-date questions like "tomorrow." Other fixes could involve providing better examples, changing retrieval behavior, or fine-tuning a model on carefully selected difficult cases. Hamel notes that if you do the work to build good evals, fine-tuning becomes almost free because you have already curated the high-signal data you need.

• Techniques: Experiment with prompt structures, add more examples to prompts, or even fine-tune models with data derived from your identified errors. As I learned with ChatPRD, even two incorrect words in a monster system prompt can significantly degrade tool calling quality.
• Advanced Analytics: For agent-based systems with multiple handoffs, you can use analytical tools like transition matrices to pinpoint where errors are most likely to occur between different agent steps (e.g., generate SQL to execute SQL).

Running a business with a centralized AI brain

Hamel applies the same systems thinking to his own business. He uses AI across consulting, writing, education, and operations, with a setup built around reusable context, fast iteration, and avoiding dependence on a single model provider.

Step 1: Centralized "Claude Projects" for every business function

Hamel uses reusable Claude workspaces (called Projects at the time of recording) for different business functions. Each one contains specific instructions, examples, context files, and writing guidance for a recurring task.

Examples include copywriting, consulting proposals, course content, lead magnets he calls "Lightning Lessons," and a legal-information workflow. Any legal drafts or summaries still require qualified human review before use.

Consulting proposals workflow

For a consulting proposal, Hamel adds a client-call transcript to a workspace that already contains his background, previous proposal examples, and rules for concise writing. He says the resulting Claude draft is almost ready to send and takes him about a minute to edit.

Course content workflow

For his Maven course on evals, Hamel built a Claude workspace packed with course materials, FAQs, transcripts, and Discord discussions. He uses it to generate standalone FAQs and other supporting content, guided by prompts that aggressively strip out filler and repetition.

Step 2: Custom software for content transformation with Gemini

Hamel also built internal tools for transforming long-form content into more digestible formats. A key part of that workflow relies on multimodal models like Gemini, which he uses to process video, transcripts, and slides together.

• His software combines a video transcript with screenshots of any slides to generate an annotated presentation. The result lets a reader scan the core argument and return to the source video only where needed.
• At the time of recording, Hamel used Gemini for the multimodal step that combines video, transcript, and slide images into a structured draft. I’ve heard from other guests that Gemini is a fan favorite for this kind of video information processing.
• He uses the output as supporting material for his educational work and always reviews it before distributing it to students.

Step 3: The GitHub monorepo as a "Second Brain" for AI workflows

Hamel's private GitHub monorepo stores notes, articles, prompts, business documents, and personal writing. The centralized AI second-brain workflow explains how he points multiple tools at that shared source instead of trapping context inside one model provider.

• Structure: The monorepo contains everything from his blog and the YouTube transcription project to copywriting instructions and proposals. Everything is interrelated.
• AI Access: He points his AI tools at this repo, providing a set of "Claude rules" within the repo itself. These rules instruct the AI on where to find specific information or context for different writing or development tasks (e.g., "if you need to write, look here").
• The repository reduces dependence on any one model provider and makes his instructions and context portable. It also concentrates sensitive information, so access permissions, secret scanning, backups, and clear rules about which files an AI tool may read are all essential.

Tie evals to failures people actually experience

Hamel’s debugging process begins with a sample of real user traces, a short note on the first visible failure, and a simple count of recurring error categories. Evals come only after the team understands these common failures, ensuring each test answers a specific, relevant product question.

The same structure appears in his business workflows: keep context reusable, let AI handle repeated transformations, and always reserve final approval for the person who is accountable for the result.

The tedious manual review is what makes Hamel's later automation trustworthy. The error-analysis workflow starts with about 100 traces, plain-language labels, and human agreement on what each test should detect before any dashboard becomes authoritative.