Step01
Log and Examine Real User Traces
Log the complete sequence needed to understand each interaction: user input, model output, retrieval, tool calls, latency, version metadata, and outcome. Redact unnecessary personal data before analysis.
IntermediateEngineering
How to Systematically Analyze and Debug Errors in AI Products
Debug an AI product from observed behavior instead of vibes: sample real traces, annotate the earliest meaningful failure, count a grounded error taxonomy, and build narrow evals that can tell whether a change fixed the problem without creating a regression.
Source episode · 05:00
How I AI: Hamel Husain's Guide to Debugging AI Products & Writing Evalswith Claire Vo
Hamel walks from real user traces through open-coded notes, error categories, counts, targeted binary evals, human-labeled judge calibration, and experiments on prompts, retrieval, and agent transitions.
Before you start
What you need
- Privacy-reviewed traces with user input, model output, retrieval, and tool events
- A sampling method and annotation workspace
- Domain experts who can identify correct and harmful behavior
- Stable versions of prompts, models, tools, retrieval, and test data
- An experiment log for proposed changes and regression results
What you’ll make
A prioritized error inventory, a human-labeled evaluation set, calibrated issue-specific evaluators, and evidence showing how a candidate change affects the targeted error and existing behavior.
Tools used
- ChatGPTVisitChatGPT
OpenAI conversational AI
Step by step
The workflow
Follow the sequence once, then adapt the prompts, checks, and handoffs to your own setup.
6 steps
Step02
Perform Manual Error Analysis
Draw a stratified random sample and read the traces with human eyes. Add a short note about the earliest meaningful error, its evidence, and user impact. Do not jump directly to a global score.
Example prompt
For each trace, identify the earliest actionable error. Return trace ID, quoted evidence, expected behavior, observed behavior, user impact, and a one-sentence open-code note. If the trace is acceptable or uncertain, say so.
Focusing on the most upstream error is a powerful heuristic. Fixing early intent clarification or tool call issues often resolves many downstream problems.
Step03
Create a Custom Annotation System
Use a low-friction annotation view that preserves the trace and its metadata. Define who may label which domain, record uncertainty, and retain disagreements for calibration instead of forcing consensus.
Step04
Categorize and Prioritize Errors
Cluster the open-coded notes into a taxonomy, then have domain experts merge, split, and define the categories. Count errors by important segment and prioritize by frequency, severity, and strategic importance.
Example prompt
Cluster these notes into candidate error categories. For each category, propose a definition, inclusion and exclusion rules, representative trace IDs, count, affected segments, and likely system layer. Preserve an other category and do not erase ambiguous notes.
Step05
Write Targeted Evaluations (Evals)
For each priority error, write a binary evaluator grounded in the human labels. Measure its agreement, precision, recall, and failure examples on held-out data before applying it at scale.
Example prompt
Create a binary evaluator for [specific error]. Define pass and fail behavior with examples from the labeled set. Return the decision plus cited evidence. Do not score unrelated qualities. Compare its predictions with held-out human labels and list every disagreement.
Step06
Iterate and Improve
Use the evidence to choose the likely system layer: prompt, context, retrieval, tool, model, or product flow. Change one versioned component, run the targeted and regression sets, inspect failures, and record the result before rollout.
Example prompt
Evaluate candidate version [version] against baseline [version]. Report target-error results, regression-suite results, changed trace IDs, new failure categories, latency, and cost. Do not summarize away individual regressions with an average score.
What good looks like
- Sampling represents important traffic segments and does not expose unnecessary personal data.
- Notes identify the earliest actionable error and retain the trace evidence behind each label.
- Each automated evaluator tests one concrete problem and is compared with held-out human labels.
- A change ships only with measured improvement on the target set and no unacceptable regression on protected behavior.
Build your next product with ChatPRD
Turn an idea into a PRD, user stories, and a plan.
After the steps
Runbook notes
How to recover when the loop fails and where human judgment helps.
Recover
If it goes sideways
- Raw traces expose personal or customer-confidential information to unnecessary people or tools
- Minimize and redact fields, enforce role-based access and retention, and use approved synthetic traces when real data is not permitted.
- The sample overrepresents recent, easy, or high-volume traffic and misses costly failures
- Stratify by channel, user segment, task, language, outcome, and risk, then add random samples and known incidents.
- Labels overlap, drift, or describe symptoms too vaguely to guide a fix
- Define each category with inclusion, exclusion, and examples, allow an other bucket, and reconcile annotator disagreements.
- A dashboard score looks scientific even though the judge disagrees with experts
- Use binary issue-specific criteria, evaluate precision and recall on held-out human labels, and keep uncalibrated metrics out of decision reporting.
- A prompt fix improves the showcased trace but breaks another task
- Version the full system, run the targeted set plus regression suites, inspect changed failures, and compare against a fixed baseline.
Keep building
Related workflows
advancedPersonalClaude
Create a Meta-Workflow to Continuously Improve Your AI Assistant's Performance
Design a weekly meta-workflow that analyzes your interactions with your AI assistant to make it smarter. This system learns your writing style from edits, suggests new automations for repetitive tasks, and helps you filter valuable AI techniques from hype.
Open
intermediateProductClaudeGranola
Build a Self-Improving AI Morning Brief to Capture Action Items and Learn Company Jargon
Create a daily AI workflow that summarizes yesterday's meetings, identifies urgent tasks, and proactively learns internal company terms it doesn't understand. This keeps your AI assistant's knowledge current and highly relevant to your work.
Open
intermediateProductClaudeNotion+1
Automate Your Weekly Planning with an AI-Powered PM Assistant
Build a recurring AI task that automatically gathers updates from your calendar, Slack, and notes to suggest weekly priorities and prepare you for upcoming meetings. Start every Monday with a clear, comprehensive plan without the manual effort.
Open
Start shipping
better products.
Join 100,000+ product managers who use ChatPRD to write better docs, align teams faster, and build products users love.
Free to start
No credit card
SOC 2 certified
Enterprise ready