A structured, data-driven workflow for improving AI product quality by logging real user traces, manually analyzing errors, creating targeted evaluations (evals), and iterating on prompts or models.
with Claire Vo
Collect and review 'traces'—full, multi-turn conversations your AI has with real users. This reveals how people actually use your product, including typos and ambiguous questions, which is crucial for understanding real-world performance. Use platforms like Braintrust or Arize for this.
Manually review a sample of traces (e.g., 100) and write a one-sentence note for the *very first error* you find in each conversation. This 'open coding' process helps identify the root cause of failures quickly.
Example error note: "Should have asked follow up questions about the question, what's up with four month rent? Because it's unclear user intent."To speed up manual review, build a simple internal app or custom view in your observability platform. The goal is to make it easy for annotators (like product managers) to quickly categorize and label issues.
Group your manual error notes into common themes or categories, either manually or with help from an LLM like ChatGPT. Count the frequency of each category to create a clear, data-driven, and prioritized list of what to fix first.
Based on your prioritized error categories, write specific evals. Use code-based evals for objective checks (like data leaks) and LLM judges for subjective issues. Crucially, validate your LLM judges against human-labeled data to ensure they are accurate and produce simple binary (pass/fail) outcomes.
With reliable evals in place, you can now confidently make changes. Use prompt engineering, fine-tuning on difficult examples, or improving your RAG system to address the identified errors and measure the impact with your evals.
Follow Sahil Lavingia's method for delegating small but impactful maintenance tasks, like fixing an internal Slack report, to the AI agent Devin. This workflow shows how to use simple, clear prompts to fix code and update system prompts without distracting your engineering team.
Learn Gumroad CEO Sahil Lavingia's process for rapidly prototyping a UI with v0, then using the AI agent Devin to autonomously code, implement, and submit the feature for review, turning a multi-day task into a one-hour workflow.
Improve your writing in real-time by using ChatGPT as a first reader. Get instant feedback on structure, clarity, and flow as you write, allowing you to polish your work section by section.
We've made ChatPRD affordable so everyone from engineers to founders to Chief Product Officers can benefit from an AI PM.