Implement a feedback loop to translate subjective expert feedback into quantitative evaluation criteria. This workflow allows you to scale the 'taste' of key stakeholders, like a lead designer, across your entire AI system, continuously raising the quality bar.
Source episode · 23:06
How I AI: Ankur Goyal's Playbook for Agent-Driven Benchmarking and AI Evalswith Claire Vo
From 23:06 to 33:12, Ankur Goyal demonstrates an evaluation loop that turns expert feedback and designer taste into reusable scoring criteria. Clip range: 23:06 to 33:12.
An expanded evaluation rubric that converts subjective expert feedback into measurable scoring criteria.
Step by step
Follow the sequence once, then adapt the prompts, checks, and handoffs to your own setup.
3 steps
First, define a set of measurable criteria for your AI's output, such as accuracy, concision, or code quality. Use a quantitative evaluation system to test the model against a dataset of inputs until it consistently reaches a high baseline score, for example, 90% or higher.
Once the quantitative scores are high, present a representative set of AI-generated outputs to your team's designated 'taste maker,' such as a lead designer or domain expert. The goal is a final qualitative review or 'vibe check' to catch issues that the numbers miss.
When the expert provides subjective feedback, such as 'The tone is helpful but feels patronizing,' translate that qualitative insight into a new, measurable evaluation criterion. Add this new criterion to your evaluation suite to systematically test for this quality in all future runs.
You are an expert in creating AI evaluation criteria. Your task is to translate subjective human feedback into a new, measurable scoring rubric that another AI can use for evaluation. Here is the context: - Original Task: [Describe the AI's task, e.g., "Answer user questions about our documentation."] - Example AI Output: [Paste the AI-generated output that was reviewed.] - Expert Feedback: [Paste the expert's qualitative feedback, e.g., "The tone is helpful but feels patronizing."] Based on this, generate a detailed scoring rubric from 1 to 5 for the quality of "[describe the quality, e.g., 'professional and helpful tone']". For each score, provide a clear definition and an example of what that score would look like. The rubric should be clear enough for an LLM to apply it consistently.
Turn an idea into a PRD, user stories, and a plan.
After the steps
How to recover when the loop fails and where human judgment helps.
Recover
Keep building
Create context-aware guardrails for your enterprise AI agent using a 'Project' governance layer. As a project owner, define rules, scope tools, and require human-in-the-loop approval for sensitive actions, ensuring safe and effective AI use.
Transform a successful AI chat session into a reusable skill that anyone in your company can use. Package prompts, tools, and code into a standardized workflow, making it easy to share and scale automation across your organization.
Use a natural language AI agent to instantly generate and iterate on custom data dashboards. This workflow queries your company's data securely, allowing non-technical users to perform their own 'last mile' data analysis without filing tickets.
Join 100,000+ product managers who use ChatPRD to write better docs, align teams faster, and build products users love.