Implement a feedback loop to translate subjective expert feedback into quantitative evaluation criteria. This workflow allows you to scale the 'taste' of key stakeholders, like a lead designer, across your entire AI system, continuously raising the quality bar.
Download the skill for this workflow. Version 1.0.0.
with Claire Vo
From 23:06 to 33:12, Ankur Goyal demonstrates an evaluation loop that turns expert feedback and designer taste into reusable scoring criteria. Clip range: 23:06 to 33:12.
An expanded evaluation rubric that converts subjective expert feedback into measurable scoring criteria.
Follow the sequence once, then adapt the prompts, checks, and handoffs to your own setup.
3 steps
First, define a set of measurable criteria for your AI's output, such as accuracy, concision, or code quality. Use a quantitative evaluation system to test the model against a dataset of inputs until it consistently reaches a high baseline score, for example, 90% or higher.
Once the quantitative scores are high, present a representative set of AI-generated outputs to your team's designated 'taste maker,' such as a lead designer or domain expert. The goal is a final qualitative review or 'vibe check' to catch issues that the numbers miss.
When the expert provides subjective feedback, such as 'The tone is helpful but feels patronizing,' translate that qualitative insight into a new, measurable evaluation criterion. Add this new criterion to your evaluation suite to systematically test for this quality in all future runs.
You are an expert in creating AI evaluation criteria. Your task is to translate subjective human feedback into a new, measurable scoring rubric that another AI can use for evaluation. Here is the context: - Original Task: [Describe the AI's task, e.g., "Answer user questions about our documentation."] - Example AI Output: [Paste the AI-generated output that was reviewed.] - Expert Feedback: [Paste the expert's qualitative feedback, e.g., "The tone is helpful but feels patronizing."] Based on this, generate a detailed scoring rubric from 1 to 5 for the quality of "[describe the quality, e.g., 'professional and helpful tone']". For each score, provide a clear definition and an example of what that score would look like. The rubric should be clear enough for an LLM to apply it consistently.
Turn an idea into a PRD, user stories, and a plan.
How to recover when the loop fails and where human judgment helps.
Connect data from your hobbies (like golf scores or chess games) to an AI agent to get automated analysis, visual dashboards, and actionable coaching insights. Turn raw data into a story that helps you improve.
Turn your unstructured Instagram 'Saved' collection into a fun, personalized shopping catalog. This workflow uses an AI agent to organize your saves and present them in a nostalgic, engaging format like the SkyMall magazine.
Resurrect your old digital content, like a forgotten blog from Blogger or Tumblr, by using an AI agent to find, consolidate, and transform it into a daily personal podcast. Reconnect with your past self in a new, listenable format.
Join 250,000+ product managers who use ChatPRD to write better docs, align teams faster, and build products users love.
