Back/Design/Braintrust
IntermediateDesignBraintrust

How to Scale Expert Judgment in AI Systems with a Human Feedback Loop

Implement a feedback loop to translate subjective expert feedback into quantitative evaluation criteria. This workflow allows you to scale the 'taste' of key stakeholders, like a lead designer, across your entire AI system, continuously raising the quality bar.

How to Scale Expert Judgment in AI Systems with a Human Feedback Loop

Tools Used

Braintrust
02Step-by-Step Guide
1

Run Quantitative Evals to Establish a Baseline

First, use a quantitative evaluation system to improve your AI model's performance until it reaches a high baseline score (e.g., 90%) based on your defined technical and functional criteria.

Pro Tip: The goal of this step is to resolve all the clear-cut, quantifiable issues so that you can present a high-quality starting point to your human expert.
2

Conduct a Human 'Vibe Check'

Once the quantitative scores are high, present the AI-generated outputs to your team's designated 'taste maker'—such as a lead designer or domain expert—for a final qualitative review and 'vibe check'.

Pro Tip: This step is for capturing the nuanced, subjective feedback that is difficult to define upfront. It respects the expert's judgment and intuition.
3

Capture and Encode Feedback into New Evals

When the expert provides subjective feedback (e.g., 'The tone is helpful but feels patronizing'), your job is to translate that qualitative insight into a new, measurable evaluation criterion. Add this new criterion to your evaluation suite to systematically test for it in all future runs.

Pro Tip: This process doesn't replace the expert; it scales them. By encoding their taste into the system, you ensure their quality standards are applied consistently, allowing them to focus on even more nuanced issues in the future.

Start shipping
better products.

Join 100,000+ product managers who use ChatPRD to write better docs, align teams faster, and build products users love.

Free to start
No credit card
SOC 2 certified
Enterprise ready