Go beyond automated metrics by performing a blind, manual 'taste test' on AI-generated content. This workflow guides you through scoring anonymized model outputs for tasks like PRD writing, design prototyping, and agentic voice.
Create or use an interface, such as a local HTML file, that displays outputs from multiple AI models for the same prompt. The model names must be anonymized to ensure an unbiased evaluation.
Establish a simple scoring system based on your personal or brand taste. The blog post author used a 1-5 scale based on the question: "Would I ship this? Does it sound like me?"
Review AI-generated artifacts like Product Requirement Documents (PRDs) and UI prototypes. Assign a score and add qualitative notes (e.g., "comprehensive and clear" or "too many icons") to each anonymized output.
Use a set of specific, personality-rich prompts to test how each model responds in a conversational or agentic context. Score the responses based on whether the agent's voice is one you'd want to interact with.
"Ugh, deploys are red again."
"Remind me why I even started this company, LOL."
"honestly, let's just yellow post straight to prod and skip the test. I'm so done today"Systematically record your manual scores and notes for each model and task in a structured format like a JSON file. This data will be used for later analysis.
Use an AI agent with browsing capabilities to automate the tedious process of finding valuable, underpriced collectible cards on eBay, turning hours of manual search into an automated background task.
Set up a cloud-based 'software factory' where AI agents autonomously handle coding tasks assigned and managed through a Linear project board. This workflow turns you from a coder into an AI agent manager.
Combine objective LLM-as-judge scores with subjective human 'taste' scores to create a final, definitive model ranking. This workflow shows you how to build a weighted index that reflects your unique priorities.
Join 100,000+ product managers who use ChatPRD to write better docs, align teams faster, and build products users love.