Combine objective LLM-as-judge scores with subjective human 'taste' scores to create a final, definitive model ranking. This workflow shows you how to build a weighted index that reflects your unique priorities.
Gather your two sets of data: the scores from your automated LLM-as-judge and the scores from your manual 'vibe check'. You should have parallel scores for each model's output on every task.
Compare the automated leaderboard with your manual leaderboard. Identify where they agree and disagree to understand the strengths of each scoring method.
Use an AI assistant to create a tool, such as an interactive HTML page with a slider, that allows you to combine the two score sets into a single weighted index.
Use your tool to decide on the weighting that best reflects your priorities. The blog author chose a 70% weight for their manual score and 30% for the automated score to create their final ranking.
Break down the final weighted scores by individual task (e.g., writing PRDs, prototyping). This allows you to generate specific recommendations for which model is best for which job, as the overall winner may not be the best for every task.
Use an AI agent with browsing capabilities to automate the tedious process of finding valuable, underpriced collectible cards on eBay, turning hours of manual search into an automated background task.
Set up a cloud-based 'software factory' where AI agents autonomously handle coding tasks assigned and managed through a Linear project board. This workflow turns you from a coder into an AI agent manager.
Go beyond automated metrics by performing a blind, manual 'taste test' on AI-generated content. This workflow guides you through scoring anonymized model outputs for tasks like PRD writing, design prototyping, and agentic voice.
Join 100,000+ product managers who use ChatPRD to write better docs, align teams faster, and build products users love.