Create a repeatable, automated benchmark to evaluate AI models on tasks relevant to your work. This workflow uses Claude Code to brainstorm, refine, and generate a complete evaluation system, including a test runner and scoring backend.
Use a context-aware AI tool like Claude Code to brainstorm the structure of your benchmark. Provide context about your audience and goals to get relevant suggestions.
Based on our work together, can you help me brainstorm how I AI benchmark and eval set we can test every time a new model comes out to consistently score different tasks That would be relevant to our podcast audience.Narrow the focus to a manageable set of specific tasks. Provide clear instructions to the AI to concentrate on what matters most to you, such as tasks for builders, PRDs, and prototypes.
Let's actually focus on task for builders, PRDs prototypes, agentic, multi-step and agentic voice basically does a pass the vibe check in my open claw.Ask the AI to generate the complete evaluation system based on your refined scope. This system should include a test runner script, an LLM-as-judge scorer, and a local HTML page for human evaluation.
Ensure the generated harness includes both an automated scorer (LLM-as-judge) and an interface for manual human scoring (a 'vibe check'). This hybrid approach combines objective, automated scoring with subjective, taste-based feedback.
Use an AI agent with browsing capabilities to automate the tedious process of finding valuable, underpriced collectible cards on eBay, turning hours of manual search into an automated background task.
Set up a cloud-based 'software factory' where AI agents autonomously handle coding tasks assigned and managed through a Linear project board. This workflow turns you from a coder into an AI agent manager.
Combine objective LLM-as-judge scores with subjective human 'taste' scores to create a final, definitive model ranking. This workflow shows you how to build a weighted index that reflects your unique priorities.
Join 100,000+ product managers who use ChatPRD to write better docs, align teams faster, and build products users love.