Step 01
Establish Centralized Measurement
Create a dashboard to track key factory metrics like automation percentage, velocity, cost, and 'human interactions per PR.' This metric acts as a proxy for how much effort is required to steer the agent.
AdvancedEngineering
How to Measure and Self-Improve Your AI Software Development Factory
Go beyond simple automation by creating a system that measures its own performance and improves itself. Learn to track metrics, score AI agent runs to find failures, and use observer agents to automatically fix your factory's code.
Try this with your AI assistant
Download the skill for this workflow. Version 1.0.1.
Source episodeHow I AI: Inside Warp's Software Factory, From Slack Kickoff to Self-Improving Code
with Claire Vo
Before you start
Tools used
The workflow
Follow the sequence once, then adapt the prompts, checks, and handoffs to your own setup.
5 steps
Step 02
Score Agent Runs to Find Failures
Implement an LLM-based scoring system where a 'judge' model retroactively analyzes and classifies every agent task against specific, predefined failure modes, such as the creation of redundant tests.
Step 03
Quantify and Isolate Failure Modes
Use the scoring data to get a quantitative view of how often specific errors occur. Once a recurring failure is identified across a significant sample of runs (e.g., 20-25), it's ready to be fixed systematically.
Step 04
Deploy an Observer Agent to Self-Improve
Unleash an 'observer agent' that analyzes the set of failed runs for a specific issue. The agent's goal is to propose a code change to the factory's own agent definitions to prevent that failure from happening in the future.
Step 05
Benchmark Models on Real Data
Use the historical task data to replay runs with different LLM configurations (e.g., Opus vs. Groq vs. Gemini Flash). This generates a Pareto chart of cost vs. quality on your own data, allowing for an evidence-based model routing strategy.
Build your next product with ChatPRD
Turn an idea into a PRD, user stories, and a plan.
Related workflows
intermediateProductCodexClaude
Build a Personal PM Operating System with a Shared Knowledge Repo
Create a repository-based system to provide an AI assistant with durable context about your team, projects, and communication channels. This enables it to perform routine tasks like summarizing daily updates and identifying stakeholders, acting as a personalized chief of staff.
Open
intermediateProductMeta Muse
Create a Personalized Family Newspaper with Meta's Muse
Use Meta's Muse to generate a printable, multi-page family newspaper that includes a weekly schedule, weather forecasts, and activity ideas. This workflow leverages Muse's strength in creating visually appealing, family-oriented artifacts.
Open
advancedProductCursorSlack
Build an Automated Growth Experiment Loop with Multiple AI Agents
Coordinate agents in Slack and Cursor to propose growth experiments for human review. Ezra’s Auto XP system was still being developed at the time of the episode.
Open
Start shipping better products.
Join 250,000+ product managers who use ChatPRD to write better docs, align teams faster, and build products users love.
Free to start · 14-day Pro trial · SOC 2 Type II
