Back/Engineering/Claude
AdvancedEngineeringClaude

Build a Self-Improving AI Agent to Automatically Fix Flaky Tests

Go beyond simple scripts by building a sophisticated AI agent that learns from every fix. This workflow shows how to create an agent that not only resolves a flaky test but also updates its own logic and proactively clears similar issues from your tech debt backlog.

Build a Self-Improving AI Agent to Automatically Fix Flaky Tests

Tools Used

Claude

Anthropic AI assistant

02Step-by-Step Guide
1

Research Flaky Test Patterns

Instruct your AI agent to research your team's entire history of flaky specs from your issue tracker to understand and identify common failure patterns.

2

Codify Knowledge into a Skill

Use the research findings to build a detailed, step-by-step debugging checklist inside a custom AI skill (e.g., a flaky_specs skill in Claude Code).

3

Add Self-Improvement Logic

Add an instruction to the skill's core prompt that directs the agent to edit its own skill file to incorporate any new, novel solutions it discovers while fixing a test.

4

Add 'Fan Out' Logic

Add a second instruction to the prompt that tells the agent to find and fix all other similar flaky tests in the codebase after successfully resolving the first one.

Prompt:
Add these two instructions to your skill's prompt: 1. 'when you fix something and it's novel, you need to update yourself as well.' 2. 'find every flaky speck that got impacted by that nature of it.'
Pro Tip: This transforms a simple script into a system that actively clears tech debt at scale.

Start shipping
better products.

Join 100,000+ product managers who use ChatPRD to write better docs, align teams faster, and build products users love.

Free to start
No credit card
SOC 2 certified
Enterprise ready