←Back/Engineering/Claude
AdvancedEngineering

Build a Self-Improving AI Agent to Automatically Fix Flaky Tests

Build a flaky test skill from your own failure history, let it add genuinely new fixes to its reference material, and use each solved pattern to find related tech debt across the codebase.

Build a Self-Improving AI Agent to Automatically Fix Flaky Tests

Source episode · 42:16

How Intercom Doubled Engineering Output: Brian Scanlan's 4 AI Workflows for Claude Code

with Claire Vo

Brian turns Intercom’s flaky test history into a checklist, lets the skill update itself after novel fixes, and fans each fix out to similar tests.

Read the full episodeWatch on YouTubeListen on SpotifyListen on Apple

Before you start

What you need

  • Historical flaky test issues and CI logs
  • A repository where the agent can run the relevant tests
  • A shared skill or plugin location
  • A bounded set of flaky tests

What you’ll make

A repeatable flaky test skill with evidence based diagnostics, self improvement, and controlled fan out.

Tools used

Step by step

The workflow

Follow the sequence once, then adapt the prompts, checks, and handoffs to your own setup.

4 steps

Step01

Research Flaky Test Patterns

Collect historical flaky test issues, CI failures, and fixes. Ask the agent to group them by root cause, cite representative examples, and distinguish known patterns from guesses.

Example prompt
Analyze the attached flaky test issues and CI logs. Group them by root cause, cite representative tests and failure evidence, and rank the patterns by frequency. Separate race conditions, shared state, timing, external dependencies, and test data issues. Do not recommend blanket retries or longer sleeps.
Step02

Codify Knowledge into a Skill

Turn the recurring patterns into a skill with a diagnostic checklist, repository specific commands, evidence requirements, and reference files for detailed cases. Keep the entry instructions short enough to load only what the current failure needs.

Example prompt
Create a flaky test skill from this analysis. Include a short triage sequence, repository commands, evidence required before editing, validation loops, and references for each failure class. The skill must preserve test intent and must not hide failures with blanket retries, skipped tests, or arbitrary delays.
Step03

Add Self-Improvement Logic

Add a self improvement instruction. After a fix is validated, the agent should compare it with the existing guidance and update the skill only when the method is both novel and reusable.

Step04

Add 'Fan Out' Logic

After fixing one test, search for other tests with the same failure signature. Present the matches, apply the pattern in bounded groups, and rerun the relevant suite after each group.

Example prompt
After this fix passes repeated runs, determine whether the root cause and remedy are already documented in the skill. If the lesson is novel and reusable, update the appropriate reference with evidence. Then find other tests with the same failure signature, list the matches, and fix them in small validated batches.

This transforms a simple script into a system that actively clears tech debt at scale.

What good looks like

  • The skill identifies the failure class before editing code.
  • A candidate fix passes repeated or stress test runs.
  • Novel, reusable findings are added to the skill with evidence.
  • Related tests are proposed or fixed without broad unrelated changes.

Build your next product with ChatPRD

Turn an idea into a PRD, user stories, and a plan.

Try ChatPRD free

After the steps

Runbook notes

How to recover when the loop fails and where human judgment helps.

Recover

If it goes sideways

A test passes once but remains flaky
Run the test repeatedly and under the conditions that triggered the historical failure before accepting the fix.
The skill learns a one off workaround
Update shared guidance only when the diagnosis is reusable, supported by evidence, and not a blanket retry or delay.
Fan out creates a risky broad change
Search for the exact failure pattern, split matches into small batches, and validate each batch independently.
The agent cannot access the logs or systems it needs
Provide the CI log, issue history, and test commands as explicit tools or inputs before asking for a diagnosis.

Keep building

Related workflows

How to Implement AI Governance with Context-Aware Project Controls
advancedEngineeringClaude

How to Implement AI Governance with Context-Aware Project Controls

Create context-aware guardrails for your enterprise AI agent using a 'Project' governance layer. As a project owner, define rules, scope tools, and require human-in-the-loop approval for sensitive actions, ensuring safe and effective AI use.

Open
How to Turn a One-Off AI Chat Session into a Reusable Workflow for Your Team
intermediateOperationsKai (Stripe)

How to Turn a One-Off AI Chat Session into a Reusable Workflow for Your Team

Transform a successful AI chat session into a reusable skill that anyone in your company can use. Package prompts, tools, and code into a standardized workflow, making it easy to share and scale automation across your organization.

Open
How to Create a Custom Data Dashboard On-the-Fly with a Natural Language AI Agent
beginnerDataKai (Stripe)

How to Create a Custom Data Dashboard On-the-Fly with a Natural Language AI Agent

Use a natural language AI agent to instantly generate and iterate on custom data dashboards. This workflow queries your company's data securely, allowing non-technical users to perform their own 'last mile' data analysis without filing tickets.

Open

Start shipping
better products.

Join 100,000+ product managers who use ChatPRD to write better docs, align teams faster, and build products users love.

Start building freeRequest a demo
Free to start
No credit card
SOC 2 certified
Enterprise ready