How Mozilla Fixed 500 Security Bugs with Claude Mythos

In this episode of How I AI, Mozilla Distinguished Engineer Brian Grinstead explains the agent pipeline behind a sharp increase in Firefox security fixes. The public chart showed nearly 500 fixes in one month, but Brian’s account makes clear that the result came from a model working inside Mozilla’s tooling, verification, and human review process.

The system does more than produce plausible bug reports. It gives an agent a Firefox checkout, lets it generate and run test cases, and requires a reproducible crash before a finding moves forward. A separate verifier checks the evidence before a patching agent attempts a fix.

Brian breaks the work into three connected jobs: prioritize promising files, search for a reproducible vulnerability, and verify both the report and proposed patch. The useful lesson is the structure around the model, not a claim that one model independently fixed every bug in the chart.

Workflow 1: Build a harness for verified bug hunting

Mozilla had already seen low-quality AI security reports from people pasting code into chatbots. Maintainers paid the cost of investigating those false positives. The team’s own system was designed around proof: a report is useful only when the agent can reproduce the failure.

The harness gives the model a terminal, a Firefox checkout, build commands, and purpose-built evaluators. That moves the work from discussing code to forming a hypothesis, creating a test, and observing whether the instrumented browser actually crashes.

The bug-hunting loop

A run follows this sequence:

1. 1. Select a target: A separate scoring workflow ranks source files so the expensive search begins in a promising place.
2. 2. Start the main agent: The Claude Agent SDK coordinates an agent with a Firefox checkout and a narrowly defined security task.
3. 3. Focus the search: The prompt tells the agent that a security bug exists in the target file and asks it to find the evidence.
4. 4. Generate a test: The agent reasons about how web content might reach the vulnerable code and writes HTML cases intended to trigger a crash.
5. 5. Run the evaluator: A browser evaluator opens each case in a Firefox build with AddressSanitizer, which can expose memory-safety failures.
6. 6. Iterate: The result feeds back into the next attempt. In Brian’s legend-element example, the agent reached a reproducing case on its fourteenth try.
7. 7. Save the proof: A successful run produces the exact HTML case that triggers the crash, giving Mozilla a reproducible artifact rather than an unsupported claim.

The model supplies hypotheses and persistence; the harness and existing security tools supply the ground truth. That distinction is what makes repeated agent attempts useful instead of noisy.

Workflow 2: Rank the search space with an LLM judge

Firefox contains far too much code to search every file with the same effort. Mozilla uses a scoring workflow to direct compute toward files that appear more likely to contain an exploitable memory-safety problem.

This prioritization does not prove that a file is vulnerable. It produces an ordered queue for the more expensive harness to investigate.

How the file scoring works

The scoring pass is deliberately simple:

1. 1. Supply context: The prompt explains relevant Firefox file types and incorporates information from Mozilla’s existing security classification work.
2. 2. Request two scores: Each file receives separate estimates for memory-safety risk and accessibility from a malicious webpage.

• Likelihood: How likely is the file to contain a memory-safety problem?
• Accessibility: How directly can webpage content reach the code?

1. 3. Rank the files: Large, web-accessible files such as document.cpp rise toward the top of the queue.
2. 4. Add operating signals: The team can combine the judge score with run history, duplicate findings, and other evidence before assigning compute.

Brian summarized the core scoring prompt this way:

You're a security expert. Here's the different kinds of files we're looking at: C++ files, IPDL files, Web IDL files. Here is some detail about each... Now, give me two scores. One score is how likely do you think there's a memory safety issue? And another is how easy could you access this from a webpage?

The same pattern can help prioritize other bounded engineering searches, but the scoring criteria need a real downstream test. Brian mentioned commit scanning and performance work as adjacent examples.

Workflow 3: Verify findings and proposed patches

A reproducing crash starts the engineering process; it does not finish it. Mozilla runs the finding through independent verification, patch generation, automated retesting, and human review.

That separation matters because a goal-seeking agent can satisfy a poorly specified target in the wrong way. Brian described cases where an agent might alter the source or rely on a developer-only preference to manufacture apparent success.

The verification and patching pipeline

1. Verifier agent: A second agent checks the exploit and looks for shortcuts, including source changes or unrealistic configuration. This filters false positives before patching.
2. Patching agent: After verification, another agent proposes a code change for the confirmed issue.
3. Automated retest: The harness applies the patch, rebuilds Firefox, and reruns the same HTML case. A missing crash is evidence that the patch addressed the reproduction, not proof that every related issue is solved.
4. Human review: The patch and evidence enter Mozilla’s normal bug process. In Brian’s example, the agent produced a valid local fix, while an engineer recognized the same pattern in three other places and broadened the solution.

The handoff preserves the strengths of both sides. The agents perform the repeated search, reproduction, and first patch attempt. Engineers decide whether the fix is architecturally complete and safe to ship.

What made the pipeline work

The three jobs reinforce one another: scoring narrows the search, the harness demands a reproducer, and the patch loop reruns the evidence. Mozilla’s prior investment in builds, sanitizers, fuzzing, and CI gave the agents reliable tools to call.

Mythos was part of the story, but Brian would not reduce the outcome to the model alone. The orchestration determines what the agent may do, what counts as proof, and when another agent or a human must review the result.

The portable pattern is a scored queue, a narrow agent objective, executable verification, and an expert review step. It can be adapted to other measurable engineering problems without pretending that every large codebase should copy Mozilla’s security workflow unchanged.