How Mozilla Fixed 500 Security Bugs with Claude Mythos

Firefox shipped nearly 500 security fixes in a single month, and the viral explanation was simple: Anthropic’s Mythos model found them. In this episode of How I AI, Mozilla Distinguished Engineer Brian Grinstead makes the more interesting argument. The breakthrough was not a magical autonomous model. It was a tightly engineered pipeline that gave an agent real tools, forced it to prove its claims, and routed every result through existing Mozilla review systems. See Use an LLM as a Security Judge to Prioritize Codebase Analysis.

The system does far more than generate polished vulnerability reports. Mozilla gives the agent a Firefox checkout, terminal access, build commands, browser evaluators, and a way to run generated HTML test cases against instrumented builds. A finding only survives if the harness can reproduce an actual crash. From there, a separate verifier agent checks whether the exploit is legitimate before a patching agent attempts a fix.

Brian breaks the pipeline into three connected jobs: rank which files deserve attention, search for a reproducible vulnerability inside those files, and independently verify both the exploit and the proposed patch. That framing matters because the lesson from Mozilla is not that one frontier model suddenly learned how to secure a browser. The useful pattern is the orchestration around the model: constrained goals, executable verification, repeated retries, and human review at the end.

Turn vague AI bug reports into reproducible crashes

Mozilla had already spent months dealing with low-quality AI security reports generated by people pasting Firefox code into chatbots. The reports often looked convincing at first glance, but maintainers paid the cost of investigating false positives. Brian described the problem as asymmetric: generating speculative bug reports was cheap, while validating them consumed real engineering time. The team designed its own system around a stricter standard. A report only mattered if the agent could actually trigger the failure.

The harness gives the model access to the same kinds of tools a security engineer would use: a Firefox checkout, shell commands, browser builds, and evaluators tied into Mozilla’s existing fuzzing infrastructure. Instead of asking a model to reason abstractly about code, the system pushes it into a loop of hypothesis, experiment, and measurement. The model studies a target file, generates HTML meant to exercise the vulnerable path, runs the test against a Firefox build instrumented with AddressSanitizer, and checks whether the browser crashes. You can follow the full implementation in Build an AI Agentic Harness for Automated Security Bug Hunting. See Build an AI Agentic Harness for Automated Security Bug Hunting.

The bug-hunting loop

A typical run looks straightforward on paper, but the power comes from the feedback loop and the willingness to let the agent keep trying long after a human would usually stop.

1. 1. Select a target: A separate scoring workflow ranks Firefox source files so the expensive search starts in code that appears most likely to contain a reachable memory-safety issue.
2. 2. Start the main agent: Mozilla uses the Claude Agent SDK to coordinate an agent running against a Firefox checkout with a tightly scoped security objective.
3. 3. Focus the search: The prompt intentionally narrows the task. Brian said they effectively tell the agent there is a security bug somewhere in the file and that its job is to find evidence for it.
4. 4. Generate a test: The agent traces how webpage content could reach the vulnerable code path, then writes HTML designed to trigger the issue. In practice this can produce surprisingly complicated pages that manipulate DOM elements, attributes, and browser behaviors in odd combinations.
5. 5. Run the evaluator: The generated test runs inside a Firefox build instrumented with AddressSanitizer. Mozilla already had years of fuzzing and security tooling behind this step, which gave the agent a reliable pass-fail signal.
6. 6. Iterate: Failed attempts feed back into the next round. Brian showed one example involving a legend HTML element where the agent failed repeatedly before finally generating a reproducing case on its fourteenth try. The value was not brilliance in one shot. It was relentless iteration against a measurable target.
7. 7. Save the proof: When the harness succeeds, it produces the exact HTML artifact that crashes the browser. Mozilla can hand engineers a working reproducer instead of a speculative writeup.

That distinction changes the economics of AI-assisted security work. The model supplies persistence, pattern matching, and endless retries. The harness and Mozilla’s existing infrastructure provide the ground truth. Without that verification layer, the system would create noise. With it, repeated agent attempts become useful engineering work. See Create an AI-Powered Patch and Verification Loop for Security Bugs.

Use an LLM judge to rank millions of lines of code

Firefox is too large for brute-force analysis. Brian described the practical problem clearly: tens of thousands of files and tens of millions of lines of code make it impossible to point an agent at the whole browser and expect meaningful results. Mozilla first narrows the search space with an LLM-based scoring pass that estimates where vulnerabilities are most likely to exist.

The scoring system does not claim that a file is definitely vulnerable. It produces a prioritized queue so Mozilla can spend compute and engineering attention where the odds appear highest.

How Mozilla scores files before running the expensive loop

The ranking pass is intentionally lightweight. Brian emphasized that this part of the pipeline is much simpler than people expect.

1. 1. Supply context: The prompt explains the types of Firefox files under review, including C++, IPDL, and Web IDL files, along with Mozilla’s existing security classification guidance.
2. 2. Request two scores: The model evaluates each file on two dimensions that matter for browser exploitation.

• Likelihood: How likely is the file to contain a memory-safety problem?
• Accessibility: How directly can malicious webpage content reach the code?

1. 3. Rank the files: Large, heavily exposed files such as document.cpp naturally rise toward the top because they combine complexity with direct web accessibility.
2. 4. Add operating signals: Mozilla layers in practical information such as whether previous runs found duplicates, whether the file has already been heavily explored, and other runtime signals before allocating more compute.

Brian summarized the core idea as asking the model to act like a security reviewer and assign rough risk scores. The output is simple, but it gives the larger pipeline direction.

You're a security expert. Here's the different kinds of files we're looking at: C++ files, IPDL files, Web IDL files. Here is some detail about each... Now, give me two scores. One score is how likely do you think there's a memory safety issue? And another is how easy could you access this from a webpage?

The pattern extends beyond browser security. Brian mentioned using similar ranking logic for commit scanning and performance work, and I pointed out that the same structure could apply to tech debt, conversion optimization, or design quality reviews. The key requirement is a downstream verification step. A score alone is not useful unless another system can test whether the recommendation actually improves something measurable.

Separate verification from patch generation

A reproducing crash is only the beginning of the engineering process. Mozilla routes findings through a second layer of verification, patch generation, rebuilds, automated retesting, and finally human review before anything ships.

That separation exists because agents optimize aggressively for the stated goal. Brian described examples where the agent quietly modified source code to manufacture a vulnerability or enabled unrealistic developer-only settings to force a successful exploit. The verifier stage exists to catch exactly those shortcuts.

How the verifier and patching agents work

1. Verifier agent: A second agent reviews the exploit attempt and checks for suspicious behavior such as source modifications, unrealistic configuration flags, or invalid assumptions. This dramatically reduces false positives before engineers ever see the report.
2. Patching agent: Once a finding passes verification, another agent proposes a fix for the confirmed issue.
3. Automated retest: The harness applies the patch, rebuilds Firefox, and reruns the original HTML reproducer. If the crash disappears, Mozilla gains evidence that the fix addressed the specific issue. Brian was careful to distinguish this from proving the entire class of vulnerabilities is solved.
4. Human review: The patch still enters Mozilla’s standard bug-review process. In one example Brian walked through, the agent proposed a correct local fix, but an experienced engineer recognized the same validation problem in several related locations and expanded the patch into a broader architectural solution.

That handoff is where the current boundary between agents and expert engineers becomes obvious. The agents excel at exhaustive search, reproduction, and producing an initial patch candidate. Human reviewers still decide whether the fix is complete, whether similar patterns exist elsewhere in the codebase, and whether the change is safe enough to ship in a browser used by millions of people.

What actually made the system effective

The three stages reinforce one another. The scoring pass narrows the search space. The harness forces the agent to prove its claims with a reproducible crash. The patch loop reruns the same evidence after every proposed fix. Mozilla’s earlier investments in fuzzing, sanitizers, CI systems, bug triage, and developer tooling gave the agents mature infrastructure to plug into rather than forcing the team to invent an entirely new workflow from scratch.

Brian repeatedly pushed back on the idea that Mythos alone explained the spike in fixes. Better models helped generate stronger hypotheses and more effective test cases, but the orchestration layer determined what counted as success, how retries worked, when verification was required, and where humans stayed in control. He estimated the contribution was closer to a shared credit between model capability and harness design than a pure model breakthrough.

The part worth copying is not Mozilla’s exact browser-security stack. Most teams do not have Firefox-scale infrastructure or a crystal-clear crash signal from tools like AddressSanitizer. The reusable pattern is narrower and more practical: rank a large search space, give the agent one constrained goal, require executable verification, and keep expert review at the end. That works especially well for repetitive engineering tasks where success can be measured concretely. The harder part, and the part that still belongs to humans, is deciding whether a local fix actually solves the broader architectural problem.