How Intercom Doubled Engineering Output: Brian Scanlan's 4 AI Workflows for Claude Code

Intercom doubled merged pull requests per R&D employee over nine months while tracking for quality regressions and operational issues along the way. In this episode of How I AI, Senior Principal Engineer Brian Scanlan explains the systems behind that increase: telemetry pipelines, enforced AI workflows, internal evals, and a culture that treats agent-first development as the default direction of technical work. A Stanford research group reviewing the company’s data reportedly found signals that code quality improved during the same period.

A big part of the conversation is that Intercom approached AI adoption like a product problem, not a procurement decision. Brian and the team instrument internal workflows, analyze session data, track which skills people actually use, and revise the tooling whenever engineers hit recurring friction. The assumption underneath all of it is simple: most technical work is going to become agent-first, so the organization needs systems that support that shift.

Intercom did more than hand engineers access to Claude Code and hope for the best. The company built hooks, skills, telemetry, evals, and feedback loops around the work itself. Brian walks through several examples: automated pull request descriptions that enforce review standards, dashboards showing which AI workflows are spreading internally, a flaky-test agent that updates its own playbook as it learns, and an experiment in making SaaS onboarding easier for agents to complete end-to-end. You can follow the full implementation in Automate High-Quality Pull Request Descriptions with a Custom AI Skill. See Design an Agent-Friendly CLI to Automate SaaS Product Onboarding. See Automate High-Quality Pull Request Descriptions with a Custom AI Skill.

Turning pull request descriptions into a measurable quality system

Brian starts with a deliberately small task inside Intercom’s massive Ruby on Rails monolith: create a redirect so a lobster emoji URL points to my product, ChatPRD. The point was not the redirect itself. It was to show what a modern Claude Code workflow looks like inside a mature engineering organization with millions of lines of code, existing review standards, and production guardrails already in place.

He gives Claude Code a short natural-language prompt and lets the agent figure out where the redirect belongs in the Rails application.

add a lobster emoji rails redirect to chatprd.ai

Claude generated the redirect and prepared a pull request automatically. That created a problem Intercom had already measured internally: AI-written PR descriptions were getting worse over time. The descriptions summarized code diffs reasonably well, but they often missed the intent behind the change, which is the part reviewers actually need. Brian said the team built an LLM judge around its own standards for a strong PR description, then ran it across months of historical data. The trend line for AI-generated descriptions was clearly degrading.

We created an LLM judge to evaluate what a good pull request description should look like, and then got an LLM judge to go through months and months of data. And yeah, the trend was awful. The trend was going in one direction.

Instead of asking engineers to remember a better process, Intercom embedded the standard directly into the workflow.

How the Create PR skill enforces review quality

1. Create the skill: the Create PR workflow uses the entire Claude Code session, not just the final diff, to generate descriptions focused on reasoning, context, and intent.
2. Add a hook: if Claude tries to open a pull request directly through the GitHub CLI, Intercom intercepts the action before the PR is created.
3. Route everything through the skill: the hook forces the agent back through Create PR so every pull request follows the same review standard automatically.

I make an important observation during this section: software teams already built highly opinionated CI/CD systems to control what happens after code is written. Skills and hooks bring some of that same determinism earlier into the development process itself. Instead of relying on wikis, SOPs, or tribal memory, the workflow becomes part of the tooling. Brian said Intercom’s internal evals showed measurable improvement in PR description quality after the Create PR system was enforced.

Instrumenting AI usage like an internal product

Intercom treats its internal AI stack the way a software company would treat a customer-facing product. The team tracks adoption, studies failure modes, and looks for the bottlenecks preventing people from getting useful results quickly.

Tracking which skills people actually use

Every internal skill sends telemetry to Honeycomb when it runs. Brian explains that any engineer building a workflow can immediately see whether people are using it, when they invoke it, and whether adoption grows over time. The most heavily used skills cluster around practical engineering work: creating pull requests, interacting with internal admin tooling, querying Snowflake logs, and troubleshooting CI systems.

• How it works: A shared API key is deployed to all developer laptops. When a skill is invoked, it fires an event to Honeycomb.
• The benefit: Any engineer who builds a skill can create a dashboard to see who is using it, how often, and what its impact is. This creates a feedback loop for skill creators to improve their work and demonstrate its value.

Analyzing Claude Code sessions for coaching and debugging

Intercom also analyzes anonymized Claude Code session data. The raw session files are uploaded into S3 after identifying details are scrubbed out, then processed through internal tooling that surfaces patterns and personalized feedback.

1. Collect data centrally: Claude Code session files are anonymized before upload so the team can study how people interact with the tools across the organization.
2. Analyze and Surface Insights: They built a simple internal tool that runs on top of this data. It provides engineers with personalized feedback and benchmarks on their AI usage.
3. Provide Actionable Feedback: The tool can tell an engineer they're in the 79th percentile of usage, or it might point out a recurring issue. Brian showed an example where the tool reminded him that he was struggling to get the agent to use their Google integrations correctly, prompting him to fix his local configuration.

The interesting part is what they do with the data afterward. Engineers can see percentile-style benchmarks for their own usage and receive practical feedback about recurring friction in their workflows. Brian showed an example where the system noticed repeated struggles getting Claude to interact properly with Google integrations, which pushed him to improve his local setup instead of repeatedly fighting the same problem. At the organizational level, the telemetry highlights systemic issues. If a workflow consistently frustrates people or causes repeated intervention, the team knows where to focus next.

Shipping a centralized skills platform across the company

Intercom keeps its AI skills inside a centralized GitHub repository and distributes them through internal IT systems rather than relying entirely on Claude Code’s native plugin updates.

We ended up using our internal IT systems to synchronize all of the plugins to the disks of everyone's laptops. So this is a great cheat code and I strongly recommend getting very close with your IT team.

Brian said the default plugin mechanism sometimes behaved inconsistently at scale, especially across hundreds of laptops and different local environments. Intercom solved that operational problem by partnering closely with IT and synchronizing skills directly onto developer machines. The repository contains foundational tools that everyone receives, alongside more specialized workflows for engineering teams. Core skills also go through higher-quality review processes with evals and testing before broad rollout.

Building a flaky-test agent that updates its own playbook

One of the strongest demos in this episode of How I AI is a Claude Code skill for fixing flaky tests. Brian describes it as evolving from a useful assistant into what he calls a "100x agent," mainly because it compounds organizational knowledge over time instead of solving each issue from scratch. See Build a Self-Improving AI Agent to Automatically Fix Flaky Tests.

I connect this to what she calls the "and then…" workflow. Instead of stopping after the first successful task, you repeatedly ask what a strong engineer would do next, then turn those follow-on steps into part of the workflow itself.

How the flaky-spec workflow compounds knowledge

1. Research the historical failures: the agent first analyzed Intercom’s backlog and issue history for flaky specs, looking for recurring categories of problems.
2. Turn those patterns into instructions: Brian converted the research into structured guidance and checklists inside a Claude Code skill.
3. Reach human-level capability first: the initial version performed roughly at the level Brian expected from himself when debugging flaky tests manually.
4. Add self-improvement loops: after the baseline worked, Brian added two additional behaviors to the workflow:

• "when you fix something and it's novel, you need to update yourself as well." The agent literally edits its own skill file within the session to incorporate new learnings.
• Find every related failure automatically: once the agent fixes one flaky spec, it searches for similar cases elsewhere in the codebase and applies the same reasoning broadly.

The most interesting part is that the workflow edits its own skill file when it discovers genuinely new fixes. That means the playbook evolves alongside the codebase instead of staying static. Brian argues this changes the economics of engineering maintenance work completely. Tasks that once accumulated endlessly in backlogs, like flaky tests or infrastructure cleanup, become realistic candidates for continuous reduction rather than permanent debt.

Designing software for agent-first onboarding

The final section shifts from internal engineering workflows to product design itself. Brian argues that if agents are increasingly the users interacting with software, SaaS products need to expose more than polished graphical interfaces. Helpful CLIs, discoverable APIs, MCP servers, clear permissions, and useful help text all become part of the product surface.

To demonstrate the idea, Brian showed an experimental Intercom CLI running against his personal website on Vercel. The prototype allowed Claude Code to attempt an Intercom signup and installation flow automatically. The setup was intentionally incomplete, and email verification remained the main point where the workflow still needed access outside the CLI itself.

How the experimental onboarding flow works

1. Start with a natural-language request: the user asks the agent to install Intercom on the site.
2. Use the CLI as the operational interface: the agent discovers the Intercom CLI and begins the setup process through commands rather than a browser-first workflow.
3. Guide the model at the right moment: when email verification blocks progress, the CLI help text nudges the agent toward the likely next step instead of failing silently.

well, maybe you could check email...

1. Close the loop with scoped permissions: if the user has already granted inbox access, gog, a Google Workspace CLI, can retrieve the verification email and feed the code back into the installation flow.

My point during this section is that agent workflows change where product friction becomes visible. Human onboarding funnels used to expose drop-off clearly through analytics dashboards and page telemetry. Agent-driven flows are messier. The user can simply stop the workflow, switch tools, or ask the model to build around your product entirely. That means companies need to think carefully about discoverability, handoffs, and how much effort an agent must spend to complete useful work.

An operating model for agent-first engineering

Across all four demonstrations, the pattern is consistent. Intercom measures behavior, studies friction, codifies best practices into workflows, and keeps tightening the feedback loop between humans and agents. The company’s reported result is materially higher throughput without a measured decline in quality, alongside improvements in areas like test maintenance and developer experience that organizations often postpone indefinitely.

The transferable idea is the operating model around the tools. Intercom treats AI workflows as systems that need telemetry, evals, rollout strategies, and ownership. Create PR hooks reinforce a strong review culture; the flaky-spec loop attacks maintenance debt; telemetry shows where adoption is actually helping. Humans still own prioritization, architecture, product direction, and the decision about where automation improves outcomes instead of merely generating more output.