How I Built a Custom AI Harness with the Claude Agent SDK for Bug Triage

Most AI coding workflows still begin with a human typing the same instructions over and over: investigate this Sentry issue, gather the evidence, check the logs, summarize the likely cause, open the follow-up work. In this episode of How I AI, I turn that repeated engineering routine into a custom harness that produces a structured triage brief instead of a messy chat transcript. The project makes "it is the harness, not the model" concrete.

A harness is just code around an AI agent that makes it better at a specific job. The harness defines the context, the tools, the permissions, the workflow, and the expected outputs. Products like Cursor, Claude Code, and Codex are broad harnesses for software work in general. This one is intentionally narrow and opinionated.

ChatPRD gave me a good test case because bug triage sits in the middle of deterministic process and real judgment. Every investigation starts the same way with a Sentry event, then moves through logs, code paths, related systems, and recommendations for the engineering team. But the actual diagnosis still depends on interpretation, missing evidence, and tradeoffs. That combination makes it a strong candidate for a custom agent workflow instead of a rigid script. You can follow the full implementation in How to Build a Custom AI Harness for Automated Sentry Bug Triage. See How to Build a Custom AI Harness for Automated Sentry Bug Triage.

The final result is a terminal application built on the Claude Agent SDK with restricted adapters for Sentry, Linear, GitHub, and Vercel, plus an artifact folder that preserves the evidence and outputs from every run.

Building a Sentry triage harness

A general coding agent can investigate the same error, but I still have to explain the job every time: what systems to inspect, what permissions it has, what counts as evidence, and what format I want back. In the custom harness, all of that operating context is already encoded. I paste in a Sentry link, choose an investigation mode, and the run starts with the correct tools, constraints, and output requirements.

The architecture

I divided the harness into four practical layers:

• Terminal interface: An Ink-based terminal UI provides an interactive workflow without building a full web application. The interface tracks previous runs, streams investigation activity live, and makes the whole process feel more operational than conversational.
• Harness core: The Claude Agent SDK handles the agent loop, tool use, planning, and file access. Each investigation spins up its own run with task-specific inputs and permissions.
• Opinionated adapters: Small modules expose only the Sentry, Linear, GitHub, and Vercel operations needed for bug triage. Instead of giving the model unrestricted API access, the adapters narrow the workflow to the exact evidence and actions the investigation expects.
• Artifact store: Every run writes logs, evidence, reports, and summaries into its own folder. That creates an inspectable record of what the agent saw, what hypotheses it formed, and why it recommended a particular next step.

Using agents to build the harness

I used parallel sessions with Codex and Claude Opus to scaffold the project itself. The brief was straightforward: use the Claude Agent SDK, investigate Sentry issues, gather evidence from internal systems, and enforce a repeatable engineering workflow.

The first attempts were surprisingly resistant to the idea of an actual agent loop. Both models kept steering toward deterministic scripts with fixed execution paths instead of an investigative workflow. I had to repeatedly clarify that the whole point was to let the agent gather evidence, reason through ambiguity, and produce recommendations. Codex ultimately generated the stronger scaffold and used the Claude Agent SDK for the implementation.

The resulting project stayed intentionally small. Its main pieces are:

• index.js: The entry point for both the terminal UI and command-line runner.
• adapters/: Restricted integrations for Sentry, Linear, GitHub, Vercel, and related systems.
• workflows/BugHunter.js: The core investigation flow that gathers evidence, analyzes likely causes, and assembles the final report.
• ui/: Ink components that render the terminal experience and stream investigation progress.

Encoding the operating instructions

A custom harness replaces a generic system prompt with instructions written for one specific workflow. Mine tells the Claude Sonnet 4.6 model which artifacts count as authoritative, what permissions it has, which tools it may call, and exactly what the final output should include.

Example harness instruction: You are working inside the ChatPRD engineering harness. Use the saved artifacts as the source of truth. Investigate this specific problem, follow the plan below, and return the requested evidence, diagnosis, and recommended next action.

Those instructions are built into the harness itself, so every run starts from the same operational baseline. I do not need to re-explain the policy, restate the investigation process, or hope the model remembers context from an earlier conversation.

Running an investigation

The terminal interface asks for a Sentry error link and an operating mode. For the demo, I pasted in a Sentry issue involving ChatPRD edit operations being dropped by agents and selected Investigate mode.

That distinction matters because the harness enforces permissions in code instead of relying on prompt discipline. Investigate mode can read files, inspect logs, and analyze evidence, but it cannot modify source code or apply fixes. If I wanted patch behavior, I would have to explicitly run the harness with different permissions.

Once the run started, the agent gathered evidence from the connected systems, formed root cause hypotheses, and streamed its reasoning and tool activity directly into the terminal UI as the investigation progressed.

The investigation artifacts

When the run completed, the output was not a disposable chat log. The harness generated a structured artifact bundle that another engineer could inspect, verify, and continue from later. The centerpiece was an investigation brief summarizing the findings and unresolved questions.

For this Sentry issue, the brief separated the investigation into four categories:

• Confirmed evidence: The warning was real, had affected roughly 150 users, and was still occurring hourly. The run also documented that Vercel logs were unavailable, which limited the investigation.
• Likely causes: The harness proposed two possible root causes involving invalid or overlapping ranges in a specific function and highlighted a blind spot in the surrounding logic.
• Verification: The report explained how to validate the hypothesis by fetching and inspecting a raw Sentry event.
• Next action: The harness recommended opening a Linear issue for follow-up work, but it explicitly declined to recommend an automatic patch because the evidence was still incomplete.

That separation between confirmed facts, hypotheses, missing evidence, and next steps is what made the output genuinely useful. The harness did not pretend to know more than it did. It created a concrete verification path for the next engineer instead of producing a confident but ungrounded answer. The run also saved an HTML summary alongside the rest of the artifacts for later review.

When a custom harness is worth building

A purpose-built harness makes sense when the same workflow keeps returning with the same context, tools, permissions, and outputs. The constraint is the feature. By narrowing the job, you reduce setup work, improve consistency, and make each run easier to audit.

The pattern here is fairly repeatable: identify a workflow that combines process with judgment, expose only the tools the job actually needs, define the evidence and artifact formats up front, and create a simple interface for launching and reviewing runs.

Bug triage worked well because the process repeats constantly, but the conclusions are never fully predetermined. That is the sweet spot for a custom harness. The most valuable part is not the terminal UI or even the agent itself. It is the operational structure around the model: constrained permissions, opinionated adapters, preserved artifacts, and explicit next-step recommendations. That approach is worth copying anywhere teams repeatedly investigate the same class of problem and need outputs another human can trust, review, and continue from.