Back/Engineering/Claude/GitHub Copilot
IntermediateEngineeringClaudeGitHub CopilotClaudeCodex

How to Build a Custom AI Harness for Automated Sentry Bug Triage

Create a specialized AI agent to automatically investigate and triage bug reports from Sentry. This harness uses the Claude Agent SDK to gather evidence, identify root causes, and generate actionable engineering reports.

How to Build a Custom AI Harness for Automated Sentry Bug Triage

Tools Used

Claude

Anthropic AI assistant

GitHub Copilot

AI pair programmer

Claude

Anthropic AI assistant

Codex

OpenAI's cloud-based AI software engineering agent that can execute code, run tests, and handle complex multi-file tasks autonomously.

02Step-by-Step Guide
1

Define the Harness Architecture

Before coding, map out the system components. Plan for a frontend (e.g., a Terminal UI with Ink), a core built on the Claude Agent SDK, specific tool adapters for APIs like Sentry and Linear, and a file-based artifact store for saving investigation results.

2

Scaffold the Harness with AI Assistance

Use a large language model like Claude or Codex to help generate the initial code structure. Be specific in your request, detailing the desired SDK and workflow to ensure the AI creates an agentic system, not just a simple script.

Prompt:
Help me build a harness. I want to use the Claude Agent SDK to triage Sentry bugs. Here's the workflow I want it to follow.
Pro Tip: You may need to push the AI model to incorporate agentic, non-deterministic parts, as it might default to suggesting a simpler, deterministic script.
3

Build Opinionated Tool Adapters

Create small, focused modules that connect to your tools (Sentry, Linear, GitHub) in a very specific way. Instead of general API access, build functions that pull only the exact data needed for bug triage, making the agent's job faster and more reliable.

4

Craft a Custom System Prompt

Write a highly specific system prompt for your agent. This prompt should be encoded directly into the harness to set the context, define the task's scope, and clarify the expected output every time the agent runs.

Prompt:
you are working inside the chat purity engineering harness. It's chat, be specific. It's not an open-ended coding system we wanna use. These artifacts as a source of truth. And here's the plan to attack a very specific problem. And what I want you to return is X, Y, and Z.
Pro Tip: A custom prompt is a key advantage of a harness, as it ensures consistent behavior without needing to re-explain context for each run.
5

Implement the User Interface

Build a simple, developer-friendly interface to run the harness. A command-line or terminal UI (TUI) built with a library like Ink (for Node.js) is an effective way to kick off investigations by simply pasting a Sentry link.

Pro Tip: In your UI, consider adding modes like an 'Investigate' mode that restricts the agent's permissions, preventing it from modifying source files.
6

Run an Investigation and Review Artifacts

Execute the harness with a specific bug report link. Once the run is complete, review the generated artifacts in the designated folder. The key output should be a structured 'investigation brief' summarizing the findings.

Pro Tip: The brief should contain confirmed evidence, likely root causes, verification steps, and clear next steps, such as creating a ticket in Linear.

Start shipping
better products.

Join 100,000+ product managers who use ChatPRD to write better docs, align teams faster, and build products users love.

Free to start
No credit card
SOC 2 certified
Enterprise ready