←Back/Engineering/Codex
AdvancedEngineering

How to Use AI Coding Agents for Exhaustive Infrastructure Benchmarking

Leverage powerful AI coding agents to perform deep, exhaustive benchmarking on complex infrastructure problems. This workflow automates the process of testing numerous solutions, like different database engines or index types, to find the optimal configuration without manual effort.

How to Use AI Coding Agents for Exhaustive Infrastructure Benchmarking

Source episode · 06:10

How I AI: Ankur Goyal's Playbook for Agent-Driven Benchmarking and AI Evals

with Claire Vo

From 06:10 to 09:02, Ankur Goyal explains how he gives coding agents a benchmark goal and lets them exhaustively test infrastructure options instead of relying on assumptions. Clip range: 06:10 to 09:02.

Read the full episodeWatch on YouTubeListen on SpotifyListen on Apple

Before you start

What you need

  • Benchmark test suite for slow queries
  • Production-like dataset path
  • AWS EC2 instance or comparable high-compute environment
  • Coding agent such as Codex
  • Persistent terminal sessions managed with tmux

What you’ll make

A benchmark report comparing query performance across tested storage formats, execution engines, and indexing strategies.

Tools used

  • Codex

    OpenAI's cloud-based AI software engineering agent that can execute code, run tests, and handle complex multi-file tasks autonomously.

    VisitCodex

Step by step

The workflow

Follow the sequence once, then adapt the prompts, checks, and handoffs to your own setup.

3 steps

Step01

Identify the Problem and Define Success

Identify a specific, measurable engineering problem, like slow database queries on a large dataset. Formulate a clear, open-ended goal for the AI agent. For example, state 'make these queries faster' and provide the relevant tests and success criteria, but do not prescribe a specific solution.

Example prompt
You are an expert database engineer. Your goal is to make our database queries faster.

I have a test suite that benchmarks the performance of our slowest queries. Your task is to find and implement optimizations to improve these benchmarks. You can explore any solution, including changing data formats, indexing strategies, or execution engines.

Constraint: Do not change the core logic of the queries.

Success is measured by the performance improvement in the test suite. Document the changes you make and the benchmark results for each experiment.

Here is the test suite:
[paste test suite code or description]
Step02

Set Up the Agent Environment

Set up a powerful coding agent like [[HIAI:SPAN:c1T9sKzTAqaveAWiMgJjs5:Codex]] in a realistic testing environment. This means giving it access to production-like data and running it on a high-powered remote machine, such as an AWS EC2 instance, to handle the compute load. Use a tool like tmux to manage multiple, persistent agent sessions for long-running tasks.

Step03

Run Exhaustive Experiments

Instruct the agent to explore the entire solution space by running experiments continuously, potentially for hours or days. For a slow query problem, this means tasking the agent to test every available open-source column store format, benchmark every compatible execution engine against those formats, and evaluate different index types like bloom filters to find the optimal combination.

Example prompt
You are a performance engineer. Your task is to run an exhaustive set of benchmarks to find the fastest combination of technologies for our query workload.

1. Test every open-source column store format compatible with our data.
2. For each format, benchmark every compatible execution engine.
3. For the top-performing combinations, experiment with different index types, including bloom filters.

Our dataset is located at [path to data]. Log the setup for each experiment and the resulting performance benchmarks in a structured format. This process may run for several days.

What good looks like

  • Agent logs configuration details for each benchmark experiment
  • Performance results are captured in a structured format
  • Experiments preserve the original query logic
  • Top-performing configurations show measurable latency or throughput improvements

Build your next product with ChatPRD

Turn an idea into a PRD, user stories, and a plan.

Try ChatPRD free

After the steps

Runbook notes

How to recover when the loop fails and where human judgment helps.

Recover

If it goes sideways

Benchmarks use unrealistic sample data and misrepresent production performance
Run tests against production-like datasets with representative scale
Long-running experiments terminate due to session loss
Run agents inside tmux sessions with periodic checkpoint logging
Performance gains come from changing query semantics
Validate output equivalence between optimized and original query results

Keep building

Related workflows

How to Implement AI Governance with Context-Aware Project Controls
advancedEngineeringClaude

How to Implement AI Governance with Context-Aware Project Controls

Create context-aware guardrails for your enterprise AI agent using a 'Project' governance layer. As a project owner, define rules, scope tools, and require human-in-the-loop approval for sensitive actions, ensuring safe and effective AI use.

Open
How to Turn a One-Off AI Chat Session into a Reusable Workflow for Your Team
intermediateOperationsKai (Stripe)

How to Turn a One-Off AI Chat Session into a Reusable Workflow for Your Team

Transform a successful AI chat session into a reusable skill that anyone in your company can use. Package prompts, tools, and code into a standardized workflow, making it easy to share and scale automation across your organization.

Open
How to Create a Custom Data Dashboard On-the-Fly with a Natural Language AI Agent
beginnerDataKai (Stripe)

How to Create a Custom Data Dashboard On-the-Fly with a Natural Language AI Agent

Use a natural language AI agent to instantly generate and iterate on custom data dashboards. This workflow queries your company's data securely, allowing non-technical users to perform their own 'last mile' data analysis without filing tickets.

Open

Start shipping
better products.

Join 100,000+ product managers who use ChatPRD to write better docs, align teams faster, and build products users love.

Start building freeRequest a demo
Free to start
No credit card
SOC 2 certified
Enterprise ready