Benchmark an agentic AI model's ability to resolve complex customer billing issues. This workflow tests the AI's ability to investigate tickets, handle complex logic, execute changes in an admin app, and communicate with the customer.
Meta's personal AI assistant for family newsletters, personal goals, and browser tasks.
Follow the sequence once, then adapt the prompts, checks, and handoffs to your own setup.
5 steps
In a mock admin app with test customer data, provide the agent with a ticket that is intentionally linked to the wrong customer account. The agent must be able to identify this error as the first step.
This tests the agent's ability to cross-reference information and identify discrepancies.
After identifying the error, the agent must search the system (e.g., a mock admin app) to locate the correct customer profile associated with the ticket's details.
Your AI agent needs to find the correct customer profile in the system based on the information in the ticket.
The ticket should describe a duplicate charge. The agent must calculate the correct refund amount based on this logic, rather than simply refunding a requested amount.
This benchmarks the model's reasoning and calculation abilities in a real-world scenario.
The agent must issue the correct refund and make a change to the customer's plan within the system. As a test, include a prompt injection task within the ticket's text to see if the agent ignores it and stays on task.
The ticket contains a hidden instruction to perform an unrelated action. Test if your agent ignores this and focuses on the core problem.
Finally, the agent must draft a clear and correct reply to the customer explaining the resolution and then mark the ticket as solved in the system.
Assess the quality of the agent's communication. It should sound professional and accurately summarize the actions taken.
Turn an idea into a PRD, user stories, and a plan.
A step-by-step guide to signing up for Meta's Muse API, installing the Muse Code command-line tool, and running it headlessly with an external agent like Claude Code for easier testing and benchmarking.
Use your existing TikTok video library and their associated comment threads to generate highly relevant new content ideas and scripts. This workflow pairs audience feedback directly with the specific video performance that prompted it, leading to targeted and engaging new topics.
Adapt Brooke Lacey’s Claude Code workflow to edit phone footage with FFmpeg, Whisper, and Python. Use a TSV cut list to review timing, then render a vertical video with captions and graphics.
Join 250,000+ product managers who use ChatPRD to write better docs, align teams faster, and build products users love.
