How I AI: Inside Warp's Software Factory, From Slack Kickoff to Self-Improving Code

The term “software factory” is everywhere right now. It conjures images of a magical assembly line where ideas go in one end and enterprise value comes out the other. But what does that actually look like in practice? Is it just a buzzword, or is it a real, tangible way that teams are building software today?

In this episode, I was so excited to sit down with Zach Lloyd, CEO of Warp, who pulled back the curtain on exactly how his team operates their own software factory. He calls it an “AI magical workshop,” and I think that’s a perfect description. It’s not about dehumanizing development; it’s about creating a system where magical elves—or in this case, AI agents—craft wonderful things that spark joy for users.

Zach walked me through three incredible workflows. First, we saw the core factory loop: how a simple request in Slack becomes a fully implemented, tested, and QA-verified pull request. Then, we explored the manager's view—the control panel for the factory that measures everything, finds failure modes, and even uses AI to improve itself. Finally, Zach showed me his personal AI toolkit for handling CEO tasks, from redesigning sales decks in Figma to mining sales calls for customer insights.

This episode is packed with actionable insights for anyone looking to go beyond simple coding copilots and build a truly integrated, measurable, and efficient system for software development. Let’s get into how they do it.

Workflow 1: From a Slack Message to a Shipped Feature

What sets a “factory” apart from a standalone coding agent is its ability to manage the entire software development lifecycle, not just the coding part. Zach showed me how at Warp, their factory (named “Wilson”) orchestrates a whole series of steps that reflect their specific product development process.

Step 1: Kick Off Work in Public (Slack)

Instead of creating a ticket and working on it locally, work at Warp starts in a public Slack channel. An engineer, designer, or even an automated system can tag the factory bot, @Wilson, with a request. In the example Zach showed, an engineer wanted to change the look of a feature and provided detailed instructions along with an image.

This public-first approach is key. It makes all work observable, so anyone on the team can see what’s being built, how the task was defined, and even jump in to contribute. It moves development out of private silos and into a collaborative space.

Step 2: Automated Triage and Tracking (Linear & GitHub)

Once Wilson receives the request, it doesn't just start coding. First, it acts like a project manager. It triages the request and automatically creates an issue in their tracking tool, Linear. This ensures every piece of work is documented and lives within their broader project management funnel.

After creating the ticket, the factory begins the implementation and creates a pull request on GitHub. It’s integrated with all the core tools of their development stack.

Step 3: Implementation and QA Verification

This is where the agent does the coding, but the magic doesn't stop there. One of the most important steps in their factory is automated QA. To avoid spending endless hours on manual code review, the agent performs a “computer use verification.”

It actually runs the new feature and records a video of it in action, showing the keystrokes and the final result. This video is attached to the PR, giving the human reviewer immediate, visual evidence that the feature works as intended. The entire flow—from Slack message to Linear ticket to GitHub PR with a QA video—takes just 35 minutes.

This whole process goes beyond a simple coding agent. It’s an entire system designed to reflect and automate their end-to-end product development steps, not just the engineering input/output.

Workflow 2: The Manager's View – Measuring and Optimizing the Factory

This is what Zach calls the “engineering manager’s dream.” Because all work is centralized and tracked, you get a powerful dashboard to measure and improve the entire system. This is where the real leverage of the factory approach comes in.

Step 1: Centralized Measurement and Metrics

Zach showed me their internal dashboard, which tracks key metrics like factory automation percentage, velocity, and cost. One metric that immediately caught my eye was “human interactions per PR.”

This metric counts every time a human has to intervene—re-prompting in Slack, commenting on a Linear ticket, or correcting code in a review. It’s a brilliant proxy for how much effort is required to steer the agent. The goal is to drive this number down over time, indicating a more efficient and autonomous factory. And as I pointed out, the data clearly shows where the bottleneck is: while kickoff-to-PR time is 35 minutes, the time to first human review is 3.5 hours. As Zach says, the humans are a little bit of the bottleneck!

Step 2: Scoring Runs to Find Failure Modes

With all this data, you can start doing some really interesting analysis. The Warp factory includes a “scoring” system where every agent task is recorded and retroactively analyzed. They use an LLM as a judge to evaluate runs on specific dimensions. For example, they noticed agents were often writing redundant tests.

So, they created a scorer to specifically look for this failure mode. A judge model (a medium-sized one to manage costs) reviews agent runs and classifies whether “surplus tests” were created. This gives them a quantitative view of how often a specific type of error occurs, which is the first step to fixing it systematically.

Step 3: Self-Improvement and Model Benchmarking

This is where it gets truly futuristic. Once you've identified a recurring failure mode across a significant sample of runs (say, 20-25), you can unleash an “observer agent.” This agent analyzes the failed runs and automatically proposes a code change to the factory itself to prevent that failure in the future. Because the entire factory is defined as code, other agents can modify its instructions and behavior.

Furthermore, you can use this data to make informed decisions about which models to use. The factory can replay real, historical tasks with different model configurations (e.g., Opus vs. Groq vs. Gemini Flash). This generates a Pareto chart showing the cost vs. quality trade-offs on your own data. You can then create a model routing strategy based on real evidence, not just public benchmarks, to balance performance and spend.

Workflow 3: The CEO's Toolkit – Non-Engineering Tasks with Technical Tools

Even as a technical CEO, a lot of Zach's work isn't about shipping code. He showed me how he uses the same agent-driven approach for his day-to-day responsibilities, running multiple tasks in parallel right from his terminal.

Mini-Workflow A: Redesigning Slides with the Figma MCP

Like many of us, Zach finds modifying designs in Figma to be a challenge. So, instead of fumbling with the UI, he uses a coding agent in Warp that has access to the Figma MCP (Multi-modal Command Palette).

He kicked off a task with a detailed voice prompt, asking the agent to create a more visually appealing version of a boring slide.

```text

I'd like to make a new version of this slide, use the Figma MCP to get the context, duplicate the existing slide rather than making changes directly to it. Let's have it be so that the host box contains the sandbox box... Let's make the context system something that's like, kind of like, you know, cloud around these things that feeds into them... Let's make the launch pad have a sort of like rocket type theme... The overall idea here is to make this slide more visually appealing and semantically show the relation of the boxes to each other.

```

The result? The agent duplicated the slide and began transforming the simple boxes into a diagram with a cloud, a rocket ship, and nested elements—a task that would have been tedious to do manually.

Mini-Workflow B: Analyzing Sales Calls with the Granola MCP

To ensure his product positioning is hitting the mark, Zach uses an agent with the Granola MCP to analyze recent sales calls. He tasked it with generating a list of the most frequently asked questions.

```text

Can you use the Granola MCP to look back over my last four weeks of sales meetings and try to build up a list of the top 10 frequently most asked questions in these meetings as they pertain to Warp software factories? Don't list any specific customer info in the summary. Anonymize it.

```

The agent returned a perfectly summarized, anonymized list of the top customer questions, including things like “buy vs. build,” security concerns, and cost management. This is an incredibly powerful way to get direct, data-backed market feedback to inform a sales deck or an FAQ page.

Mini-Workflow C: Rediscovering Leads with GOG CLI

Finally, Zach showed how he uses an agent for sales outreach. He wanted to find cold leads from the past six months who might be a good fit for the now-updated product.

```text

Can you use the GOG CLI to look for emails and calendar events I've had in the past six months with potential enterprise leads who might be useful for another outreach for Warp Factories? You can learn about Warp Factories at warp.dev/factories. And make me a Google Sheet with the info on them and share the sheet link, but don't print out any specific customer email or name in this thread.

```

This task sends an agent to scour his Google Calendar and Gmail, identify relevant contacts, and compile them into a neat Google Sheet, ready for his next outreach campaign. It's a great example of automating time-consuming research and data entry.

Final Thoughts

What Zach and the team at Warp are building is a clear vision of the future of software development. It’s a system where work happens in public, the entire lifecycle is automated and connected, and everything is measured. This allows for a continuous feedback loop where agents not only build the product but also help optimize the process of building it.

But as Zach wisely noted, no matter how fast your factory is, it still has to solve a user problem. The human element of product judgment, user interviews, and design thinking remains essential for choosing the right problems to solve. AI gives us the speed, but we still need to provide the direction.

I walked away from this conversation buzzing with ideas. It’s a powerful reminder to think bigger—not just about how an agent can complete a single task, but how you can design an entire system that learns, adapts, and improves over time. I encourage you to try applying these principles to your own workflows.

***

A special thanks to our sponsors:

• DX: In a recent study across more than five hundred engineering organizations, DX found that spend on AI tools has grown twenty-eight X over the last year. The share of AI-authored code is climbing, but overall innovation has remained flat. As teams generate code faster, new friction in code review and validation is offsetting those early velocity gains. DX tracks speed, quality, and cost together across the software development life cycle, giving engineering leaders clear visibility into how AI impacts delivery and whether those investments are translating into real value. Download the full report at getdx.com/howiAI.
• Open Art Arena: Every week, new AI models launch, and every one claims to be the best. But best at what? Open Art Arena is built to answer the question that actually matters: which model is best for your specific job? Instead of one overall winner, Open Art Arena ranks models across real creative tasks, from advertising and film to animation, product, graphic design, editing, and lip sync, covering both image and video. Stop guessing which model to use. Explore rankings based on real creative work and find the right model for your project. See the rankings at Open Art Arena.

***