> For the complete documentation index, see [llms.txt](/llms.txt).
> Markdown versions of each page are available by appending .md to any URL.

# Measure and improve a factory

Measure factory activity and costs, evaluate completed conversations, compare agent configurations, and turn failures into follow-up work.

Warp Factories tracks what your factory produces and how well it performs, so you can spot a problem, test a fix, and decide whether to keep it.

| Feature | What it tells you |
| --- | --- |
| Dashboard metrics | How much work the factory produced, and what it cost. |
| Scorers | Whether completed conversations meet criteria you define. |
| Benchmarks | How different configurations perform on the same tasks. |
| Self-improvement | Which repeated failures get investigated and turned into follow-up work. |

## Read dashboard metrics

The control room dashboard shows activity, cost, autonomy, and evaluation results:

| Metric | What it shows |
| --- | --- |
| **Total runs** | All agent runs, with breakdowns by agent type, status, source, model, and more. |
| **PRs opened** | Pull requests created from factory work. |
| **PRs merged** | Pull requests that merged. |
| **Autonomy %** | The share of merged work that shipped without human edits. |
| **Time saved** | An estimate of engineer-hours saved by merged factory PRs. |
| **PR latency** | How long pull requests spend in each stage. |
| **Cost per PR** | An estimate of credits spent per pull request. |
| **Most expensive PRs** | The highest-cost pull requests. |
| **Scorer cards** | Results from your Scorers. |
| **Self-improvement PRs** | The three newest Self-improvement pull requests, regardless of the selected date range. |

**Time saved** and **Cost per PR** are estimates, not billing figures. Time saved converts merged pull requests into engineer-hours, and the per-teammate view divides that estimate by your team size. Cost per PR counts recorded credits and can undercount actual usage.

Caution

**PRs merged**, **PR latency**, and the detail in **Most expensive PRs** require the GitHub App and only cover activity from after you install it. The ratio of **PRs merged** to **PRs opened** isn’t a merge rate; the two are counted differently. Run counts include evaluation, benchmark, and Self-improvement runs.

Use the dashboard to pick which runs to investigate, not to conclude what caused a change. A higher run count with a flat PR count could mean harder tasks, retries, or measurement activity.

## Configure Scorers

A **Scorer** uses an LLM judge to classify completed conversations against criteria you write, such as “did the agent run the tests before opening a PR?” Scorers classify conversations rather than grading them on a numeric scale. Keep each Scorer focused on one question so its failures point to a specific fix.

Configure these fields:

-   **Judge instructions** - The criteria the judge checks for.
-   **Classifications** - The labels the judge can assign, each with a score.
-   **Pass threshold** - The score a conversation needs to pass.
-   **Sample rate** - The portion of completed conversations to evaluate.
-   **Judge model** - The model that acts as the judge.

You can apply a Scorer to all agents or only the ones you choose, and pause it at any time. Shortly after a conversation completes, the judge evaluates it and records a classification, a score, and its reasoning.

| Mode | Use it when | You get |
| --- | --- | --- |
| Manual | You want to score one conversation or test new judge instructions. | An evaluation of the selected conversation. |
| Periodic | You want an ongoing sample to track over time. | A baseline to compare against after a change. |

Changing **Pass threshold** updates how past scores display as pass or fail; the recorded results don’t change.

## Compare configurations with benchmarks

A benchmark compares configurations of a single agent on the same fixed tasks, so you can test a model, harness, or runner change before adopting it. A suite includes:

-   **Agent** - The agent whose configurations you compare.
-   **Tasks** - Fixed prompts with success criteria.
-   **Configurations** - The harness, model, and runner combinations to test.
-   **Scorers** - Your classification Scorers, applied to every trial.
-   **Repetitions** - The number of trials per task and configuration.

You can create a benchmark task from a completed run’s detail pane, and Warp copies the run’s input into the task. Add success criteria before you launch.

Every benchmark also runs **Correctness**, a built-in scorer that marks each trial as pass or fail against the task’s success criteria. Results show pass rates, cost, and quality for each configuration, with per-task detail. Warp doesn’t combine these signals into one score or pick a winner; you weigh the results and decide. Benchmark credit totals don’t include model usage, so the true cost is higher.

## Configure and review Self-improvement

Turn on **Self-improvement** for each Scorer whose failures you want investigated automatically. Self-improvement groups related failures and files follow-up tasks as ordinary agent runs. A follow-up run can propose changes to application code, or to the factory’s prompts, skills, and configuration when that repo is available. Follow-up runs can open pull requests, but nothing is adopted without your review.

The factory-level **Analysis model** setting chooses the model used for this analysis. Factories managed from files in a GitHub repo can’t change this setting in the control room.

Each Self-improvement pull request includes a **Regressions addressed** section that links the failing runs and Scorer results behind it, so you can trace the change back to its evidence.

## Run a practical improvement loop

Change one measurable thing at a time:

1.  **Define a Scorer.** Pick one agent and one failure mode you can observe. Write the judge instructions and classifications, then compare the judge’s results against a few conversations you review yourself.
2.  **Collect a baseline.** Let periodic scoring run until results reflect normal work. Record the Scorer settings, date range, and relevant costs.
3.  **Inspect failures.** Read the judge’s reasoning and the underlying conversations. Look for causes like missing context, unclear instructions, or missing tools. Turn on Self-improvement when the same failure keeps repeating.
4.  **Benchmark a candidate.** Compare configurations of that agent on the same tasks, with enough repetitions to trust the difference.
5.  **Review and adopt.** If the evidence supports the change, make it. Review Self-improvement pull requests with the same standards as human-authored ones.
6.  **Keep monitoring.** Leave the Scorer active and compare new results against your baseline. Pause or revise the Scorer when its criteria no longer match what your team needs.

## Next step

Record an adopted change in [factory definitions as code](./factory-as-code) so your team can review the factory configuration.