Evaluation needs a denominator

A success rate looks precise because it is a number. Its meaning still depends on what counted as an attempt, what counted as success, and which attempts entered the calculation.

Let $s_i$ be 1 when attempt $i$ satisfies a stated success condition, and 0 otherwise. For $n$ attempts, the observed rate is

$$\hat{p} = \frac{1}{n}\sum_{i=1}^{n} s_i.$$

The expression is simple. The decisions behind it are not contained in the expression.

A small example

Suppose an illustrative test set contains ten tasks. Eight produce the expected answer, one times out, and one produces an answer that violates a required constraint. Counting the timeout as an attempt gives eight successes out of ten. Removing it gives eight out of nine.

Neither fraction tells the reader how the tasks were chosen. Neither establishes how the system will behave on an unfamiliar task.

Keep the question close to the measurement

A useful evaluation record can state:

• The task and permitted tools.
• The success condition, including constraints.
• The model and system configuration.
• The treatment of timeouts, retries, and partial results.
• The individual outcomes, not only their average.

These fields make the number interpretable. They also make disagreements easier to locate: a disagreement about success is different from a disagreement about which tasks are representative.

An open question

How should an evaluation reflect a rare but consequential failure? An average over a convenient task set does not settle that question. It needs an explicit choice about what the system is meant to do and which failures matter.

This is an initial explanatory note with illustrative numbers, not a benchmark of a deployed agent.