Compare benchmark tests

You can evaluate and compare the performance of two different runs of the same benchmark test. This helps identify which configuration produced the best answers, track improvements or regressions, and make data-driven decisions about your AI solution's performance.

Procedure

Navigate to Benchmarking > Tests.
The Tests page displays the test cards for each benchmark test.
Select Compare for the test you want to evaluate.

The <benchmark test name> - Compare results page appears.
On the Result 1 list, select the first run and on the Result 2 list, select the second test run for comparison, then select Compare.
This executes the comparison between selected runs. The side-by-side evaluation of two execution runs for the same benchmark test displays the following:
- The Result 1 and Result 2 status indicators display the execution state of each benchmark test run. Point to the test run status to view a tooltip containing the test description associated with that benchmark execution. The tooltip provides additional context about the benchmark configuration at the time the test was run.
- Summary metrics with high-level comparison statistics for both runs. Average score, success rate, average response time, total prompts, and tokens used.
  - Average score on a scale of 0 to 5 with overall evaluation score for each run. 5 is excellence, 4 is acceptable, 3 is partial, 2 is weak, 1 is Poor, and 0 is incorrect.
  - Success rate calculated across all evaluated prompts with the percentage of responses meeting the success threshold. A response is considered successful if the score is greater than or equal to 4.
  - Average response time that measures average execution time per prompt.
  - Total number of evaluated prompts.
  - Total tokens used for each run.
- Detailed result comparison for both runs. This includes the following:
  - Expected answer that defines expected response generated from the knowledge base or source documents.
  - Expected references that lists the expected source documents/pages supporting the answer.
  - Expected token count that displays the expected token size for the benchmark response.
  - Response answer that displays the actual generated response from the evaluated run.
  - Token count that displays the actual number of tokens used by the generated response. Only tokens associated with prompts executed in both test runs are included in the token comparison.
  - Time taken that displays execution time for the individual prompt.
  - References match status message that indicates whether retrieved references match expected references. (References match: Correct documents/pages retrieved and References do not match: Incorrect or missing references.
  - AI Evaluator comment that provides qualitative evaluation feedback explaining the assigned score.
When a generated response is considered more accurate, complete, or appropriate than the current expected result, select Set as expected result. You can select either run response as the new expected result.

On confirmation, the selected response becomes the new reference answer used for future evaluations, and this is recorded in the audit log.

More info