---
name: "prompt-ab-evaluator"
description: "Compare two prompts on the same fixed cases using a stated rubric. Use when the user needs: Get per-case scores, failures and a cautious comparison."
license: "MIT"
metadata: {"version": "2026-10-17.1", "category": "research-analysis", "publisher": "AIDiveForge"}
---

# Prompt A/B Evaluator

Get per-case scores, failures and a cautious comparison.

## Requirements

Provide both prompts, cases, expected behavior and access to the model you intend to evaluate.

## Workflow

1. Freeze the cases, model/version, settings and rubric before running either prompt.
2. Run A and B on every case under the same settings; preserve raw outputs and run errors.
3. Score outputs without revealing the prompt label where practical. Prefer deterministic checks for exact requirements.
4. Report paired per-case differences and representative failure examples; do not hide ties or failed runs.
5. Separate observed sample results from a general claim. Suggest new held-out cases based on failures.

## Worked example

Read `examples/input.txt` and `examples/expected.txt` before adapting the workflow.

## Acceptance checks

Use the same cases and rubric; keep sample size and model settings visible; do not claim statistical significance.

## Limits

This recipe does not include API credits, a hidden judge or an automated benchmark runner.

Worked example included. Model-specific performance and vendor compatibility have not been benchmarked.