⚖️

Prompt A/B Evaluator

Research & Analysis · by AIDiveForge · Apr 17, 2026 · Advanced · ✓ 1 verified compat

Compare two prompts on the same fixed cases using a stated rubric.

Get per-case scores, failures and a cautious comparison.

Download skill ZIP See the worked example

Free · 2 KB · SKILL.md, instructions and example files. Provide both prompts, cases, expected behavior and access to the model you intend to evaluate.

A worked example

A small example to try before using your own data.

Input

{
  "rubric": "1 if output is valid JSON with key ok; otherwise 0",
  "A_outputs": [
    "{\"ok\":true}",
    "Done"
  ],
  "B_outputs": [
    "{\"ok\":true}",
    "{\"ok\":false}"
  ]
}

Expected result

{
  "A_scores": [
    1,
    0
  ],
  "B_scores": [
    1,
    1
  ],
  "A_mean": 0.5,
  "B_mean": 1.0,
  "conclusion": "B passed one more of these two cases; this is not evidence of general superiority."
}

What to check: Use the same cases and rubric; keep sample size and model settings visible; do not claim statistical significance.

Limits: This recipe does not include API credits, a hidden judge or an automated benchmark runner.

Worked example included. Model-specific performance and vendor compatibility have not been benchmarked.

Installation options and other formats

Install this skill

An agent skill starts with SKILL.md. The ZIP also includes its examples and any helper scripts. Drop the file into your tool of choice — or pick a different format if you use Cursor, Windsurf, Copilot, or something else.

Format Download SKILL.md

Quick install — paste into your terminal

python3 - <<'ADF_SKILL_INSTALL'
import hashlib, io, os, stat, tempfile, urllib.request, zipfile
from pathlib import Path, PurePosixPath
folder = "prompt-ab-evaluator"
root = Path(os.environ.get('ADF_SKILL_ROOT', str(Path.home() / '.claude' / 'skills'))).expanduser()
destination = root / folder
if destination.exists() or destination.is_symlink():
    raise SystemExit('An installation already exists. Back it up and merge the ZIP deliberately.')
root.mkdir(parents=True, exist_ok=True)
request = urllib.request.Request("https://aidiveforge.com/skill/prompt-ab-evaluator.zip", headers={'User-Agent': 'AIDiveForge-Resource-Installer/1.0 (+https://aidiveforge.com/skills/)'})
with urllib.request.urlopen(request, timeout=30) as response:
    data = response.read(8 * 1024 * 1024 + 1)
if len(data) > 8 * 1024 * 1024 or hashlib.sha256(data).hexdigest() != "7e38e77f843ea992273edf8a145bc428bcac0d2d52ff073d04f5ae29cd77006f":
    raise SystemExit('The download did not match this reviewed bundle. No files were installed.')
with zipfile.ZipFile(io.BytesIO(data)) as archive, tempfile.TemporaryDirectory(prefix='.adf-skill-', dir=root) as temporary:
    members = archive.infolist()
    if len(members) > 100 or sum(m.file_size for m in members) > 16 * 1024 * 1024:
        raise SystemExit('Unexpected archive size')
    for member in members:
        name = PurePosixPath(member.filename)
        if name.is_absolute() or '..' in name.parts or not name.parts or name.parts[0] != folder or '\\' in member.filename or stat.S_ISLNK(member.external_attr >> 16):
            raise SystemExit('Unsafe archive entry; nothing was installed')
        target = Path(temporary).joinpath(*name.parts)
        if member.is_dir():
            target.mkdir(parents=True, exist_ok=True)
        else:
            target.parent.mkdir(parents=True, exist_ok=True)
            with target.open('xb') as stream:
                stream.write(archive.read(member))
    staged = Path(temporary) / folder
    if not (staged / 'SKILL.md').is_file() or destination.exists() or destination.is_symlink():
        raise SystemExit('The skill could not be installed without replacing existing data')
    os.rename(staged, destination)
print('Installed the complete skill folder:', destination)
ADF_SKILL_INSTALL

Claude Code: ~/.claude/skills/prompt-ab-evaluator/SKILL.md. Codex: ~/.codex/skills/prompt-ab-evaluator/SKILL.md. Keep the complete ZIP folder when helper scripts are included.

🧠 Why it works

Use the same cases and rubric; keep sample size and model settings visible; do not claim statistical significance.

⚙️ How it works

Freeze the cases, model/version, settings and rubric before running either prompt.

Run A and B on every case under the same settings; preserve raw outputs and run errors.

Score outputs without revealing the prompt label where practical. Prefer deterministic checks for exact requirements.

Report paired per-case differences and representative failure examples; do not hide ties or failed runs.

Separate observed sample results from a general claim. Suggest new held-out cases based on failures.

Description

Get per-case scores, failures and a cautious comparison.

Provide both prompts, cases, expected behavior and access to the model you intend to evaluate.

Limits: This recipe does not include API credits, a hidden judge or an automated benchmark runner.