Prompt A/B Evaluator

Prompt engineering without A/B evaluation is vibes-driven development. Pairwise judging is cheaper than absolute scoring, less biased than single-shot grading, and matches how humans actually compare outputs. By fixing the dataset and the judge model, you isolate the one variable you care about: the prompt change. The skill makes the whole loop a one-call operation instead of custom notebooks every time.

1) Load dataset: [{input, tags?}...]. 2) Run variant A and variant B in parallel against every input (bounded concurrency). 3) For each pair, call the judge LLM with a rubric and the two responses randomly ordered; require JSON {winner: A|B|tie, criteria: {correctness, brevity, style, reasoning}, rationale}. 4) Aggregate: per-criterion win rate, overall win rate, failure examples where A won but B lost and vice versa. 5) Bootstrap-sample the pairs to produce a 95% CI on the win rate. 6) Output is a markdown report + a CSV for further analysis.