GEDD

Generic eval dashboards show agents as rows of metrics — which tells you nothing about whether the agent actually handles the edge cases your domain expert would catch in five minutes. GEDD reframes evaluation as a scenario-first session: a PM loads realistic launch-risk scenarios, a domain expert reviews the agent in the shape of the task it actually performs, names the failure modes in their own language, and the session produces a release report alongside a validated evaluation dataset. The core workflow is a review-and-annotation workbench, not an automated scorer — a human stays in the loop at every judgment call.

The differentiating design choice is what the docs call task-shaped interfaces. Rather than forcing a healthcare reviewer to score a clinical agent through the same generic table used for a code assistant, GEDD allows teams to build evaluation interfaces that match the specific agent type being judged. That surface area change is the reason domain experts can name failure modes that generic metrics miss entirely — the interface speaks the task’s vocabulary, not the eval framework’s.

GEDD fits teams crossing the gap from ad-hoc manual review toward structured, repeatable evaluation — particularly product managers who own release readiness but lack a rubric, and AWS-native teams already running Bedrock or SageMaker. The tool runs locally; there is no vendor-hosted SaaS. It carries an MIT-0 license and is self-hosted from the GitHub source. Where it breaks: GEDD offers no API, so teams that need evaluation results consumed programmatically by a CI/CD pipeline must build that extraction layer themselves. At that point they are maintaining two systems.

Installation depends on three pip packages — sagemaker-mlflow, grounded-evals, and mlflow — documented in the repository’s SETUP.md. The project is an AWS sample, not a managed AWS service, which means support is community-driven through GitHub issues rather than AWS support channels.