Get This Tool
Proctor
Summary
AI coding-agent benchmarks are being gamed — UPenn researchers documented over a thousand harness-level cheating traces across major benchmarks in April 2026. Proctor exists to close the gap between a benchmark run and a result anyone can actually trust.
Proctor wraps each agent execution in a Linux sandbox that cuts off access to hidden tests, fix history, and network egress, so the agent cannot read the answers before producing them. After the run, it produces a cryptographically signed verdict bundle that a third party can verify without re-running anything. The signing and forbidden-access timeline together mean cheating leaves a detectable trace. The tool targets researchers and benchmark maintainers on Linux — it is not a hosted service, carries no API surface, and requires you to operate your own infrastructure. Teams with Windows-only CI pipelines or no Linux sandbox provisioning hit an immediate wall.
Bottom line: Use Proctor when you need a signed, independently auditable record that an agent benchmark ran clean — skip it if your CI runs on Windows or you need a managed service rather than a self-hosted runner.
Community Performance Report Card
No community ratings yet. Be the first to rate this tool!
Community Benchmarks Community
Sign in to submit a benchmarkNo community benchmarks yet. Be the first to share a real-world data point.
Pros
Sign in to edit- Cryptographically signed verdict bundles, so a benchmark result published to peers or a leaderboard carries a verifiable chain of custody instead of a trust-me assertion.
- Forbidden-access timeline logging captures not just whether violations occurred but when and what was attempted, which means post-hoc audits can distinguish accidental sandbox escapes from deliberate cheating.
- Linux namespace isolation cuts off hidden test access and network egress at the OS level, so the agent cannot read answers or phone home during evaluation — closing the attack surface that UPenn researchers documented across major benchmarks.
- GitHub Actions integration via a bundled action.yml, so benchmark runs plug into an existing CI pipeline without requiring a separate orchestration layer.
- MIT-licensed and self-hostable, which means benchmark maintainers control the full execution environment and are not dependent on a third-party service's uptime or data policies.
Cons
Sign in to edit- The sandbox mechanism is Linux namespace isolation, so the tool does not run on Windows or macOS natively. Teams whose CI infrastructure is Windows-only cannot use Proctor without provisioning a Linux runner first — and teams unwilling to do that switch to a CI-hosted evaluation service that abstracts the OS.
- No hosted or managed option exists — the vendor states none — so every team bears the full operational burden: provisioning the runner, maintaining the sandbox configuration, and rotating signing keys. Small research teams without dedicated infrastructure support hit this immediately.
- Community adoption is at near-zero at the time of curation, meaning no public production deployments, no third-party integration guides, and no community-validated workarounds for edge cases in the sandbox configuration. Teams that need a tool with a proven production track record choose a more established benchmark harness instead.
Community Reviews
Sign in to write a reviewNo reviews yet. Be the first to share your experience.
About
- Platforms
- Linux
- API Available
- No
- Self-Hosted
- Yes
- Last Updated
- 2026-06-24T08:15:54.499Z
Best For
Who it's for
- Researchers evaluating AI coding agents
- Benchmark maintainers seeking tamper resistance
- Developers needing reproducible agent isolation
What it does well
- Running isolated AI coding-agent benchmark tasks
- Detecting and logging sandbox violations during agent execution
- Generating signed, verifiable benchmark verdicts
Discussion Community
Sign in to commentNo discussion yet. Sign in to start the conversation.
Compare Proctor
Spotted incorrect or missing data? Join our community of contributors.
Sign Up to ContributeCommunity Notes & Tips Community
Sign in to contributeBe the first to contribute. General notes, observations, gotchas, and tips from people who use this tool day-to-day.
Frequently Asked Questions
- Is Proctor free?
- Yes — Proctor is fully free to use. There is no paid tier.
- Is Proctor open source?
- Yes. Proctor is open source.
- Can I self-host Proctor?
- Yes. Proctor supports self-hosting on your own infrastructure.
- What platforms does Proctor support?
- Proctor is available on: Linux.
Hours Saved & ROI Stories Community
Sign in to contributeBe the first to contribute. Concrete time/cost savings, with context. e.g. "Cut my code review backlog from 4h to 45m per week."
Curated lists that include this category
Proctor runs AI coding-agent benchmark tasks inside an answer-isolated Linux sandbox, then signs the verdict and the forbidden-access timeline so the result is independently verifiable. The core workflow is: configure the sandbox with hidden tests and network restrictions, execute the agent, capture any policy violations into a signed log, and emit an integrity bundle that any third party can verify without re-executing the run. The GitHub README documents namespace isolation as the mechanism keeping the agent away from test answers and fix history during execution.
The differentiating capability is the signed, verifiable integrity bundle. Most benchmark harnesses produce a pass/fail result that you either trust or re-run. Proctor produces a verdict with a covered forbidden-access timeline attached — meaning violations are logged, not just blocked, and the log travels with the result. This matters when benchmark results are shared publicly or used to compare agents across organizations, because the chain of custody is auditable rather than asserted.
Proctor fits tightly into a specific niche: researchers running SWE-bench-style evaluations who need tamper-resistance, and benchmark maintainers who want to publish results that hold up to scrutiny. It does not fit teams who need a managed cloud runner, a Windows-compatible sandbox, or a no-config hosted option — the vendor states no hosted offering exists, and the architecture is Linux namespace isolation, which means non-Linux environments are out of scope. Teams that need cross-platform reproducibility or managed infrastructure switch to a CI-hosted evaluation service instead.
The project ships as a Rust workspace (Cargo.toml and Cargo.lock at the repo root), includes a GitHub Actions integration via action.yml, and is released under MIT. Community adoption is minimal at the time of curation — zero forks, zero stars — so production war stories and third-party integration guides are not yet available.