MTPLX — AI Tool Overview

Most speculative decoding implementations trade statistical correctness for throughput by accepting draft tokens greedily — matching argmaxes instead of running the proper probability-ratio check. MTPLX is a native MTP inference runtime for Apple Silicon that refuses that trade. Per the vendor’s documentation, each speculative cycle drafts K tokens from the target model’s own built-in MTP heads, verifies all K positions in a single batched forward pass, and accepts or rejects each position via Leviathan–Chen rejection sampling with fp32 ratio arithmetic and a (p − q)+ residual correction for rejections. The result is verified max_diff = 0.0 against single-token autoregressive reference output.

The architectural differentiator is that there is no external drafter model. The draft heads are baked into the target checkpoint itself, so you load one model, not two. The vendor reports D4 acceptance rates on Qwen3-27B that exceed vLLM’s MTP-5 CUDA per-position acceptance on the same prompts — 75.61% at depth 4 versus vLLM’s 50.9% at depth 4 — which the docs attribute to the Leviathan–Chen path rather than greedy approximation. Custom Metal kernels handle the verify hot path, and GraphBank-compiled verify shapes cover the small-M matrix multiply cases that dominate the verify cycle.

The runtime fits precisely one deployment context: Apple Silicon Macs running models with native MTP heads. Installation is Homebrew-based; a first-run wizard selects model, serving mode, and surface (browser or terminal). The API layer exposes /v1/chat/completions and /v1/messages endpoints with streaming SSE, making it a drop-in local backend for OpenAI- or Anthropic-compatible clients. It does not run on Linux, Windows, or CUDA hardware — the vendor states Apple Silicon only, and there is no mention of a cross-platform roadmap. The project is at v0.1.0-preview.1, built solo by a single developer under Apache-2.0.