Polygraph — prove what actually ran

Arm Create · Cloud AI · judge demo

Verify the Arm64 inference path before trusting the benchmark.

Polygraph separates what was built, what was selected, and what actually ran—then separately tests throughput, latency, memory, and recovery before a cloud candidate is promoted.

Static judge demo · inspectable JSON · no hidden backend or live benchmark

1. WatchReproduce a software-claim mismatch locally.
2. InspectL1 static · L2 selection · L3 dispatch.
3. VerifyRead the receipt and reproduce it.
THE 90-SECOND IDEA

A banner is not proof. A debugger count is.

The smallest reproducible demo builds two binaries from the same C source. Both print using fast path: yes and return the correct answer. One never calls the fast function. Polygraph catches the mismatch instead of trusting the banner or timing alone.

01MISMATCH

The liar

Fast symbol exists. Banner says yes. Debugger hits: 0.

exit 1 · NO_DISPATCH_OBSERVED
02MATCH

The honest build

Same banner. Same answer. Debugger proves the fast function ran.

exit 0 · OTHER_DISPATCHED_ADHOC
03AUTOMATE

Ship the gate

Use the exit-code contract in CI. Mismatch fails. Missing evidence is undetermined—not a false pass.

tools/polygraph check --json

This is a real local fixture, not a mockup. It demonstrates the verifier and exit-code contract; it is not the Arm benchmark. Reproduce it with git clonemake demo; no model download is required.

THE ARM CASE

The same method found a software build gap on an Arm CPU.

On one documented llama.cpp KleidiAI source build, software reported acceleration while the needed matmul kernels were missing. An explicit Arm feature-target build restored the kernels on the tested Cortex-X925/A725 CPU.

Scope: this is a broken-versus-corrected build measurement, not a Polygraph-authored kernel, GPU result, stock-release defect, or universal speedup claim. The repository keeps the raw JSON and build provenance so a judge can inspect the exact boundary.

7B prefill · corrected / broken build
Loading committed receipt…
WHY THREE LAYERS

Find the gap, explain it, prevent it.

L1

Built

Static symbols prove the accelerated kernel was compiled into the binary. They do not prove runtime use.

L2

Selected

Verbose runtime logs show what the dispatcher says it selected. Still not proof of execution.

L3

Executed

A non-halting debugger breakpoint counts real kernel entry points during inference. This proves dispatch for the measured workload—not speed or readiness.

One commandRun the same check locally or in CI.
Machine-readableJSON output and contractual exit codes.
Agent-readyMCP tools expose features, dispatch, explanation, and recommendations.
Fail-closedMismatch is failure. Missing evidence is undetermined.
THE HONEST NEGATIVE RESULT

Dispatch is not speed: Polygraph rejected its own candidate.

A stricter paired Arm64 synthetic gate compared distinct baseline and candidate binaries. The candidate passed readiness and replay checks, but failed the throughput promotion threshold. The system rolled back instead of publishing a favorable isolated result as a cloud optimization.

Candidate / baseline throughputmedian of within-round paired ratios
VerdictFAILROLLBACK_TO_BASELINE
Traffic boundarysyntheticactualProductionTraffic: false

Try it, then inspect the proof.

Local walkthrough

git clone https://github.com/tomyimkc/polygraph
cd polygraph
make demo

Read the submission

Start with the evidence map, then follow the raw receipts. Every headline number is scoped and source-backed.

Open Devpost submission ↗

Loading committed receipt timestamps…