conediff

Diff two recorded AI runs across a model or tool swap. See exactly what changed in outcome, cost, and latency — with a per-step timeline.

What it does

You swapped a model, re-ordered a tool chain, or changed a prompt, and you have two recorded runs. conediff compares them deterministically and tells you what actually changed: the outcome diff, the cost delta, the latency delta, and a per-step attribution so you can see which step moved.

It is honest by construction. A field that is missing, untyped, or corrupt stays unknown — it is never silently coerced to zero, pass, or fail. The report is both machine-readable JSON and a human-readable timeline, and it is pure: the same input always produces byte-identical output.

Try the local CLI

No account, no upload, no provider keys. It runs on local fixtures:

git clone https://github.com/gitdhillonai/conediff.dev
cd conediff.dev
node src/cli.mjs fixtures/positive-outcome-cost-changed.json --text

Point it at a Langfuse-style trace pair to normalize a real recorded-run export into the comparison contract:

node src/cli.mjs --langfuse baseline-trace.json candidate-trace.json --pretty

Why it is trustworthy

Same product, two feature names

Run ledger is the portable timeline of a recorded pair. Tool waterfall is the per-step latency attribution. Both are pages on conediff. The names runledgerkit.dev and toolwaterfall.dev are aliases of this product. DNS is not cut over, and neither name is a separate company or repository.

Get started

The CLI and fixtures are the free, local trial. Hosted run history is a proposed paid offer. Checkout stays disabled until a test key is configured. See checkout.

Create a free account Read the docs on GitHub