> ## Documentation Index
> Fetch the complete documentation index at: https://supaschema.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmarks

> Latency, accuracy, and replay-safety results for supaschema compared with Supabase CLI diff engines.

The benchmark harness compares `supaschema` with Supabase CLI diff engines on identical fixtures.

Each generated migration is:

1. applied once;
2. applied again;
3. compared against the target catalog.

Reference run: 2026-07-21 on a MacBook Pro (Apple M4 Max, 14 cores, 36 GB), macOS 26.5.1, Node 24.18.0, PostgreSQL 17.6, Supabase CLI 2.109.1, and supaschema 0.4.3. Every cell used one warmup and three measured iterations, except XXL with one measured iteration.

<Info>
  Short version: supaschema stays fast at large schema sizes, keeps F1 at
  `1.000`, and produces migrations that survive a second apply.
</Info>

## Speed and accuracy

At 1,000 tables (\~7,000 objects), `supaschema` against each of the five Supabase CLI engines — median diff latency, accuracy (F1 vs a ground-truth change manifest), and whether the migration survives a second apply:

<Frame>
  <img src="https://mintcdn.com/supaschema/S_dB4SqQqbQpfDn2/images/benchmarks/head-to-head-xl.svg?fit=max&auto=format&n=S_dB4SqQqbQpfDn2&q=85&s=de10eb08fcc7c3bfc90ae50f7b5c99b6" alt="supaschema vs diff engines at 1,000 tables: latency bars, F1 accuracy, and replay-safety" width="1000" height="648" data-path="images/benchmarks/head-to-head-xl.svg" />
</Frame>

The gap widens with scale. At 1,000 tables, supaschema measured `2.08-2.53s` while the five Supabase diff engines measured `38.7-57.7s`. At 2,500 tables (\~17,500 objects), supaschema measured `4.71-5.80s` while the engines measured `268-352s`:

<Frame>
  <img src="https://mintcdn.com/supaschema/S_dB4SqQqbQpfDn2/images/benchmarks/head-to-head-xxl.svg?fit=max&auto=format&n=S_dB4SqQqbQpfDn2&q=85&s=7341c84288279cff3c16c57fd5d8c4aa" alt="supaschema vs diff engines at 2,500 tables: latency bars, F1 accuracy, and replay-safety" width="1000" height="648" data-path="images/benchmarks/head-to-head-xxl.svg" />
</Frame>

## Full workflow

The diff is only half the loop. Getting a migration **and** refreshed TypeScript/Zod contracts is one `supaschema sync` dry-run workflow — against the CLI it takes three commands (`db diff`, apply, `gen types`) and a database that has already caught up. At 1,000 tables, the measured workflow medians were `7.34s` for supaschema and `43.0-57.6s` for the five Supabase workflows:

<Frame>
  <img src="https://mintcdn.com/supaschema/S_dB4SqQqbQpfDn2/images/benchmarks/head-to-head-workflow-xl.svg?fit=max&auto=format&n=S_dB4SqQqbQpfDn2&q=85&s=6977e4c2a56494a7af8916c0b6e47a94" alt="full workflow vs diff engines at 1,000 tables: supaschema sync migration plus refreshed contract outputs in one command versus db diff, apply, and gen types per engine" width="1000" height="586" data-path="images/benchmarks/head-to-head-workflow-xl.svg" />
</Frame>

## Accuracy

Diff output is scored two ways:

* **F1** against a ground-truth change manifest by object identity.
* **Catalog fingerprint** after applying the generated migration to a throwaway database.

`supaschema` scores F1 `1.000` on every manifest-carrying fixture in source-file, live-catalog, and full-workflow modes.

Successful Supabase direct and workflow output scores `0.800-1.000` across the fixtures. On the realistic, XL, and XXL fixtures, every engine misses the same RLS policy change, scoring `0.982`, `0.999`, and `0.999` respectively.

## Replay safety

<Frame>
  <img src="https://mintcdn.com/supaschema/S_dB4SqQqbQpfDn2/images/benchmarks/xl-correctness.svg?fit=max&auto=format&n=S_dB4SqQqbQpfDn2&q=85&s=f07b7a882fcda6a9d22daf5d1067097f" alt="migration verification at 1,000 tables across supaschema and Supabase CLI engines" width="1200" height="528" data-path="images/benchmarks/xl-correctness.svg" />
</Frame>

The chart above is the direct diff lane: supaschema source-file and live-catalog output applies twice on every fixture. Successful Supabase direct rows reach the target on the first apply but fail the second apply on additive, realistic, XL, and XXL because they emit unguarded column or index changes. The separate workflow lane produces the same replay pattern, while the replace-only functions/policies fixture succeeds twice in both lanes.

The publication set retains four measured Supabase command failures: three on the smallest fixtures and one XL row. There were no supaschema failures, timeouts, skips, or unsupported rows. See the [dated reference report](https://github.com/jmclaughlin724/supaschema/blob/main/benchmarks/REFERENCE_RESULTS.md) for the exact adapters and iterations.

## Reproduce

Start a disposable local PostgreSQL instance, then run the complete publication harness from this repository:

```bash theme={null}
export SUPASCHEMA_COMPARE_DATABASE_URL='postgresql://postgres:postgres@127.0.0.1:54322/postgres?sslmode=disable'
SUPABASE_TELEMETRY_DISABLED=1 BENCH_ALL_SEQUENTIAL=1 bash benchmarks/tools/bench-all.sh
npm run bench:plot:docs
```

<Warning>
  Never point the harness at development, staging, or production. It creates and
  drops comparison databases. Publication results are advisory and sensitive to
  the machine and installed tool versions; `npm run benchmark` is a separate
  internal threshold suite, not the publication command or a CI gate.
</Warning>

The harness and comparison utilities live in the repository's `benchmarks/` directory. The ignored comparison JSONs and generated summary are the source evidence; the 14 tracked SVGs are the publication artifacts.