Skip to main content
The benchmark harness compares supaschema with Supabase CLI diff engines on identical fixtures. Each generated migration is:
  1. applied once;
  2. applied again;
  3. compared against the target catalog.
Reference run: 2026-07-21 on a MacBook Pro (Apple M4 Max, 14 cores, 36 GB), macOS 26.5.1, Node 24.18.0, PostgreSQL 17.6, Supabase CLI 2.109.1, and supaschema 0.4.3. Every cell used one warmup and three measured iterations, except XXL with one measured iteration.
Short version: supaschema stays fast at large schema sizes, keeps F1 at 1.000, and produces migrations that survive a second apply.

​
Speed and accuracy

At 1,000 tables (~7,000 objects), supaschema against each of the five Supabase CLI engines — median diff latency, accuracy (F1 vs a ground-truth change manifest), and whether the migration survives a second apply:
supaschema vs diff engines at 1,000 tables: latency bars, F1 accuracy, and replay-safety
The gap widens with scale. At 1,000 tables, supaschema measured 2.08-2.53s while the five Supabase diff engines measured 38.7-57.7s. At 2,500 tables (~17,500 objects), supaschema measured 4.71-5.80s while the engines measured 268-352s:
supaschema vs diff engines at 2,500 tables: latency bars, F1 accuracy, and replay-safety

​
Full workflow

The diff is only half the loop. Getting a migration and refreshed TypeScript/Zod contracts is one supaschema sync dry-run workflow — against the CLI it takes three commands (db diff, apply, gen types) and a database that has already caught up. At 1,000 tables, the measured workflow medians were 7.34s for supaschema and 43.0-57.6s for the five Supabase workflows:
full workflow vs diff engines at 1,000 tables: supaschema sync migration plus refreshed contract outputs in one command versus db diff, apply, and gen types per engine

​
Accuracy

Diff output is scored two ways:
  • F1 against a ground-truth change manifest by object identity.
  • Catalog fingerprint after applying the generated migration to a throwaway database.
supaschema scores F1 1.000 on every manifest-carrying fixture in source-file, live-catalog, and full-workflow modes. Successful Supabase direct and workflow output scores 0.800-1.000 across the fixtures. On the realistic, XL, and XXL fixtures, every engine misses the same RLS policy change, scoring 0.982, 0.999, and 0.999 respectively.

​
Replay safety

migration verification at 1,000 tables across supaschema and Supabase CLI engines
The chart above is the direct diff lane: supaschema source-file and live-catalog output applies twice on every fixture. Successful Supabase direct rows reach the target on the first apply but fail the second apply on additive, realistic, XL, and XXL because they emit unguarded column or index changes. The separate workflow lane produces the same replay pattern, while the replace-only functions/policies fixture succeeds twice in both lanes. The publication set retains four measured Supabase command failures: three on the smallest fixtures and one XL row. There were no supaschema failures, timeouts, skips, or unsupported rows. See the dated reference report for the exact adapters and iterations.

​
Reproduce

Start a disposable local PostgreSQL instance, then run the complete publication harness from this repository:
Never point the harness at development, staging, or production. It creates and drops comparison databases. Publication results are advisory and sensitive to the machine and installed tool versions; npm run benchmark is a separate internal threshold suite, not the publication command or a CI gate.
The harness and comparison utilities live in the repository’s benchmarks/ directory. The ignored comparison JSONs and generated summary are the source evidence; the 14 tracked SVGs are the publication artifacts.
Last modified on July 24, 2026