πŸ¦† duck_diff

Two tables, maybe two entirely different databases, one question: are these actually the same rows? duck_diff is a DuckDB extension that answers per key and per column β€” across any pair of systems DuckDB can reach: Postgres, MySQL, ClickHouse, BigQuery, Snowflake, Iceberg, Delta, MongoDB, plain Parquet files, and more. Everything below runs locally in the duckdb CLI; your data never leaves your pond.

Since duck_diff is deterministic, it's a validation step Claude can run in loops (i.e. with Claude Code's /goal β€” examples below): safely refactor a model, optimize a slow query, or transpile SQL to another dialect, checking its own correctness after every change and stopping only when the diff comes back clean.

$ duckdb INSTALL duck_diff FROM community; -- one time LOAD duck_diff;

1 Β· Spot the odd duck diff two tables in one line

Every row gets a verdict β€” identical, different, left_only, right_only β€” plus a JSON summary of exactly which columns changed, and typed <col>_left / <col>_right columns you can filter on. Take the whole thing with SELECT *, or project just the slice your use case needs. Paste this into any duckdb shell:

spot the odd duck (click it)

2 Β· Get your ducks in a row verify CDC pipes, replicas, and migrations are in sync

The relation arguments are just query strings, so each side can live in a different system. Choose source and destination and copy the recipe. It always has three parts: a schema diff (do the columns and types even line up?), a summary (one row of counts β€” in sync means everything is n_identical), and a drill-down (every drifted key and exactly which columns changed). Credentials come from environment variables β€” nothing sensitive lands in the SQL. In SQL query mode, connectors with a native pass-through (Postgres, MySQL, SQL Server, BigQuery, Snowflake, ClickHouse) send your query to the remote system in its own dialect β€” no catalog introspection, and only the rows you asked for come back.

Comma-separate for a composite key.
Advanced options β€” type promotion, tolerances, ignored columns
Metadata churn you don't want flagged.
Type promotion (upcast_types)Reconcile differing types β€” MySQL INT vs BigQuery INT64, Postgres NUMERIC vs Snowflake NUMBER β€” on their common super-type. Auto-enabled for cross-system pairs.
NULL = '' (null_equals_empty)Treat NULL and empty string as equal for text columns β€” common after CSV round-trips.
Numeric tolerance (numeric_tolerance)Floating-point values rarely survive two engines bit-for-bit. Treat |left βˆ’ right| ≀ Ξ΅ as identical.
Ξ΅
Timestamp truncation (timestamp_precision)Drop sub-second (or finer) noise lost in transit before comparing.
Full-row context (context := ['*'])Also expand every non-compared column as <c>_left/<c>_right, so left_only/right_only rows arrive with their whole row visible.

Export credentials & launch (placeholders β€” swap in real values, or use a secret-manager wrapper like op run / doppler run)

Run the SQL in duckdb

    Playbook β€” load a ready-made scenario

    Seven scenarios from the repo's demo suite β€” each card loads the matching recipe into the builder above, and links to a fully runnable version (env-var secrets, HTML reports, and a docker-compose playground with intentionally drifted data).

    3 Β· Same query, new pond transpile SQL between dialects, with Claude in the loop

    LLMs translate SQL dialects well but can't tell when they got it right β€” the diff can. This prompt is wired to the source and destination you picked above: copy it into Claude Code and it converts one section at a time, using the diff as a mechanical acceptance test it cannot rationalize around, and doesn't stop until the diff is clean. Expand it with your own schema and context β€” or use the repo's sql-migrate skill, which any Claude Code session in the repo picks up automatically.

    Then set the goal as its own message β€” slash commands only trigger at the start of the input, so pasting it inside the prompt above does nothing. Sent separately, Claude Code keeps working across turns until the condition verifiably holds:

    4 Β· Fly faster, land the same optimize a query without changing its answer

    The same acceptance loop turns performance tuning into a safe search: rewrite β†’ time it β†’ diff it β†’ keep the rewrite only if it's faster AND the diff says identical β†’ repeat. A real example on a public dataset (231k US flights, vega-datasets CSV) β€” the quadratic self-join takes 17.0 s, the window-function rewrite 0.01 s (~1700Γ—), and the diff proves all 177 result rows are byte-identical:

    The repo's sql-optimize skill runs this loop autonomously: one rewrite hypothesis at a time, timing as the fitness function, the diff as the correctness gate β€” because the check is mechanical, Claude can verify its own work at every step.

    Hand it to Claude

    Paste this as a message with your slow query…

    …then set the goal as its own message (slash commands only trigger at the start of the input):