agentleFS
srimon12srimon12/duckdb-2-skills/skills/duckdb-migrate/SKILL.md

duckdb-migrate

Skill1 starsChanged 15 days agoMITView on GitHub

Migrate DuckDB 1.4 or 1.5 deployments to 2.0. Covers storage version 69, lambda and regex breaks, removed settings, export/import, and phased cutover. Use when the user asks to migrate or upgrade to DuckDB 2.0, when 1.x binaries cannot open a file, or to pin backward-compatible storage versions.

  • Deletes or force-pushes

The file

We split this skill into the rules it gives an agent

4 rules. About 0 are common, 1 is distinctive, and 3 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 3

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Back up: Create file backups or export catalogs and data.

    line 28
  2. Very specific:

    If the oldest reader is 1.4, use -storage-version v1.4.0.

    line 65
  3. Very specific:

    When migrating regex filters (~, !~), verify match semantics.

    line 126
  4. Very specific:

    Verify that the CLI reports 2.0.x and the newly created database reports compatibility tag v2.0.0+.

    line 174

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

DuckDB 2.0 migration

Execute migration steps sequentially. For Python API breaking changes, see duckdb-python. For lambda syntax rules, see duckdb-sql. Before upgrading a binary shared across processes, upgrade all readers to 2.0 or pin new files to a legacy storage version.

Support windows:

LineEnd of community support
2.0.0 LTS2027-10-22
1.4 LTS2026-11-17
1.52027-03-01

The release calendar tentatively schedules 2.0.1 for 2026-11-18. For mission-critical workloads, wait for patch releases and validate on a shadow copy; 1.5 community support continues through 2027-03-01.

Order of work

  1. Inventory: Catalog every process, database file, client library, and extension accessing DuckDB.
  2. Pin format: Enforce backward-compatible storage versions on any file that older readers must open.
  3. Back up: Create file backups or export catalogs and data; verify the restore procedure.
  4. Shadow run: Spin up a shadow database and replay production workloads against DuckDB 2.0.
  5. Upgrade: Install the 2.0 binary and update all extensions concurrently.
  6. SQL checks: Scan queries for syntax and semantic breaks, then run the full test suite.
  7. Cut over: Shift live workloads to 2.0, retaining previous binaries and backups until verification completes.

For step-by-step gate criteria and automated search patterns, see references/checklist.md.

Storage

DuckDB 2.0 creates files using storage version 69 by default. DuckDB 1.x binaries cannot read version 69 files. DuckDB 1.5.6 reads versions 64 through 68; DuckDB 1.5.5 and earlier print version number 999. A default 1.5.6 file uses storage version 64, not 68 (version 68 requires explicit -storage-version v1.5.6). DuckDB 2.0 reads and writes older storage versions without automatic upgrade; writes preserve the file's existing storage version.

Reader floorStartup pinCompatibility tagStorage version
2.0 onlydefault or latestv2.0.0+69
1.5.6-storage-version v1.5.6v1.5.0+68
1.4-storage-version v1.4.0v1.4.0+67

A 1.4 reader cannot open storage 68. A 1.5 reader cannot open storage 69. For client, BI, dbt, Ibis, and MCP compatibility, see references/clients.md.

To inspect the compatibility floor of an attached database:

SELECT database_name, tags['storage_version']
FROM duckdb_databases();

The tag reflects the compatibility floor: version 69 reports v2.0.0+, version 68 reports v1.5.0+, and default 1.5.6 reports v1.0.0+. The on-disk header for version 69 stores 999; never parse versions via hexdump.

Storage version pinning

To create a new database accessible to 1.5.6 readers:

duckdb my.db -storage-version v1.5.6 -c "CREATE TABLE t (x INT);"

If the oldest reader is 1.4, use -storage-version v1.4.0. To pin storage version within SQL:

SET storage_compatibility_version = 'v1.5.6';
ATTACH 'file.db' (STORAGE_VERSION 'v1.5.6');

In Python connections, supply config={"storage_compatibility_version": "v1.5.0"} (or "v1.4.0" for 1.4). The configuration sets the minimum compatibility floor; check tags['storage_version'] after the first write. latest resolves to version 69.

Scoping rules:

  • -storage-version sets the version for new databases and subsequent attaches within that CLI session; it is ignored on an existing main file.
  • SET storage_compatibility_version applies only to subsequent ATTACH statements, never to the currently open main database.
  • To upgrade an existing database to 2.0 format, run ATTACH 'existing.db' (STORAGE_VERSION 'latest') and execute a write.
  • Specifying a lower STORAGE_VERSION on an existing database raises an error; storage versions cannot be downgraded in place.

Database transfer: COPY vs EXPORT/IMPORT

To move an existing database to 2.0 storage, back up the file first. Choose the transfer method based on structure:

ATTACH 'old.db' AS old (READ_ONLY);
ATTACH 'new.db' AS new (STORAGE_VERSION 'v1.5.6');
COPY FROM DATABASE old TO new;

COPY FROM DATABASE copies schema, constraints, indexes, sequences, macros, and data into an attached target. The target's STORAGE_VERSION determines output format. Existing schemas are reused; same-named tables, sequences, or macros abort the operation.

For cross-version migration across different environments, use EXPORT and IMPORT:

# Export using source binary (this example writes Parquet; omit FORMAT for CSV)
duckdb old.db -c "EXPORT DATABASE 'export_dir' (FORMAT parquet);"
# Import using target binary
duckdb new.db -c "IMPORT DATABASE 'export_dir';"

The export directory must be empty. IMPORT DATABASE aborts if any same-named object or schema already exists in the target database.

To flush the write-ahead log (WAL) before copying, issue CHECKPOINT or FORCE CHECKPOINT. Clean exits remove the WAL automatically. To configure detachment checkpoint behavior, set checkpoint_on_detach to ENABLED, DISABLED, or DEFAULT (not boolean).

SQL text

Scan application queries, dbt models, and scripts for syntax and semantic breaking changes:

Pattern2.0 behaviorReplacement
Legacy arrow (->)Disabled by defaultlambda x: expr
1 // 0, 1 % 0Raises division-by-zero errorTRY(expr); error_on_division_by_zero is deprecated
col ~ 'pat', col !~ 'pat'Matches substring (PostgreSQL standard)regexp_full_match(col, 'pat') for 1.x whole-string match
s3://bucket/key?s3_access_key_id=...Query parameters treated as literal key pathStore credentials in a Secret; s3_enable_deprecated_url_parameters temporarily
custom_profiling_settingsSetting removedtracked_metrics
enable_http_logging, http_logging_outputSettings removedCALL enable_logging('HTTP')
profiling_modeDeprecated and ignoredSET enable_profiling
preserve_identifier_caseBoolean deprecated; expects string enum'preserve_case', 'lowercase', or 'uppercase'
error_on_division_by_zero = falseDeprecatedWrap expressions in TRY(expr)
table_function_identifier_conversionDeprecatedExplicitly quote table-function arguments
Unquoted read_csv(file.csv)Implicit identifier conversion deprecatedread_csv('file.csv')
CREATE TABLE t AS SELECT NULL AS xAssigns column type "NULL"; typed inserts failSELECT NULL::INTEGER AS x; legacy_disable_null_type is deprecated
X'4142', B'1010'Inferred as BLOB and BIT, not VARCHARExplicitly cast or compare matching data types
2 * 3!Evaluates to 12 (2 * (3!)), not 720Parenthesize (2 * 3)! to retain 1.x factorial precedence

When migrating regex filters (~, !~), verify match semantics. DuckDB 1.x matched entire strings; DuckDB 2.0 matches substrings. Unadjusted queries like WHERE msg ~ 'ERROR' return additional matching rows.

To temporarily unblock legacy queries during migration:

SET enable_legacy_parser = true;

This setting autoloads the legacy parser extension as a syntax fallback, not a semantic time machine. It rejects simplified FROM ... NEAREST syntax (explicit INNER JOIN ... NEAREST remains supported). Use it strictly as a temporary bridge while updating query text.

CLI and deployment

For CLI output formatting and max_execution_time, see duckdb.

CLI sessions since 1.5 route messages through shell_log_storage, omitting them from duckdb_logs. To query CLI logs via SQL, enable memory storage:

CALL enable_logging(storage = 'memory');
SELECT * FROM duckdb_logs();

Deployment constraints:

  • ICU collation and time zones: ICU logic moved from core to the icu extension. Official binaries autoload icu automatically (no SQL changes required); custom builds that previously linked ICU must explicitly load the icu extension.
  • Build prerequisites: Building DuckDB 2.0 requires a C++17 compiler and CMake (amalgamation builds are deprecated). Precompiled Linux binaries require glibc 2.28 or newer.
  • C API v2: Ships with 2.0 but remains experimental; do not assume ABI stability.
  • macOS x86_64 deprecation: Community extensions no longer build for osx_amd64 (core extensions still build). DuckDB 2.1 will drop osx_amd64 completely; plan hardware upgrades prior to 2.1.

Python and extensions

Python installations require Python 3.11+. For breaking changes to relations, fetching, Arrow conversion, thread safety, and transactions, see duckdb-python.

Extension binaries are strictly compiled against specific DuckDB versions. After updating the DuckDB binary, run UPDATE EXTENSIONS; or FORCE INSTALL <name>; and restart the process. Core extensions json, parquet, and icu ship statically linked with official binaries.

Rollback

Retain previous binaries and uncompressed file backups until the 2.0 deployment completes all workload cycles. In-place storage version upgrades cannot be reversed. If a database file was written in version 69, restore from the backup copy or execute a down-copy via COPY FROM DATABASE into a target attached with STORAGE_VERSION 'v1.5.6'. Never attempt to open a storage version 69 file with DuckDB 1.x.

Verify

Validate the binary version and new database storage format:

duckdb -no-agent --version
tmp=$(mktemp -d)
duckdb -agent "$tmp/check.duckdb" -c "CREATE TABLE t AS SELECT 42 AS a; SELECT tags['storage_version'] FROM duckdb_databases();"
rm -rf "$tmp"

Verify that the CLI reports 2.0.x and the newly created database reports compatibility tag v2.0.0+. In-memory databases do not report a storage version tag. After binary verification, execute the full application test suite against a restored shadow database.

References

  • DuckDB 2.0 Migration Guide (breaking changes and migration overview)
  • Storage Versions (storage formats and compatibility matrix)
  • Release Calendar (release timelines and support lifecycles)
  • ATTACH Statement (STORAGE_VERSION syntax)
  • Python Migration Guide (client-specific breaking changes)
  • Regular Expressions (~ and !~ match semantics)
  • Checklist & Grep Patterns (phase gates and automated scanning patterns)
  • Client Compatibility (BI, dbt, Ibis, and MCP compatibility matrix)
  • Storage tags, pinned 1.5 storage files, COPY FROM DATABASE, and agent-mode outputs verified on DuckDB 2.0.0.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.