DuckDB on TiDB File System: Let the Process Go, Keep the Data

An experiment in reconnecting compute to the same files—and, experimentally, the same native database.

I recently read A Real DuckDB Database in S3, With Nothing Else to Run. It describes a filesystem extension that stores native DuckDB databases in object storage, using SlateDB and OpenDAL underneath.

That gave me a question: what if the filesystem already had the machinery DuckDB needed?

TiDB File System already had a shared namespace, file revisions, extent storage, sessions, and locks. Could DuckDB use those capabilities to leave its data behind, then let another process reopen it somewhere else?

I built tidbfs, a DuckDB filesystem extension to find out. The experiment works, with some important boundaries. Ordinary analytical files are useful today. Writable native databases are still experimental.

The process should be easier to replace than the data

Starting another DuckDB process is easy. Moving the state it needs can be the annoying part.

A job produces a database on one machine. Another job needs it elsewhere. Suddenly the workflow includes copying files, deciding which copy is current, and making sure nothing is still writing the copy being moved.

I wanted a different workflow: start DuckDB, connect to an authorized filesystem, do the work, and exit. A later process reconnects to the same paths. The data does not need to travel with the process's suitcase.

This is useful for short-lived jobs, replacement workers, and services whose compute comes and goes. The experiment uses an existing filesystem service, backed by TiDB and object storage. That service remains part of the architecture.

Two useful paths into the same filesystem

The extension registers two URI schemes:

• tidbfs:///path/file.parquet reads and writes ordinary analytical files. It provides per-file consistency and publication, without a dataset-wide transaction.

• tidbfs+db:///warehouse.duckdb runs DuckDB’s native storage engine on extent files. It is experimental, with one writer or shared read-only processes.

Both address the same filesystem namespace. The second scheme selects extent I/O and database locking; it does not create a separate storage volume.

SQL executes in DuckDB. The filesystem handles files, metadata, and bytes. Native database mode reuses the existing extent path.

The ordinary file path supports CSV, Parquet, and JSON, including writes through COPY, partitioned output, and database export/import. DuckDB still chooses the columns, partitions, and row groups it needs. The extension turns those choices into remote file operations.

The native path lets DuckDB use its own database format, WAL, and checkpoints. The adapter links the public TiDB File System C++ SDK directly, using the existing extent APIs for inode metadata, immutable object blocks, sessions, advisory locks, and fsync. It runs inside DuckDB, with no Go bridge, Go runtime, or separate helper process.

The filesystem server does not interpret DuckDB transactions or recognize special WAL filenames. It serves the same general file and extent interfaces used by other clients. The native-mode guide describes that mapping.

Make the filesystem useful to DuckDB's optimizer

Remote storage is less interesting if every query starts by downloading everything.

For ordinary files, the extension shares a metadata lookup across a query's readers, pins the selected file generation, reuses connections and object URLs, and uses bounded read-ahead. Conditional Range requests make the selected bytes available without silently switching versions halfway through a scan.

The server changes stay in the existing /v1/fs API: strong ETags, conditional reads, and correct byte ranges. For inline content stored in TiDB, SQL byte projections reduce the bytes transferred from TiDB to the filesystem server. They do not prove fewer physical TiKV page reads. Large single-object files use object-store range reads; existing extent files use the extent adapter.

SQL predicates still run in DuckDB. The filesystem makes the resulting selective I/O practical. It does not become a second query engine. The server contract spells out the required behavior.

The difficult test is the old writer coming back

Writing rows and reopening a file is a useful smoke test. It is not the most interesting failure test.

Consider this sequence:

1. Writer A loses access to metadata and object storage.

2. Its filesystem session expires and cleanup releases its locks.

3. Writer B acquires ownership and continues.

4. Writer A's connection comes back.

A must not resume publishing old state.

We checked the existing session and revision mechanisms before adding protection. Revisions already handled optimistic publication conflicts. The remaining problem was admitting a delayed mutation from a session that had expired or been collected.

The fix belongs in the shared extent protocol. The public client carries its existing session ID. A supporting server validates that session in the same transaction as the metadata mutation, while session cleanup serializes against admitted writes. No DuckDB-specific lease or new set of tables is needed.

The adapter also stops using a failed or expired session. It does not quietly acquire a new identity beneath an already-open DuckDB instance. Servers must advertise session_guard: true for the transactional stale-session guarantee; older servers retain their legacy behavior.

The reconnect test records takeover, rejection of the old writer's COMMIT, and full-row validation. Other live tests use real TiDB/TiKV and a disposable RustFS object store to exercise lost responses, interrupted uploads, truncated reads, and recovery. These are evidence for the tested cases, not a certification of every production failure.

Try it

The current target is DuckDB 2.0.0, with macOS arm64 and Linux amd64 validation. The extension must match the DuckDB version and platform that loads it. It is unsigned and is not yet available through a public INSTALL tidbfs repository.

For the macOS SDK build, install a C++17 compiler, matching DuckDB headers, CMake, OpenSSL, libcurl, and Python 3.10 or newer, then:

git clone --recurse-submodules https://github.com/drive9-ai/duckdb.git
cd duckdb
make build
duckdb -unsigned

Both platforms produce one tidbfs.duckdb_extension file; no companion runtime library is needed. For Linux setup or a reproducible release build, follow the repository's build instructions.

Create a filesystem in the TiDB Cloud console and save its owner token when it is first displayed. For an existing filesystem, the TiDB Cloud CLI can issue another owner token; the token guide covers both paths. Use a filesystem token (fs_token), rather than a TiDB Cloud API key pair.

Load the extension and configure your filesystem credentials:

LOAD 'build/tidbfs.duckdb_extension';

CREATE TEMPORARY SECRET fs (
    TYPE tidbfs,
    ENDPOINT 'https://YOUR_FILESYSTEM_ENDPOINT',
    TOKEN 'YOUR_FILESYSTEM_TOKEN'
);

The endpoint is the filesystem data-plane base URL for your filesystem's region, not a TiDB SQL endpoint. Do not append /v1/fs; the extension adds it. The examples use an unscoped owner token, required for native databases and existing extent files. Supply the plaintext token without a Bearer prefix. Keep real credentials out of source control and process arguments.

For an ordinary Parquet file:

COPY (
    SELECT i AS id, i * 10 AS amount
    FROM range(1000) AS t(i)
)
TO 'tidbfs:///totals.parquet' (FORMAT PARQUET);

SELECT SUM(amount)
FROM read_parquet('tidbfs:///totals.parquet');

For an experimental native database:

ATTACH 'tidbfs+db:///warehouse.duckdb' AS warehouse;

CREATE TABLE warehouse.items (
    id BIGINT PRIMARY KEY,
    name VARCHAR
);

BEGIN;
INSERT INTO warehouse.items VALUES (1, 'hello');
COMMIT;

CHECKPOINT warehouse;
DETACH warehouse;

After the writer releases the database, another process with the extension loaded and credentials configured can reopen it:

ATTACH 'tidbfs+db:///warehouse.duckdb'
AS warehouse (READ_ONLY);

SELECT * FROM warehouse.items;

What I would use it for—and what I would not promise

The immediate value is a durable handoff. An ephemeral job can leave analytical files or an experimental native database for the next process. A browser tool can inspect ordinary files produced by a CLI job. A worker can be replaced without manually shipping its database to the replacement.

The native concurrency model is one exclusive writer or multiple shared readers. It does not provide concurrent independent writers, or a writer alongside external read-only processes. The locks are advisory: every client touching a live database and its WAL must cooperate. An ordinary authorized client that bypasses those locks can still damage the database.

Multi-file exports are not atomic filesystem transactions. A timeout during COMMIT can leave an unknown outcome; reopen and reconcile application state before deciding what to replay. Production object-store and TiKV failover coverage remains incomplete. The compatibility matrix is part of the contract.

The original question now has a practical answer: an existing filesystem can supply enough machinery for this experiment, without learning DuckDB's internals. The next work is to measure and harden that boundary, while keeping the storage APIs useful to clients beyond DuckDB.

Disclosure: I dictated the content of this article, then used GPT 6.1 sol to organize and edit it.