S3 and httpfs

中文 · Docs

DuckLocal can read objects from S3 and S3-compatible storage through DuckDB's httpfs extension. Configure it from the S3 button in the title bar.

Configure credentials

The dialog has four fields:

Field	Default	Notes
Endpoint	s3.amazonaws.com	http:// is honored; otherwise HTTPS is used
Region	us-east-1	Sent as typed
Access Key ID	—	Required
Secret Access Key	—	Required, masked, with a reveal toggle

Confirming runs the equivalent of:

INSTALL httpfs; LOAD httpfs;
CREATE OR REPLACE SECRET ducklocal_s3 (
  TYPE s3, PROVIDER config,
  KEY_ID '…', SECRET '…',
  REGION '…', ENDPOINT '…'
);

Credentials are session-only. Nothing is written to disk, and they are dropped when you open or switch a database — reconfigure afterwards. There is no way to clear them other than reconfiguring.

Browse

Once configured, an S3 root appears in the sidebar. It loads lazily: expanding the root lists buckets, and expanding a bucket or prefix lists its objects. Buckets and prefixes are sorted, and objects show a human-readable size. A refresh button reloads a level, and also serves as the retry after an error.

Listing is capped at 1,000 entries per level, even when the response says more are available. There is no deeper paging, no prefix filter box, and no search — a bucket with more than a thousand objects at one level will appear complete when it is not.

Query objects

Clicking an object whose key has a data-file extension fills the active editor with:

SELECT *
FROM 's3://bucket/key'
LIMIT 100;

That is an ordinary query: the read goes through httpfs, so DuckDB's read_parquet / read_csv_auto behaviour, and its pushdown, apply. The query replaces whatever the active tab held.

Keys that are not recognized as data files are not clickable.

Limitations

• Browsing is read-only. The browser has no upload, download-to-file, delete, copy, or bucket-creation action. Queries still download remote data.
• No session tokens. The X-Amz-Security-Token header is never sent, so temporary STS credentials will not work — use long-lived keys, or an endpoint that accepts key/secret alone.
• Endpoint style is mostly automatic. Virtual-hosted addressing is used when the host contains amazonaws.com, path style otherwise. For self-hosted gateways this is usually what you want; for one that needs the other style, there is no override.
• A wrong region is retried once, using the region DuckDB reports back. If that fails too, the error is shown as-is.
• Network is required: INSTALL httpfs fetches the extension on first use. Browsing sends HTTP or HTTPS requests to the configured endpoint (http:// selects HTTP; otherwise HTTPS is used). These requests send metadata such as bucket names and key prefixes, plus authentication information in signed headers; browsing does not upload local database files or query results. Queries on remote objects download data to your machine through httpfs.

Troubleshooting

Symptom	Likely cause
Browsing fails immediately after configuring	Region or endpoint wrong; check the message shown in the sidebar
Errors on a self-hosted endpoint	Wrong addressing style, or an endpoint that requires a session token
httpfs installation error	No network, or a proxy that blocks the extension download
Credentials seem forgotten	A database was opened or switched, which clears them by design

When DuckLocal is packaged as a signed app, loading the httpfs extension needs the com.apple.security.cs.disable-library-validation entitlement; the shipped packaging script already includes it.