New view_table_changes() opens a table's change feed in an interactive
viewer, an htmlwidget. A sidebar lists the schema changes, the columns
and cells that changed, the rows inserted, updated, and deleted, and the
snapshots in range (each with its author and message), with a count on
every entry. The main panel shows the selected part as a table. An
update is one row, shown from its old or its new side, with the changed
cells highlighted and both values on hover, and a cells view lists every
change as snapshot, rowid, column, old, new. It takes the lazy feed from
get_table_changes(), so dplyr filters run in DuckDB before anything is
collected, or that feed collected into a data frame. The layout follows
Hadley Wickham's data-diff (https://github.com/hadley/data-diff), with
thanks. htmlwidgets joins Suggests, with jsonlite for the viewer's
tests.
get_table_changes() defaults start and end to the table's full
history, and its result carries a ducklake_changes attribute (table,
lake, range) that view_table_changes() reads. The attribute survives
dplyr verbs on the lazy table and is dropped by collect().
New create_check() and run_checks() (experimental) keep data checks
in the lake. A check is a view, in a schema of its own, that returns the
rows breaking a rule, with the rule's label as its comment, so the rules
live in the catalog, are versioned with the data, and run from any client
of the lake. create_check() takes the rule as it is said (cyl %in% c(4, 6, 8), dose != 0), keeps the rows where it is false, and writes
the schema, the view, and the label as one snapshot; a rule that is
easier to state as its failure is a pipeline into create_view().
run_checks() returns one row per check: its label and how many rows
fail. Inside with_transaction() the counts include pending writes, so
a load that fails a check can be rolled back before it commits, and on a
snapshot-pinned attach the rules and the data are both read as of that
snapshot. vignette("data-checks") walks through it and sets the
approach next to affirm and data-dict. DuckLake enforces NOT NULL and
nothing else, and its catalog takes no custom tags from SQL, which is why
the rules are views.
set_table_comment() comments a view as well as a table. It used to send
COMMENT ON TABLE for every name, which DuckLake refuses for a view, so
get_table_comments() could read a view's comment but nothing in the
package could write one.
create_view() keeps a view's comment when it replaces the view. DuckLake
stores the replacement as a new catalog entry and keys the comment to the
entry, so CREATE OR REPLACE VIEW by itself drops it. create_view()
reads the comment first and sets it again in the same snapshot, as
replace_table() does for a table's comments. A check's label is its
view's comment, so revising a rule keeps the label.
get_table_comments() reads comments through DuckDB's catalog functions
(duckdb_tables(), duckdb_views(), duckdb_columns()) instead of the
DuckLake metadata tables. It used to return the lake's latest committed
comments whatever the session was looking at, which was wrong in two
situations. On a lake attached with snapshot_version or snapshot_time
it returned present-day comments, so collect() restored labels that
were not in force at that snapshot. Inside an open transaction it missed
the comments that transaction had set, so a replace_table() there put
the older committed comments back over them, and create_table() from a
query did not inherit them. All of these now follow what the session
sees.
A time-travel read restores the variable labels in force at its snapshot.
collect() on get_ducklake_table_version() or
get_ducklake_table_asof() used to attach today's labels to the
historical rows, so a label reworded since showed its new text and a
label added since appeared where none had been. The lazy table now
records the version or timestamp it asked for, and collect() reads the
column comments as of that snapshot from the catalog's versioned rows,
finding the table under the name and id it had then. A table written from
a time-travel read still takes the present-day comments, as
restore_table_version() does.
replace_table() inside a transaction no longer fails on a table that has
table-scoped options. DuckLake cannot set options on a table created in
the open transaction, so the rewrite is meant to go ahead and warn with
the set_ducklake_option() calls to run after the commit, but the
warning itself raised a cli pluralization error, which rolled the whole
transaction back. Outside a transaction, replace_table() and
restore_table_version() now carry target_file_size,
parquet_row_group_size_bytes, and parquet_version over. DuckLake
reports the sizes as a bare number of bytes and the version as V2,
forms set_option() refuses, so the reapply failed after the data had
already been replaced and the option stayed behind on the dropped table.
replace_table() respects partition and sort keys changed earlier in the
same transaction. It reads the keys to carry over from DuckLake's
metadata tables, which hold committed rows only, and no other surface
shows pending keys. A table given keys with set_table_partitioning() or
set_table_sorting() and rewritten in one transaction therefore lost
them, and keys removed with reset_table_partitioning() or
reset_table_sorting() came back. Those four functions now note what
they did while a transaction is open, tagged with its transaction id so
that nothing outlives it, and the rewrite uses the note. Keys changed
with a raw ALTER TABLE statement in the same transaction are still not
seen.
restore_table_version() refuses to run inside an open transaction, and
says why, before it touches anything. It commits for itself, so its own
BEGIN failed there with "cannot start a transaction within a
transaction", under a hint about snapshots that did not apply, and the
error handler's rollback discarded the caller's uncommitted work.
The lifecycle stage is now stable, with ducklake on CRAN since 0.6.0 (published 2026-09-09): the interface is settled, and any breaking change will come with a deprecation cycle.
commit_transaction() and with_transaction() name the snapshot this
connection committed, read from DuckLake's last_committed_snapshot(),
instead of the lake's newest snapshot right after the commit. The two
agree on a lake with one writer, but with several sessions writing at
once the newest snapshot could be a neighbor's, and a transaction that
changed nothing could be reported as the commit of a snapshot that was
not its own. The "no changes" confirmation still names the snapshot the
lake stands at. An extension without the function falls back to the
previous reading.
attach_ducklake() labels the creation snapshot of a lake it creates.
DuckLake writes snapshot 0 itself when it makes a lake, with no author
and no commit message, so every history began with an <NA> row. The
new commit_message (default "Create lake") and author arguments
are written onto snapshot 0 right after the lake is created, with the
same metadata update set_snapshot_metadata() uses, because DuckLake
does not take set_commit_message() for that snapshot. A lake that
already exists is left alone, and so is a read-only, snapshot-pinned, or
in-transaction attach; commit_message = NULL keeps the old behavior.
set_snapshot_metadata() gains snapshot_id, so the creation snapshot
of an existing lake, or any other snapshot, can be labeled after the
fact; it still fills blanks only unless overwrite = TRUE, and its
confirmation now names the snapshot.
New ducklake.author option: the author recorded on every snapshot the
session commits without naming one, through commit_transaction(),
with_transaction(), restore_table_version(), and the creation
snapshot attach_ducklake() labels. An author argument still wins.
set_snapshot_metadata() does not read it, and a write made outside a
transaction records no author, as before.
The metadata readers honor the metadata_schema a lake was attached
with. They looked in the catalog's default schema whatever the attach
said, so set_snapshot_metadata(), get_table_comments(),
get_table_partitions(), get_table_sorting(), get_table_info(), and
get_metadata_table() failed on such a lake; the schema is now kept
with the lake's registry entry and used by all of them.
plot_snapshots() classifies each snapshot by what it did to the table
being drawn. A transaction that updated one table in place and rebuilt
another used to color both as "created", because the whole snapshot was
classified by the highest-precedence change it carried; the single-table
timeline and the swimlane now read only the change-map entries that name
the table, so the updated table shows a data change and the rebuilt one a
creation. Swimlane lanes are schema-qualified whenever the lake keeps
tables outside main, so bronze.dm and silver.dm no longer share a
lane; a lake with everything in main keeps bare names.
The Clinical Trial Data Lake article is rewritten, and it is now a
pkgdown-only article (vignettes/articles/), so it can use packages a
CRAN vignette cannot declare: dplyneage for column lineage, haven for
the XPT transfer it loads, admiral and pharmaversesdtm for the
derivations. The lake is laid out as bronze, silver, gold, and results
schemas. SDTM arrives as an XPT transfer, so the silver blank-to-NA
step has blanks to convert; pharmaversesdtm stores missing values as
NA already, which made the old version of that step a no-op. ADSL and
ADAE are derived the way the admiral 1.4 templates do it, with
imputation flags, Y/N population flags, derive_var_trtemfl(), and
occurrence flags from derive_var_extreme_flag() under
restrict_derivation(). A treatment-emergent adverse event summary is
stored in the lake with its lineage drawn, and a correction flows from
SDTM through ADaM to that summary in one transaction. The ADPC dataset
and the section that stored define.xml and JSON blobs in table cells are
gone, and the wording on audit trails names the elements the FDA 2024
Q&A lists rather than claiming Part 11 compliance. The eight packages
only that vignette used leave Suggests (admiral, pharmaversesdtm,
DiagrammeR, jsonlite, lubridate, purrr, stringr, tidyr).
The ducklake DuckDB extension no longer needs an install step. The code
never required one: attach_ducklake() has always installed it on first
use, after a message naming the directory. The README and the articles
now say so, and install_ducklake() is documented as the way to download
the extension ahead of time (container images, CI, machines that are
offline when the lake is attached). A first-use install now also clears
the answer ducklake_extension_available() cached, so it reports TRUE
afterwards. The hint that follows an install into a temporary directory
did not show on macOS for a first-use install, because the directory
does not exist until INSTALL creates it and the check only recognized
existing paths; it shows now. No load-time check was added: the package
neither downloads nor starts DuckDB when it is loaded.
The minimum duckdb version is now 1.5.5, the release that settled where
the R package keeps downloaded extensions: ~/.duckdb when it exists
(duckdb offers to create it the first time it connects in an interactive
session), DUCKDB_R_HOME or the duckdb.home option when set, otherwise
a per-session temporary directory (?duckdb::duckdb_storage).
ducklake_extension_available() now opens its probe on the directory
duckdb would resolve for a new connection, so it finds an existing
~/.duckdb without triggering that offer, and the hint printed after an
install into a temporary directory describes these options.
with_transaction() and commit_transaction() confirm a commit with one
line naming the snapshot it created, with the author and commit message
when they were given: "Committed snapshot 3 (Data Engineer): Add the
Motor Trend car data". A transaction that changed nothing says so, and
begin_transaction() is silent. The two lines they replace ("Transaction
started.", "Transaction committed.") said nothing about the snapshot; the
new one gives the version number that time travel uses.
The Getting Started article (vignette("ducklake")) is rewritten as one
short session: install, attach a lake, add a table, read it back, change
it, see its history, and detach, with the reasons to wrap changes in
with_transaction() explained along the way. Its recipes move to two new
articles. "Loading Data" (vignette("loading-data")) collects the ways
data gets into a lake, from data frames and files to registering Parquet
in place and migrating from DuckDB or Iceberg. "Views, Comments, and
Labels" (vignette("views-comments-labels")) covers the query logic and
documentation that live in the catalog, and its labels example uses
labelled::set_variable_labels(), so labelled is now a suggested
package.
New article "Choosing a Deployment" (vignette("deployment")): what a
lake looks like on disk, which catalog backend fits one person, a team on
a shared drive, or many users on object storage, where data can go and
the Windows limits, how reader and writer roles map onto catalog grants
and storage permissions, the day-one settings (extension persistence,
schemas, retention policy, commit messages), and upgrading.
New get_ducklake_info() describes an attached lake in one row: backend,
catalog, data path, DuckLake format version, extension version,
encryption, and current snapshot.
New set_column_not_null() sets or drops a NOT NULL constraint, the
one constraint DuckLake supports.
attach_ducklake() creates a local lake_path (and the directory of a
local catalog file) when it does not exist and creating the lake is
allowed, instead of failing with DuckDB's "Cannot open file" error.
?set_ducklake_option lists every option DuckLake 1.0 persists, with its
default and what it controls.
New cookbook recipes migrate an existing DuckDB database into a lake with
COPY FROM DATABASE and exchange tables with an Iceberg catalog; the
time-travel vignette documents the rowid and snapshot_id hidden
columns.
backup_ducklake() copies the catalog with DuckDB's COPY FROM DATABASE
while the lake stays attached, a consistent snapshot taken inside one
transaction, instead of shutting the connection down to release file
locks and copying the file. Nothing is detached any more: other attached
lakes, in-memory secrets, and a connection registered with
set_ducklake_connection() (whose locks the old approach could not
release, leaving a 0-byte catalog) are left as they are. A SQLite catalog
is copied into a SQLite file. Restoring a backup is documented with
create = FALSE, so a mistyped path is an error rather than a new lake.
New package option ducklake.verbose: set it to FALSE to silence the
confirmations the package emits after each operation ("Transaction
committed.", "Added column ..."). Warnings, errors, and notices about
extension downloads stay on. The messages carry the condition class
ducklake_message.
Schema-qualified table names work end to end. Every function that takes
a table name accepts "schema.table": get_ducklake_table() hands
dbplyr a proper table path for it (the duckdb driver's tbl() turned a
dotted name into raw SQL that rows_*() could not write to), and the
metadata readers (get_table_comments(), get_table_partitions(),
get_table_sorting(), get_table_info(), list_table_snapshots(),
get_table_changes(), list_ducklake_files()) resolve the schema
instead of matching the bare name. Functions with a schema_name
argument take the schema from either place. The readers gain a
schema_name column. New create_schema() and drop_schema() manage
schemas, the natural home for medallion layers; the README example now
keeps bronze, silver, and gold in schemas of their own.
attach_ducklake() gains create (FALSE opens an existing lake and
errors on a wrong path or name instead of creating a new, empty lake)
and metadata_schema (several lakes in one PostgreSQL database, each in
its own schema).
New set_ducklake_retry() sets how DuckLake retries a transaction that
races with another writer, and the transactions vignette explains which
concurrent changes conflict and which are retried.
create_table() and replace_table() now run dplyr pipelines inside
DuckDB. A lazy table on the package's connection is written with
CREATE TABLE ... AS; a replacement is materialized in DuckDB's
temporary storage first (the query may read the table it replaces) and
the table is rebuilt from it. Rows no longer pass through R, so silver
and gold layers derived from large bronze tables cost DuckDB memory, not
R memory. Column comments follow the data: each output column keeps the
comment of the same-named column in the tables the query reads, so
variable labels survive a pipeline as they did through the old collect
path. Data frames, and lazy tables on other connections, load as before.
replace_table() and restore_table_version() now carry a table's
metadata over to the rewritten table: the table comment, column comments
(and so variable labels), partition keys, sort order, and table-scoped
options. Both go through DROP + CREATE, which gives the table a new id,
and DuckLake keeps all of these against the id, so they were silently
lost before. Partition and sort keys are set before the rows are written,
so the rewrite itself lands partitioned and sorted. Table-scoped options
are re-set right after the rewrite commits (DuckLake cannot set options
on a table created in the open transaction); inside a transaction you
opened yourself they cannot be re-set, and a warning lists the calls to
make after committing. New get_table_sorting() reads a table's sort
keys from the catalog, the counterpart of get_table_partitions().
The set_table_partitioning() recipe for re-partitioning existing data
(set the keys, then rewrite with replace_table()) now works; before,
the rewrite dropped the keys it was meant to apply.
checkpoint_ducklake() no longer claims to expire old snapshots and
reclaim their files on its own. A checkpoint does so only when the lake
carries a retention policy (expire_older_than and delete_older_than,
set with set_ducklake_option()); without one it flushes and compacts
but keeps every snapshot and every file. The storage and data-inlining
vignettes and the cookbook now show the policy.
set_snapshot_metadata() fills in only empty fields by default and stops
when a supplied field already has a value; pass overwrite = TRUE to
replace one. It writes to the catalog outside DuckLake's transaction
model, and an overwrite leaves no trace of the previous value, so the
audited path is metadata set at commit time with with_transaction() or
commit_transaction(). The documentation now says so plainly.
The minimum duckdb version is now 1.5.2, the release that ships DuckLake
1.0. The extension built for DuckDB 1.5.1 writes the earlier 0.4 catalog
format, so a lake created there needs a one-time migration after the
upgrade: attach_ducklake() gains automatic_migration = TRUE for that
(DuckLake's AUTOMATIC_MIGRATION option).
Where the ducklake extension is installed depends on the duckdb R
package: from 1.5.2 on it is a per-session temporary directory unless
DUCKDB_R_HOME (or the duckdb.home option, or an existing ~/.duckdb)
points somewhere durable, so "install once per machine" needs that
setting. install_ducklake() and the automatic extension installs now
report the directory they used and say when it is temporary; the README,
ducklake_extension_available(), and
create_storage_secret(persistent = TRUE) describe the setting.
The clinical trial vignette's silver layer called
admiral::convert_blanks_to_na() on a lazy lake table, which leaves the
table untouched, so the cleaning it described never ran. The conversion
now runs inside DuckDB through a small dbplyr helper. (The pharmaverse
test data already stores missing values as NA, so the vignette's
output does not change; XPT exports do carry blanks.)
Vignettes follow the package's own guidance on how to change a table:
derived columns are declared with add_table_column() and filled with
ducklake_exec(), corrections use rows_update() and rows_delete(),
and replace_table() is kept for bulk rewrites.
get_ducklake_table_asof() and get_ducklake_table_version() return the
same tbl_ducklake class as get_ducklake_table(), so collect()
restores stored column labels on time-travel reads too.
list_table_snapshots(table_name) matches snapshots by parsing the
change map rather than by a regular expression over its printed form,
and resolves table ids within the table's schema.
Functions that read a named lake's metadata (get_metadata_table(),
list_table_snapshots(), get_table_partitions(),
get_table_comments(), plot_snapshots()) resolve the catalog backend
from that lake rather than from the current database, so a PostgreSQL or
MySQL lake that is attached but not current is qualified correctly.
ducklake_exec(.quiet = FALSE) emits its SQL trace as messages instead
of printing it, and no longer prints the lazy table itself, which ran a
preview query. show_ducklake_query() prints through cli.
The documentation notes that the aws and azure DuckDB extensions are not
available on Windows for the duckdb R package, so
create_storage_secret(provider = "credential_chain") and Azure secrets
do not work there. The README now says Quack needs duckdb 1.5.3 or
newer, not 1.5.4.
add_table_column() now works with logical, Date, and POSIXct defaults.
DuckLake accepts only plain constants in a DEFAULT clause and rejected
the typed literals the package rendered (TRUE, DATE '...',
TIMESTAMP '...') as "non-literal"; such values are now passed as
quoted strings, which DuckLake converts to the column type.
A failed schema evolution statement no longer leaves the shared connection in an aborted transaction. Some DuckLake DDL failures do that even in autocommit mode, so that every later statement failed with "Current transaction is aborted"; the wrappers now roll back before re-raising the error when they did not inherit a transaction.
The unreferenced and broken inst/examples/with_transaction_demo.R was
removed; the with_transaction() examples cover it.
New meta_encryption_key argument in attach_ducklake() encrypts the
DuckDB catalog database file itself with AES-256-GCM (DuckLake forwards
META_-prefixed options to the metadata catalog). The key is set when
the catalog is created and required on every later attach. The catalog
is where encrypted = TRUE stores its Parquet keys, so encrypting it
closes that loop; pass askpass::askpass() as the value to be prompted
instead of writing the key in code (#46, suggested by @frankpopham).
The "duckdb" backend now honors catalog_connection_string as the path
for its catalog file, so a single-writer lake can keep the catalog on
local disk while lake_path points at object storage. The documentation
already promised this argument worked for the duckdb backend; now it does
(#44).
attach_ducklake() stops early when the duckdb backend would create its
catalog file on object storage, which DuckDB cannot write and which
previously surfaced as a confusing IO error. The message points to
catalog_connection_string, to read_only = TRUE (attaching an existing
remote catalog stays supported), and to the other backends. httpfs is
now loaded up front whenever a remote path is involved instead of relying
on DuckDB's mid-statement autoload.
The create_storage_secret() example and the storage vignette attached
an S3 lake with the default backend and no catalog path, which would put
the catalog file itself on S3. Both now show the split layout, and
backup_ducklake() finds a catalog that lives outside lake_path.
create_storage_secret(provider = "credential_chain") now loads DuckDB's
aws extension itself for "s3", "gcs", and "r2" secrets, installing
it on first use. The provider lives in that extension, and DuckDB's
automatic mid-statement install of it could fail, leaving CREATE SECRET
erroring with "Install it first" (#43).
First CRAN release.
New ducklake_extension_available() reports whether the ducklake DuckDB
extension is installed and loadable. It probes with automatic extension
installation switched off, so it never downloads anything, and it is what
the package's own examples, tests, and vignettes gate on.
Every example that can run against a temporary lake now does, guarded by
@examplesIf ducklake_extension_available(). Only the cases that need
outside infrastructure stay unevaluated: Quack servers, PostgreSQL and
MySQL catalogs, and remote URLs.
backup_ducklake() no longer needs the fs package. It was the one place in
the package that called a suggested dependency unconditionally, so backups
failed for anyone without fs installed.
The package now tells you before it installs a DuckDB extension for you.
attach_ducklake() and the functions that load httpfs, quack, or a
backend extension used to download into the extension cache in your home
directory without a word.
Fixed the with_transaction() rollback example, which reused a table name
that already existed and so failed before reaching the error it meant to
demonstrate.
New rows_upsert() completes the dplyr rows_* family: rows that match
on the key columns are updated and the rest are inserted, as one atomic
MERGE INTO statement (DuckLake tables have no primary keys, so the
ON CONFLICT upsert other databases use does not apply). One call is one
snapshot, with the updates and inserts recorded individually in the
change feed.
New merge_into() exposes the full SQL MERGE surface for the cases
rows_upsert() cannot express: conditional matched and not-matched
clauses, deleting matched rows, and delete_missing = TRUE to drop
target rows absent from the source (a staging-table sync). DuckLake
currently allows one update/delete action per MERGE statement, so syncs
that need both run as MERGE plus DELETE inside a single transaction and
snapshot.
New table documentation family, built for labelled-data workflows:
set_table_comment() and set_column_comments() store descriptions in
the lake's catalog (COMMENT ON), and get_table_comments() reads
them back as a tidy data frame. create_table() gains a labels
argument (default TRUE) that stores haven/labelled variable labels as
column comments at load time, and collect() on a lake table
reattaches stored comments as label attributes -- so gtsummary, gt,
and other label-aware tools work as if the data never left R, and every
other client of the lake can read the same documentation.
New create_view() stores a dplyr pipeline as a SQL view in the lake:
shared business logic that reads current data and that every client --
R, Python, or plain SQL -- sees identically. drop_view() removes one.
SQL macros stay unwrapped on purpose (a macro body is raw SQL and
unreachable from dplyr pipelines); the cookbook shows the
DBI::dbExecute() escape hatch.
New list_ducklake_tables() answers "what is in this lake?" with a
tidy frame of tables and views.
New schema evolution family: add_table_column() (with optional
default, which DuckLake applies to existing rows too),
drop_table_column(), rename_table_column(), set_column_type()
(widening promotions only, with the add-copy-drop-rename recipe in the
error when a change would narrow), and rename_ducklake_table(). All
are metadata-only ALTER TABLE operations: no data files are rewritten,
and earlier snapshots keep the earlier schema. Until now schema changes
went through replace_table(), which collects the whole table into R;
its documentation now points here, and it remains the tool for bulk
data transformations. Derived columns combine the two styles:
add_table_column() then a mutate() pipeline through
ducklake_exec() fills the column with an in-database UPDATE.
New plotting functions, all requiring the suggested ggplot2 package and
shown in action in a new "Visualizing Your Lake" vignette:
plot_snapshots() draws a table's snapshot history as a commit-log
timeline (snapshots in order, with authors, commit messages, and inline
markers for long idle gaps) or, without a table name, the whole lake as a
swimlane with one row per table; plot_table_changes() draws the rows
each snapshot inserted, updated, and deleted as diverging bars; and
plot_table_files() draws each table's Parquet file count and size on
disk.
New get_table_info() returns per-table file statistics (data and delete
file counts and sizes) from the DuckLake catalog, wrapping DuckLake's
ducklake_table_info() function.
New add_data_files() registers existing Parquet files with a table
without copying or rewriting them -- the migration path for data that is
already in Parquet. A vector of files is registered atomically in one
snapshot, and create = TRUE can bootstrap a new target table directly
from the Parquet schema. list_ducklake_files() shows the files backing a
table, optionally as of a past snapshot.
New sorted-table support: set_table_sorting() and
reset_table_sorting() manage a table's declared sort order, the
complement to partitioning for pruning on high-cardinality columns.
New set_ducklake_option() and get_ducklake_options() expose DuckLake's
full option system (parquet_compression, target_file_size,
sort_on_insert, require_commit_message, ...) at lake, schema, or
table scope. set_inlining_row_limit() now builds on the same internals.
New create_storage_secret() stores object-storage credentials (S3, GCS,
R2, Azure) via DuckDB's secrets manager, so a lake's lake_path can live
on cloud storage. backup_ducklake() now errors clearly for remote data
paths instead of failing partway through.
attach_ducklake() gains snapshot_version and snapshot_time
arguments to attach a lake pinned to a historical snapshot -- a frozen,
read-only view for reproducing past analyses.
replace_table() now runs its drop and create as one transaction, so a
failed create no longer leaves the table dropped. When the caller has
already opened a transaction, replace_table() defers to it as before.
set_snapshot_metadata() now validates ducklake_name like the rest of
the package, and resolves the catalog backend from that name instead of
the current database.
create_table() no longer leaves a temporary view registered on the
shared connection when the CREATE statement fails.
replace_table(.quiet = FALSE) reports progress via messages (cli) rather
than printing to the console, so it can be suppressed and captured like
the rest of the package's output.
The package now declares R (>= 4.1) explicitly (the tests and examples use the base pipe), and is prepared for CRAN: extension-dependent tests skip on CRAN, and vignettes evaluate only where the ducklake DuckDB extension can be loaded.
New targeted maintenance wrappers complement checkpoint_ducklake():
expire_snapshots() (with older_than, versions, and dry_run),
merge_adjacent_files(), cleanup_old_files(), delete_orphaned_files(),
and rewrite_data_files() (#16, suggested by @stefanlinner).
New partitioning support: set_table_partitioning() and
reset_table_partitioning() manage a table's partition keys (identity,
year/month/day/hour, and bucket transforms), and
get_table_partitions() lists the keys from the metadata catalog (#16,
suggested by @stefanlinner).
New get_table_changes() exposes DuckLake's data change feed: the exact
inserts, deletes, and update pre/post images between two snapshots, as a
lazy table that composes with dplyr verbs (#16, suggested by
@stefanlinner).
Timestamps passed to the new functions as POSIXct are converted to UTC before interpolation, matching how DuckLake records snapshot times.
get_ducklake_table_asof() and restore_table_version() now also convert
POSIXct timestamps to UTC. Previously they rendered local time, which
DuckLake reads as UTC, silently shifting the queried instant by the UTC
offset -- a bare Sys.time() looked hours in the past (or future) unless
the session's timezone was UTC. Timestamps taken from
list_table_snapshots()$snapshot_time are unaffected.
attach_ducklake() now collapses duplicate slashes in lake_path (remote
URIs are untouched). DuckLake compares file paths as exact strings, so a
doubled slash -- which R's tempdir() produces on macOS -- made
delete_orphaned_files() treat every live data file as orphaned.
The dplyr-to-DuckLake translation behind ducklake_exec() and
show_ducklake_query() is now built from dbplyr's structured query
objects (dbplyr::sql_build()) instead of pattern-matching rendered SQL
text. Classification no longer depends on what the SQL happens to look
like, which fixes several latent bugs:
get_ducklake_table("staging") |> filter(...) |> ducklake_exec("target")) was translated into a DELETE
on the target table; it now appends the matching rows, as intended.filter(note != "WHERE is it")) were refused as "too complex"; they
now translate fine.mutate() that adds a new column is refused upfront with a pointer
to replace_table(), instead of failing with a database binder error.INSERT translations now list columns explicitly, so appends from
another table match columns by name rather than by position, and joined
or unioned sources can be appended in one step.
Pipelines that compile to a subquery over the target table (grouped
filters, filtering on a just-mutated column), and clauses with no
in-place equivalent (arrange(), head(), distinct()), are detected
structurally and refused with a clear message rather than mistranslated.
dbplyr (>= 2.5.0) is now required; both dbplyr 2.5.x and the select-list format introduced in dbplyr 2.6.0 are supported.
This release focuses on production hardiness: self-contained connection management, working detach/restore, SQL identifier safety, Quack remote access, and a documentation overhaul.
Added support for Quack, DuckDB's client-server protocol, which became a core extension in DuckDB 1.5.3 (#20, @JavOrraca). A DuckLake served by one DuckDB instance can now be queried and modified by other R sessions over the network. For concurrent access this is a lighter-weight option than a PostgreSQL or SQLite catalog, since the whole setup stays inside DuckDB and DuckLake.
attach_quack() connects to a remote Quack server and attaches it as a catalog in the current session.detach_quack() disconnects from a remote Quack server.install_quack() installs the Quack DuckDB extension.quack_query() runs a one-off query against a remote Quack server and returns a data.frame.quack_serve() serves the current session, including an attached DuckLake, to other clients over Quack.quack_stop() stops a running Quack server.attach_ducklake() gains an encrypted argument: pass encrypted = TRUE
to have DuckLake encrypt the Parquet files it writes (#18). Note that the
encryption keys are stored in the catalog database, so protect the catalog.
The httpfs extension is loaded automatically for encrypted lakes, since on
some platforms (notably Windows) DuckDB's built-in crypto module is
read-only.restore_table_version() now works. It previously generated a
RESTORE TABLE statement that does not exist in DuckLake and failed on
every call. It now recreates the table from a time-travel read inside a
transaction, recording the restore as a new snapshot so history is
preserved. It also gains author and commit_message arguments so the
restore snapshot carries full audit-trail metadata.get_ducklake_backend() gains a ducklake_name argument and tracks each
attached lake separately, so sessions with several lakes on different
catalog backends resolve backend-specific behaviour correctly.detach_ducklake() now actually detaches. Previously the DETACH ran
while the lake was still the session's current database, which DuckDB
refuses, and the error was silently swallowed -- the lake stayed attached.
The session now switches back to the connection's own catalog first.
Relatedly, restoring a backup to a new location requires
override_data_path = TRUE (as documented); the storage vignette example
has been corrected.DBI::dbQuoteIdentifier() and friends), so
names with spaces or quotes no longer produce malformed statements.rows_insert(), rows_update(), and rows_delete() now also dispatch as
S3 methods on tables returned by get_ducklake_table(). Previously, if
dplyr was loaded after ducklake, dplyr's generics masked ducklake's
wrappers and calls failed with conflict = "error" complaints; load order
no longer matters.rows_insert(), rows_update(), and rows_delete() now work inside
with_transaction(), so several row operations can be grouped into a
single snapshot with an author and commit message. Previously, passing a
local data frame made dbplyr copy it to a temporary table inside its own
transaction, which DuckDB rejects when one is already open. Local data
frames are now sent as inline queries (dbplyr::copy_inline()), which is
also faster for the small changesets these functions are designed for.backup_ducklake() backs up every schema directory, not just main.create_table() now converts factor columns to character (with a message)
instead of failing with "unsupported type ENUM" -- DuckLake does not
support DuckDB's ENUM type, which is what factors become.ducklake_exec() no longer uses
sink() (which could leak diverted output on error), and now refuses
queries with subqueries or multiple WHERE clauses instead of generating
incorrect SQL.ducklake_exec() no longer executes its statement twice. The internal
translation step also executed the SQL before ducklake_exec() ran it
again, so every call created two snapshots and non-idempotent updates
(e.g. v = v + 1) were applied twice.show_ducklake_query() is now a true preview: it previously executed
the translated statement against the lake while displaying it.ducklake_exec() now translates any mutate() into an UPDATE, not just
those that compile to CASE WHEN. Previously a simple transformation
like mutate(v = round(v, 1)) fell through to an INSERT of the table's
own rows, silently duplicating the table. Plain self-reads with nothing
to translate are now refused for the same reason, and UPDATE assignments
containing commas inside function calls are parsed correctly.list_table_snapshots(table_name) no longer misses snapshots created by
rows_insert(), rows_update(), and rows_delete(). DuckLake records
row-level changes against the table's numeric id rather than its name;
the filter now resolves and matches those ids, so the per-table audit
trail is complete. Filtered listings also number their rows from 1
instead of leaking the row positions of the unfiltered result.ducklake now creates and manages its own DuckDB connection instead of
reaching into duckplyr's unexported internals. This removes the package's
last ::: calls and the duckplyr dependency entirely.
set_ducklake_connection().set_ducklake_connection() (returning by popular demand, now safer):
point ducklake at any DuckDB connection you manage — for example one
shared with other DBI tools. Connections you supply are never closed by
ducklake; only its own automatically created connection is shut down by
detach_ducklake(shutdown = TRUE) and at session exit.This release aligns the package with the DuckLake v1.0 stable specification, which requires DuckDB v1.5.2+ (compatible with duckdb R package >= 1.5.1).
DuckDB version requirement bumped from 1.3.0 to 1.5.1 (duckdb R package)
/ 1.5.2 (DuckDB engine/CLI) to match DuckLake v1.0.
install_ducklake() now enforces this at the engine level.
commit_transaction() and with_transaction() now use the official
CALL ducklake.set_commit_message() API to set commit metadata
within the transaction before COMMIT, consistent with the v1.0
specification. set_snapshot_metadata() retroactively updates the
ducklake_snapshot_changes metadata table directly.
DuckLake now supports PostgreSQL, SQLite, and MySQL as catalog backends in addition to DuckDB (#15, @stefanlinner). This aligns with the DuckLake 1.0 specification and enables concurrent multi-client access when using PostgreSQL or SQLite.
attach_ducklake() gains backend, catalog_connection_string, read_only,
and override_data_path parameters for multi-backend support.install_ducklake() gains a backend parameter to pre-install backend
extensions (e.g., install_ducklake(backend = "postgres")).get_ducklake_backend() returns the active catalog backend type.detach_ducklake() gains a shutdown parameter. By default it now performs a
soft detach (SQL DETACH + USE memory;) instead of shutting down the
connection, allowing backend switching within a session.backup_ducklake() is now backend-aware: file-based backends (DuckDB, SQLite)
get catalog + data copied; PostgreSQL/MySQL get data only with guidance to use
pg_dump/mysqldump. Also fixes a pre-existing bug where catalog backups
were silently 0 bytes due to DuckDB holding file locks during file.copy().attach_ducklake() now requires lake_path (previously optional).set_ducklake_connection() has been removed. The package now exclusively uses
duckplyr's singleton DuckDB connection.detach_ducklake() no longer shuts down the DuckDB connection by default.
Pass shutdown = TRUE for the previous behaviour.get_metadata_table(),
time_travel.R, transactions.R) to handle PostgreSQL/MySQL backends that
don't use the .main. schema prefix.build_attach_sql(), ensure_extensions(),
shutdown_and_reset_singleton().Initial release of ducklake, an R package for versioned data lake infrastructure built on DuckDB and DuckLake.
create_table() - Create new tables in the data lakeget_ducklake_table() - Retrieve tables as tibblesreplace_table() - Replace entire table contents with versioningrows_insert() - Insert new rows with automatic versioningrows_update() - Update existing rows with audit trailrows_delete() - Delete rows while maintaining historywith_transaction() - Execute code blocks within transactionsbegin_transaction(), commit_transaction(), rollback_transaction() - Manual transaction controlget_ducklake_table_asof() - Query table state at specific timestampsget_ducklake_table_version() - Retrieve specific table versionslist_table_snapshots() - View complete version historyrestore_table_version() - Roll back to previous versionsget_metadata_table() - Access comprehensive metadataset_snapshot_metadata() - Add author, commit messages, and tagsinstall_ducklake() - Install/update DuckLake extensionattach_ducklake() - Initialize data lake connectionsdetach_ducklake() - Clean up connectionsget_ducklake_connection() - Retrieve the active DuckDB connectionducklake_exec() - Execute SQL with automatic assignment handlingshow_ducklake_query() - Preview translated SQL queriesextract_assignments_from_sql() - Parse SQL table assignmentsbackup_ducklake() - Create incremental backupsThis package is currently in experimental status. The API may change as we gather feedback from early users, but core functionality is stable and ready for pilot projects.