Data Inlining

How it works

Every insert, delete, or update that affects fewer rows than the inlining threshold is stored in a lightweight table inside the catalog (DuckDB, SQLite or PostgreSQL) instead of writing a Parquet file to the data path. At query time DuckLake seamlessly combines inlined and Parquet data, so results are always correct regardless of where data lives.

When you’re ready, you can flush the inlined data to consolidated Parquet files with a single call.

Setup

attach_ducklake("sensor_lake", lake_path = vignette_temp_dir)

Small writes are automatically inlined

Start with a small sensor-readings table. The data has only 3 rows, well below the default inlining threshold of 10, so DuckLake stores it directly in the catalog instead of writing a Parquet file.

readings <- data.frame(
  sensor_id = 1:3,
  temperature = c(21.5, 22.1, 21.8),
  ts = as.POSIXct(
    c("2025-03-27 10:00:00", "2025-03-27 10:00:10", "2025-03-27 10:00:20"),
    tz = "UTC"
  )
)

with_transaction(
  create_table(readings, "readings"),
  author = "Sensor Team",
  commit_message = "Initial sensor readings"
)

No Parquet files were created. All of the data is inlined in the catalog:

conn <- get_ducklake_connection()
DBI::dbGetQuery(
  conn,
  sprintf("SELECT count(*) AS parquet_files FROM glob('%s/**/*.parquet');", vignette_temp_dir)
)

Yet queries return all rows seamlessly:

get_ducklake_table("readings") |> collect()

Incremental updates stay inlined

Small modifications also stay in the catalog when the changeset is below the threshold. Adding a derived column is a metadata change (add_table_column()), and filling it is an in-database UPDATE:

with_transaction({
  add_table_column("readings", "temp_calibrated", "DOUBLE")
  get_ducklake_table("readings") |>
    mutate(temp_calibrated = temperature - 0.3) |>
    ducklake_exec()
},
  author = "Sensor Team",
  commit_message = "Add calibrated temperature"
)

get_ducklake_table("readings") |> collect()

Removing rows

A small delete is inlined too: DuckLake records it in the catalog rather than writing a delete file.

# Remove sensor 2's reading
with_transaction(
  rows_delete(
    get_ducklake_table("readings"),
    data.frame(sensor_id = 2L),
    by = "sensor_id"
  ),
  author = "Sensor Team",
  commit_message = "Remove faulty sensor 2 reading"
)

get_ducklake_table("readings") |> collect()

Large writes bypass inlining automatically

When a write exceeds the threshold, DuckLake writes directly to Parquet, with no configuration needed:

# 50 rows, well above the default threshold of 10
large_batch <- data.frame(
  sensor_id = 101:150,
  temperature = rnorm(50, mean = 22, sd = 1),
  ts = seq(as.POSIXct("2025-03-28 00:00:00", tz = "UTC"), by = "10 sec", length.out = 50),
  temp_calibrated = rnorm(50, mean = 21.7, sd = 1)
)

with_transaction(
  create_table(large_batch, "readings_bulk"),
  author = "Sensor Team",
  commit_message = "Bulk sensor upload"
)

# This table has a Parquet file
DBI::dbGetQuery(
  conn,
  sprintf("SELECT count(*) AS parquet_files FROM glob('%s/**/*.parquet');", vignette_temp_dir)
)

Configuring the inlining threshold

# Increase the threshold for a streaming workload
set_inlining_row_limit(50)
get_inlining_row_limit()

attach_ducklake(
  "streaming_lake",
  lake_path = "/data/streaming",
  data_inlining_row_limit = 100
)

set_inlining_row_limit(100, table_name = "readings")

set_inlining_row_limit(0)
# All writes now go directly to Parquet, even single rows

Flushing inlined data to Parquet

When inlined data accumulates, flush it to consolidated Parquet files:

flush_result <- flush_inlined_data()
flush_result

Data remains correct after flushing:

get_ducklake_table("readings") |> collect()

flush_inlined_data(table_name = "readings")

Checkpoint: one-stop maintenance

checkpoint_ducklake() runs DuckLake’s maintenance operations in sequence: it flushes inlined data, merges small files, and rewrites heavily deleted files. It also expires old snapshots and deletes the files they released, but only once the lake carries a retention policy (the expire_older_than and delete_older_than options); see vignette("storage-and-backups").

checkpoint_ducklake()

Run checkpoints periodically (e.g., after a batch of streaming inserts) to keep query performance optimal and consolidate inlined data.

Note for Windows users: with a DuckDB-file catalog, the file-cleanup step of CHECKPOINT can fail because Windows does not allow the catalog file to be opened a second time while the lake is attached (a current DuckDB limitation). flush_inlined_data() is unaffected; on Windows, prefer it for routine use and run full checkpoints from a fresh session, or use a PostgreSQL/SQLite catalog.

Time travel with inlined data

Inlined data fully supports DuckLake’s time-travel capabilities. Each inlined insert or delete creates a snapshot, just like a regular write:

# Check available snapshots
snapshots <- list_table_snapshots("readings")
snapshots

# Query an earlier version
if (nrow(snapshots) > 0) {
  first_version <- snapshots$snapshot_id[1]
  get_ducklake_table_version("readings", first_version) |> collect()
}

When to use inlining

What inlining does not change

Data inlining is a storage optimisation, not a change to DuckLake’s data integrity model. Whether data is inlined or written to Parquet: - ACID transactions are fully enforced. - Every modification creates a snapshot with author, timestamp, and commit message metadata. - Time-travel queries work identically. - The audit trail is complete and unaffected.

The only difference is where the data physically resides until it is flushed: in the catalog database (inlined) or in Parquet files on the data path.

Summary

• Inlining is on by default (threshold = 10 rows). No setup required.
• Small inserts, deletes, and updates are stored in the catalog.
• Use flush_inlined_data() or checkpoint_ducklake() when ready to materialise to Parquet.
• Full time-travel support is preserved for inlined data.
• Adjust the threshold with set_inlining_row_limit() or at attach time via data_inlining_row_limit.

For more details, see the DuckLake data inlining documentation.