Quack Remote Access

Introduction

Most of the time a DuckLake lives in one place and one person works with it. But data analysis is usually a team activity: a few statisticians share the same study data, an analytics group works from one set of cleaned tables, several people review the same numbers before a deliverable goes out. The usual answer is to copy files around (a CSV here, an XPT there) and hope everyone is looking at the same version.

Quack offers a different answer. It is a protocol, a core DuckDB extension stable since DuckDB 2.0, that lets one DuckDB instance act as a server while other DuckDB instances connect to it as clients. A DuckLake held by the server can then be read and updated by other R sessions over the network, with the same snapshots and time travel you get locally. Several people can write at the same time without locking each other out.

This vignette shows the two halves of that picture: connecting to a lake someone else is serving, and serving a lake of your own. The audience here is the everyday R user, so the examples stay close to ordinary analysis work and do not assume any server administration experience.

The code below is shown with its expected output but is not run when the vignette is built, because Quack needs a live server.

Installing the extension

Quack is a core extension, so DuckDB loads it automatically the first time it is used. On a machine that has no internet access at query time you can install it ahead of time:

install_quack()
#> Installed quack extension.
#> Loaded quack extension.

Connecting to a shared lake

This is the common case. A colleague, or a shared workstation, runs a Quack server that holds the team’s DuckLake. You connect to it from your own R session and pull the data you need.

quack_query() sends a single SQL query to the server and returns the result as a data frame. The lake is named by its catalog on the server, which is the name it was attached under (here, trial):

adsl <- quack_query(
  "quack:trial-server.example.org",
  "SELECT * FROM trial.adsl",
  token = "super_secret"
)

adsl
#>   USUBJID SAFFL AGE
#> 1  01-001     Y  45
#> 2  01-002     N  52

The address starts with quack:. A bare host name works too and the scheme is added for you. The default port is 9494; include a different one as quack:host:port. The token is whatever password the server was started with.

Once a query comes back you have an ordinary data frame, so the rest of your analysis is the dplyr and base R you already use:

adsl |>
  group_by(SAFFL) |>
  summarise(n = n(), mean_age = mean(AGE))
#> # A tibble: 2 x 3
#>   SAFFL     n mean_age
#>   <chr> <int>    <dbl>
#> 1 N         1       52
#> 2 Y         1       45

For large tables it is faster to let the server do the filtering and summarising and send back only what you need, rather than pulling every row across the network:

quack_query(
  "quack:trial-server.example.org",
  "SELECT USUBJID, AGE FROM trial.adsl WHERE SAFFL = 'Y' AND AGE >= 18",
  token = "super_secret"
)
#>   USUBJID AGE
#> 1  01-001  45

Because Quack supports concurrent writers, you can also add data to the shared lake. The query runs on the server, so it refers to the lake by its catalog name and returns the number of rows affected:

quack_query(
  "quack:trial-server.example.org",
  "INSERT INTO trial.adsl VALUES ('01-099', 'Y', 70)",
  token = "super_secret"
)
#>   Count
#> 1     1

attach_quack("warehouse", "quack:data.example.org", token = "super_secret")

get_ducklake_table("warehouse.sales") |>
  filter(region == "EMEA") |>
  collect()

detach_quack("warehouse")

Serving a lake of your own

The other half of the picture is publishing a lake so others can connect to it. Say you have built a small medallion lake and a few colleagues want to read from it. You attach the lake as usual, then start a server.

attach_ducklake("trial", lake_path = "~/lakes/trial")

with_transaction(
  create_table(mtcars, "vehicles"),
  author = "You",
  commit_message = "Seed the shared lake"
)

install_quack()
quack_serve(token = "super_secret")
#> Quack server listening on "quack:localhost".

The server runs in the background of the DuckDB instance, so your R session stays free for other work. Anyone who can reach the address can now connect with the client code from the previous section, pointing at your machine and using the same token.

Choose the address to match who should reach the server. "quack:localhost" (the default) stays on your own machine, which is useful for trying things out. To accept connections from other machines, listen on a host name or address they can route to.

When you are finished, stop the server:

quack_stop()
#> Quack server on "quack:localhost" stopped.

Serving the catalog instead of the tables

The server above serves a lake’s tables: clients query them, and only the server touches the files. DuckLake 1.1 adds a second arrangement, experimental for now, where a Quack server holds a lake’s catalog open and every client attaches the lake through it. The catalog is a plain DuckDB file, which on its own admits one process at a time; served, it takes any number of sessions, each of them a full client of the lake that reads and writes the Parquet files itself. That makes the arrangement a fit for a team on a shared drive that would rather not run PostgreSQL.

The server is a DuckDB process that opens the catalog file and serves it. Any DuckDB client will do; from R:

con <- DBI::dbConnect(duckdb::duckdb(), dbdir = "/shared/lakes/trial/catalog.duckdb")
DBI::dbExecute(con, "INSTALL ducklake; LOAD ducklake; INSTALL httpfs; LOAD httpfs; INSTALL quack; LOAD quack")
DBI::dbGetQuery(con, "CALL quack_serve('quack:catalog.example.org', token = 'super_secret', allow_other_hostname = true)")

Each client attaches the lake with backend = "quack", the server’s address as the catalog connection string, and the data path as every client reaches it:

attach_ducklake(
  "trial",
  backend = "quack",
  catalog_connection_string = "quack:catalog.example.org",
  quack_token = "super_secret",
  lake_path = "/shared/lakes/trial/data"
)

with_transaction(
  create_table(mtcars, "vehicles"),
  author = "You",
  commit_message = "Seed the shared lake"
)
list_table_snapshots("vehicles")

From here on nothing is different: snapshots, time travel, comments, and the maintenance functions all go through the served catalog, and commits run on the server. Back up the catalog by stopping the server and copying its file; backup_ducklake() copies the data files only.

Quack or a shared catalog?

ducklake already supports PostgreSQL and SQLite catalogs for concurrent access (see ?attach_ducklake and the package NEWS). Quack covers a similar need from two angles, and the right choice depends on where your data lives and who can reach it.

A common pattern is to use two of them: a PostgreSQL catalog so the lake itself supports many writers, and a Quack server in front so analysts who cannot mount the storage can still connect.

Caveats

Quack reached its stable release in DuckDB 2.0, the version ducklake requires, so the server and every client speak the same settled protocol. For very high write volumes a purpose built database is still the better tool, but for sharing a study lake across a small team Quack is a light way to get everyone onto the same versioned data.

Learn more

• Quack overview, the DuckDB documentation.
• ?quack_serve, ?attach_quack, and ?quack_query for the function reference.
• The Storage and Backup Management vignette for how a DuckLake is laid out on disk.