Watcher backends

Honker watches for committed database changes and wakes consumers so they can re-read state. By default, one background thread per database asks SQLite “did anything change?” every millisecond. Several commits inside one polling interval can coalesce into one wake.

Two opt-in alternatives exist for advanced users. In the Rust-backed Python and Node packages, both are experimental and source-build-only. The JVM implements equivalent backends in Java and ships them in its artifact; configuration entry points vary in the other bindings. See Binding support before selecting one outside the Python, Node, or Rust APIs shown here.

This page is for people who:

• Run many honker’d databases on one host and care about idle CPU.
• Want to test event-driven wake on their platform before we ship it broadly.
• Need to understand what happens when the database file is swapped under the watcher (litestream restore, NFS remount, atomic backup swap).

If none of those apply, the default works and you can skip the rest.

TL;DR

db = honker.open("app.db")                              # polling, the default
db = honker.open("app.db", watcher_backend="kernel")    # experimental: OS file events
db = honker.open("app.db", watcher_backend="shm")       # experimental: mmap WAL index, WAL only

const db = honker.open('app.db');
const dbK = honker.open('app.db', undefined, 'kernel');
const dbS = honker.open('app.db', undefined, 'shm');

use honker_core::{SharedUpdateWatcher, WatcherBackend, WatcherConfig};

let polling = SharedUpdateWatcher::new(db_path.clone());
let kernel = SharedUpdateWatcher::new_with_config(
    db_path.clone(),
    WatcherConfig { backend: WatcherBackend::KernelWatch },
);
let shm = SharedUpdateWatcher::new_with_config(
    db_path,
    WatcherConfig { backend: WatcherBackend::ShmFastPath },
);

In the Rust watcher, the kernel and shm backends are gated behind Cargo features (kernel-watcher, shm-fast-path). The published Python wheel and Node native package build polling-only. To use the experimental Rust backends today, build from source with the feature on — see Source builds below. These Cargo flags do not control the JVM’s Java implementations.

At a glance

backend	wake source	spurious wakes	delayed/coalesced wakes	journal mode	Python/Node prebuilt package
polling (default)	PRAGMA data_version every 1 ms	never	possible between polls	any	yes
kernel	OS file events on db file + parent dir + -wal/-shm/-journal	possible	possible (if the OS drops events)	any	no — source only
shm	mmap read of SQLite’s WAL index every 1 ms	never	possible between polls (mmap inode-change → panic)	WAL only	no — source only

When to choose which

• Stay on polling unless you have a specific reason to switch. It does not depend on an OS event queue, and consumers re-read state after a wake; multiple commits between checks can still coalesce. It works in every journal mode and is the backend in the prebuilt Python and Node packages. The CPU cost — about 2.0 ms per second of busy work on the benchmark machine — is rounding error on any modern machine.
• Try kernel if you’re running many honker’d databases on one host and want idle CPU near zero, or if you’re on Linux and want wakes faster than 1 ms. The catch: some wakes can be lost if the OS drops events under load. Consumers re-read state on every wake, so a lost wake means delayed work, not lost work.
• Try shm if you’re on a high-QPS WAL workload and the per-check cost of the 1 ms poll matters (rare). The cadence stays at 1 ms like polling, but each check drops from a ~2.0 µs PRAGMA data_version read to a memory load. The catch: it reads SQLite’s WAL index format directly, so a future SQLite version that changes that format would break it.

Rule of thumb: if you can’t say in one sentence what you’d gain by switching, stay on polling.

The backends in detail

Polling (default)

One background thread per database. Every millisecond, it runs PRAGMA data_version on a read-only connection. That counter changes after commits by another connection, in any journal mode. Around 2.0 microseconds per check on the standalone Rust benchmark machine. One thread regardless of subscriber count.

The polling backend keeps its connection through transient SQLITE_BUSY / SQLITE_LOCKED errors. Those happen when a writer is mid-commit on a rollback-journal mode (DELETE / TRUNCATE / PERSIST) and the watcher’s PRAGMA can’t get a lock. Instead of dropping the connection and reconnecting (which would silently re-baseline and miss wakes), it just retries the next millisecond. So non-WAL workloads get reliable wake delivery even during writer-locked windows.

Kernel

Wakes on file events from the OS — inotify on Linux, kqueue on macOS, ReadDirectoryChangesW on Windows. Watches the database file, the parent directory, and -wal/-shm/-journal sidecars when they exist.

Good:

• Lower idle CPU. No 1 ms tick — wakes only when something changes on disk.
• Faster wakes on Linux/macOS for WAL workloads (typically under a millisecond).

Bad:

• Spurious wakes. Any file change in the directory wakes consumers. They re-read state from SQLite on every wake anyway, so this is wasted work, not incorrect.
• Missed wakes. If the OS drops or coalesces an event, that commit’s wake doesn’t fire. The 5 second idle_poll_s paranoia poll is the only backstop.
• macOS kqueue can’t reliably catch in-place writes to existing files for non-WAL journal modes (TRUNCATE / PERSIST in particular). WAL works. Use the kernel backend with WAL.

Shm fast path

Reads iChange at byte 8 of SQLite’s WAL index header (-shm file) via mmap. SQLite increments iChange on every WAL write. The watcher compares the current value to the previous one every millisecond and fires a wake on change.

Good:

• Cheaper checks. A memory load instead of a SQLite query — the per-check cost drops from ~2.0 µs to sub-microsecond (the poll cadence is still 1 ms).
• Same low cost under heavy load.

Bad:

• WAL only. No -shm file exists in DELETE / TRUNCATE / PERSIST. If you ask for shm in those modes, honker.open() raises immediately.
• Reads SQLite’s WAL index format directly. A future SQLite version that changes that format would break it. We have an equivalence test against current SQLite to catch this.

Database file swapped under the watcher

Every backend records the database file’s identity at startup and re-checks it every 100 ms. If the identity changes — atomic rename, litestream restore, NFS volume remount, VACUUM — the watcher panics with a clear message:

honker: database file replaced: expected (dev=X, ino=Y), found (dev=X, ino=Z)
at "/path/to/app.db". The watcher cannot recover; close the Database
and reopen with honker.open().

The shm backend additionally panics if the -shm file is recreated (the mmap would be on a dead inode otherwise).

When the watcher panics, every active subscriber’s wake channel is closed. Consumers find out programmatically:

The examples and failure shapes in this section cover bindings that expose the raw update-event stream directly. Other bindings surface the same watcher failure through their queue, listener, or stream abstraction; check the binding’s language guide for its public API.

• Rust: update_events().recv() returns Err(RecvError).
• Python: await events.__anext__() raises RuntimeError("honker: update watcher died (database file likely replaced; recreate honker.open() to recover)").
• Node: await events.next() rejects with a 'watcher died' message. Older versions reject with "receiving on a closed channel" instead.

Not a silent hang. Not just a line in stderr you’ll never read.

Recovery pattern

The recovery snippets below are Python and Node examples, not a universal API shape. In every binding, the invariant is the same: close the failed database handle and open a new one after the database file has been replaced.

import honker

async def main():
    db = honker.open("app.db")
    while True:
        try:
            async for _ in db.update_events():
                # ... do work ...
                pass
        except RuntimeError as e:
            if "watcher died" not in str(e):
                raise
            db.close()
            db = honker.open("app.db")  # fresh handle, fresh watcher, fresh baseline

async function consume() {
  let db = honker.open('app.db');
  while (true) {
    try {
      const events = db.updateEvents();
      while (true) {
        await events.next();
        // ... do work ...
      }
    } catch (e) {
      // 'watcher died' on current versions; older versions say
      // 'closed channel' — which close() also rejects a pending wait with.
      if (
        !String(e.message).includes('watcher died') &&
        !String(e.message).includes('closed channel')
      ) throw e;
      db.close();
      db = honker.open('app.db');
    }
  }
}

For litestream-style restore: a successful restore atomically replaces the database file, which looks identical to any other replacement and trips the watcher. Recreate the Database after restore. The error from update_events() tells you when.

Why update_events() blocks briefly at startup

update_events() waits for the watcher to record its starting state before it returns. After that, you can change the file immediately and the watcher won’t get confused about what was there before. Useful in tests and recovery paths.

The wait is microseconds — one Connection::open, one stat, one channel send. Once per update_events() call.

Source builds

These commands cover the Rust wrapper and the Python/Node bindings that compile the Rust watcher directly. The FFI bindings depend on the watcher capabilities of the native artifact they load or link. The JVM is different again: its watchers are implemented in Java and are not selected by Cargo feature flags. See Binding support.

# Python (maturin)
cd packages/honker
maturin develop --release --features kernel-watcher,shm-fast-path

# Rust (Cargo dependency)
[dependencies]
honker-core = { version = "...", features = ["kernel-watcher", "shm-fast-path"] }

# Node (napi-rs)
cd packages/honker-node
npm run build -- --features kernel-watcher,shm-fast-path

If the requested backend can’t start at runtime (shm needs WAL plus an open connection; kernel needs the OS notify API), honker.open() raises right away. If you ask for a backend whose Cargo feature isn’t compiled into your binary (the published-wheels case), the open call fails loudly too — it throws with an error naming the missing feature. No binding silently falls back to polling.

What we haven’t tested yet

The headline contracts are tested — per-backend wake correctness across journal modes, latency, file-replacement panic, cross-binding death propagation. The long tail is the next phase of work — Phase Atlas in the ROADMAP:

• Rollback wake counts per backend
• VACUUM behavior (all three should panic)
• External non-SQLite writes
• Multiple databases in the same directory (kernel cross-pollination)
• NFS / SMB / FUSE detection in probe()
• Symlinks, fork-without-exec, system suspend

If you hit one of these in production, open an issue with the scenario; it sharpens the work.