Watcher backends

Honker wakes consumers on every database commit. By default, one background thread per database asks SQLite “did anything change?” every millisecond. That’s it.

Two opt-in alternatives exist for advanced users. Both are experimental and source-only — not in published wheels yet.

This page is for people who:

• Run many honker’d databases on one host and care about idle CPU.
• Want to test event-driven wake on their platform before we ship it broadly.
• Need to understand what happens when the database file is swapped under the watcher (litestream restore, NFS remount, atomic backup swap).

If none of those apply, the default works and you can skip the rest.

TL;DR

db = honker.open("app.db")                              # polling, the default
db = honker.open("app.db", watcher_backend="kernel")    # experimental: OS file events
db = honker.open("app.db", watcher_backend="shm")       # experimental: mmap WAL index, WAL only

const db = honker.open('app.db');
const dbK = honker.open('app.db', undefined, 'kernel');
const dbS = honker.open('app.db', undefined, 'shm');

use honker_core::{SharedUpdateWatcher, WatcherBackend, WatcherConfig};

let polling = SharedUpdateWatcher::new(db_path.clone());
let kernel = SharedUpdateWatcher::new_with_config(
    db_path.clone(),
    WatcherConfig { backend: WatcherBackend::KernelWatch },
);
let shm = SharedUpdateWatcher::new_with_config(
    db_path,
    WatcherConfig { backend: WatcherBackend::ShmFastPath },
);

The kernel and shm backends are gated behind Cargo features (kernel-watcher, shm-fast-path). Published wheels (pip install honker, npm install) build polling-only. To use the experimental backends today, build from source with the feature on — see Source builds below.

At a glance

backend	wake source	spurious wakes	missed wakes	journal mode	shipped in wheels
polling (default)	PRAGMA data_version every 1 ms	never	never	any	yes
kernel	OS file events on db file + parent dir + -wal/-shm/-journal	possible	possible (if the OS drops events)	any	no — source only
shm	mmap read of SQLite’s WAL index every 1 ms	never	never (mmap inode-change → panic)	WAL only	no — source only

When to choose which

• Stay on polling unless you have a specific reason to switch. It’s the only backend that promises every wake fires once and only once. It works in every journal mode. It’s the only one in published wheels. The CPU cost — about 3.5 ms per second of busy work — is rounding error on any modern machine.
• Try kernel if you’re running many honker’d databases on one host and want idle CPU near zero, or if you’re on Linux and want wakes faster than 1 ms. The catch: some wakes can be lost if the OS drops events under load. Consumers re-read state on every wake, so a lost wake means delayed work, not lost work.
• Try shm if you’re on a high-QPS WAL workload and the 1 ms default wake is too slow (rare). Wakes drop to under a microsecond. The catch: it reads SQLite’s WAL index format directly, so a future SQLite version that changes that format would break it.

Rule of thumb: if you can’t say in one sentence what you’d gain by switching, stay on polling.

The backends in detail

Polling (default)

One background thread per database. Every millisecond, it runs PRAGMA data_version on a read-only connection. That counter goes up after any commit by any process, in any journal mode. Around 3.5 microseconds per check. One thread regardless of subscriber count.

The polling backend keeps its connection through transient SQLITE_BUSY / SQLITE_LOCKED errors. Those happen when a writer is mid-commit on a rollback-journal mode (DELETE / TRUNCATE / PERSIST) and the watcher’s PRAGMA can’t get a lock. Instead of dropping the connection and reconnecting (which would silently re-baseline and miss wakes), it just retries the next millisecond. So non-WAL workloads get reliable wake delivery even during writer-locked windows.

Kernel

Wakes on file events from the OS — inotify on Linux, kqueue on macOS, ReadDirectoryChangesW on Windows. Watches the database file, the parent directory, and -wal/-shm/-journal sidecars when they exist.

Good:

• Lower idle CPU. No 1 ms tick — wakes only when something changes on disk.
• Faster wakes on Linux/macOS for WAL workloads (typically under a millisecond).

Bad:

• Spurious wakes. Any file change in the directory wakes consumers. They re-read state from SQLite on every wake anyway, so this is wasted work, not incorrect.
• Missed wakes. If the OS drops or coalesces an event, that commit’s wake doesn’t fire. The 5 second idle_poll_s paranoia poll is the only backstop.
• macOS kqueue can’t reliably catch in-place writes to existing files for non-WAL journal modes (TRUNCATE / PERSIST in particular). WAL works. Use the kernel backend with WAL.

Shm fast path

Reads iChange at byte 8 of SQLite’s WAL index header (-shm file) via mmap. SQLite increments iChange on every WAL write. The watcher compares the current value to the previous one every millisecond and fires a wake on change.

Good:

• Sub-microsecond wake detection. A memory load instead of a SQLite query.
• Same low latency under heavy load.

Bad:

• WAL only. No -shm file exists in DELETE / TRUNCATE / PERSIST. If you ask for shm in those modes, honker.open() raises immediately.
• Reads SQLite’s WAL index format directly. A future SQLite version that changes that format would break it. We have an equivalence test against current SQLite to catch this.

Database file swapped under the watcher

Every backend records the database file’s identity at startup and re-checks it every 100 ms. If the identity changes — atomic rename, litestream restore, NFS volume remount, VACUUM — the watcher panics with a clear message:

honker: database file replaced: expected (dev=X, ino=Y), found (dev=X, ino=Z)
at "/path/to/app.db". The watcher cannot recover; close the Database
and reopen with honker.open().

The shm backend additionally panics if the -shm file is recreated (the mmap would be on a dead inode otherwise).

When the watcher panics, every active subscriber’s wake channel is closed. Consumers find out programmatically:

• Rust: update_events().recv() returns Err(RecvError).
• Python: await events.__anext__() raises RuntimeError("honker: update watcher died (database file likely replaced; recreate honker.open() to recover)").
• Node: await events.next() rejects with the same message.

Not a silent hang. Not just a line in stderr you’ll never read.

Recovery pattern

import honker

async def main():
    db = honker.open("app.db")
    while True:
        try:
            async for _ in db.update_events():
                # ... do work ...
                pass
        except RuntimeError as e:
            if "watcher died" not in str(e):
                raise
            db.close()
            db = honker.open("app.db")  # fresh handle, fresh watcher, fresh baseline

async function consume() {
  let db = honker.open('app.db');
  while (true) {
    try {
      const events = db.updateEvents();
      while (true) {
        await events.next();
        // ... do work ...
      }
    } catch (e) {
      if (!String(e.message).includes('watcher died')) throw e;
      db.close();
      db = honker.open('app.db');
    }
  }
}

For litestream-style restore: a successful restore atomically replaces the database file, which looks identical to any other replacement and trips the watcher. Recreate the Database after restore. The error from update_events() tells you when.

Why update_events() blocks briefly at startup

update_events() waits for the watcher to record its starting state before it returns. After that, you can change the file immediately and the watcher won’t get confused about what was there before. Useful in tests and recovery paths.

The wait is microseconds — one Connection::open, one stat, one channel send. Once per update_events() call.

Source builds

# Python (maturin)
cd packages/honker
maturin develop --release --features kernel-watcher,shm-fast-path

# Rust (Cargo dependency)
[dependencies]
honker-core = { version = "...", features = ["kernel-watcher", "shm-fast-path"] }

# Node (napi-rs)
cd packages/honker-node
CARGO_BUILD_FEATURES=kernel-watcher,shm-fast-path npm run build

If the requested backend can’t start at runtime (shm needs WAL plus an open connection; kernel needs the OS notify API), honker.open() raises right away. If you ask for a backend whose Cargo feature isn’t compiled into your binary (the published-wheels case), it falls back to polling and logs once to stderr. Either way, no silent swap.

What we haven’t tested yet

The headline contracts are tested — per-backend wake correctness across journal modes, latency, file-replacement panic, cross-binding death propagation. The long tail is the next phase of work — Phase Atlas in the ROADMAP:

• Rollback wake counts per backend
• VACUUM behavior (all three should panic)
• External non-SQLite writes
• Multiple databases in the same directory (kernel cross-pollination)
• NFS / SMB / FUSE detection in probe()
• Symlinks, fork-without-exec, system suspend

If you hit one of these in production, open an issue with the scenario; it sharpens the work.