server: persist the --cache-ram prompt cache to a dir with per-entry idle sync #1

2026-10-22T16:58:43-05:00

hal9000 commented

2026-10-22 16:58:43 -05:00

What

Reworks prompt-cache persistence from a single shutdown-written file into a directory of per-entry files with live idle-sync, replacing --prompt-cache-file with:

--prompt-cache-dir PATH — one file per prompt-cache entry, named <content-hash>.bin (created if missing)
--prompt-cache-sync-idle SEC — how long an entry must be unchanged before a sweep writes it (default 300)

Design

Per-entry files: the prompt cache is a shared prefix pool (entries have no slot identity), so the natural unit is the cache entry, not the slot. Each file is self-describing: magic PCD2, version, file size, model fingerprint (n_ctx/n_embd/n_layer/n_head/n_vocab/model path/build commit), then the KV blobs. The dir listing is the index — no manifest.
Live idle-sync: a background sweep thread (same pattern as the stream-session GC) runs every 30 s and writes entries idle for --prompt-cache-sync-idle. Idle is per entry (time since finalized/mutated; LCP reads don't count). Shutdown/sleep-entry flushes become near-no-ops.
Strict mirror: files no longer in the RAM cache (stolen or evicted) are pruned. Prune membership is re-checked under the cache mutex right before each unlink. Prune only touches files this instance has mirrored (seeded from the loaded cache at start, extended by sweeps) — a run that loads nothing (e.g. wrong -c) cannot wipe a good mirror.
Integrity = the filename: no CRC field. On load the content hash is recomputed from the parsed payload and compared against the file's name; mismatch ⇒ skipped with a warning. Truncation is caught by the size header, model mismatch by the fingerprint. Writes are atomic (.tmp + fsync + rename), so kill -9 can never yield a half-written entry.
Concurrency: cache entries are refcounted immutable blobs (shared_ptr); the sync thread only copies finalized bytes and never touches llama_context. Steal path erases instead of clearing in place.
Sleep mode: flush + live-slot capture on entering sleep (contexts are destroyed there); flush on wake reload. Skips capture when already sleeping — the old path serialized slots through llama_state_seq_* against destroyed contexts (use-after-free, crash on SIGTERM-while-sleeping).
Shared dirs: dir is flock'd; a second instance on the same dir logs an error and disables persistence rather than corrupting the mirror.

Verification

Full battery on qwen2.5-0.5b (live sweep, restore + cache-hit warm eval 73 ms/tok vs 5.3 s cold, steal-prune, eviction-prune, byte-corruption reject, truncation reject, fingerprint reject, wipe-guard, flock, kill -9, sleep flush, sleep+SIGTERM): 6/6 scenarios green on the final build, incl. (shutdown) flush writing the live conversation and clean .lock removal while sleeping.

Known (pre-existing, upstream)

SIGTERM landing during the sleep transition can self-deadlock: queue sleeping-state callbacks run while mutex_tasks is held, and the signal handler's terminate() takes the same mutex. Not introduced here — this commit only widens the window by ~25 ms of flush work.

## What Reworks prompt-cache persistence from a single shutdown-written file into a **directory of per-entry files with live idle-sync**, replacing `--prompt-cache-file` with: - `--prompt-cache-dir PATH` — one file per prompt-cache entry, named `<content-hash>.bin` (created if missing) - `--prompt-cache-sync-idle SEC` — how long an entry must be unchanged before a sweep writes it (default **300**) ## Design - **Per-entry files**: the prompt cache is a shared prefix pool (entries have no slot identity), so the natural unit is the cache entry, not the slot. Each file is self-describing: magic `PCD2`, version, file size, model fingerprint (`n_ctx`/`n_embd`/`n_layer`/`n_head`/`n_vocab`/model path/build commit), then the KV blobs. The dir listing is the index — no manifest. - **Live idle-sync**: a background sweep thread (same pattern as the stream-session GC) runs every 30 s and writes entries idle for `--prompt-cache-sync-idle`. Idle is **per entry** (time since finalized/mutated; LCP reads don't count). Shutdown/sleep-entry flushes become near-no-ops. - **Strict mirror**: files no longer in the RAM cache (stolen or evicted) are pruned. Prune membership is re-checked under the cache mutex right before each unlink. Prune only touches files this instance has mirrored (seeded from the loaded cache at start, extended by sweeps) — a run that loads nothing (e.g. wrong `-c`) cannot wipe a good mirror. - **Integrity = the filename**: no CRC field. On load the content hash is **recomputed from the parsed payload** and compared against the file's name; mismatch ⇒ skipped with a warning. Truncation is caught by the size header, model mismatch by the fingerprint. Writes are atomic (`.tmp` + `fsync` + rename), so `kill -9` can never yield a half-written entry. - **Concurrency**: cache entries are refcounted immutable blobs (`shared_ptr`); the sync thread only copies finalized bytes and never touches `llama_context`. Steal path erases instead of clearing in place. - **Sleep mode**: flush + live-slot capture on entering sleep (contexts are destroyed there); flush on wake reload. **Skips capture when already sleeping** — the old path serialized slots through `llama_state_seq_*` against destroyed contexts (use-after-free, crash on SIGTERM-while-sleeping). - **Shared dirs**: dir is flock'd; a second instance on the same dir logs an error and disables persistence rather than corrupting the mirror. ## Verification Full battery on qwen2.5-0.5b (live sweep, restore + cache-hit warm eval 73 ms/tok vs 5.3 s cold, steal-prune, eviction-prune, byte-corruption reject, truncation reject, fingerprint reject, wipe-guard, flock, kill -9, sleep flush, sleep+SIGTERM): 6/6 scenarios green on the final build, incl. `(shutdown)` flush writing the live conversation and clean `.lock` removal while sleeping. ## Known (pre-existing, upstream) SIGTERM landing *during* the sleep transition can self-deadlock: queue sleeping-state callbacks run while `mutex_tasks` is held, and the signal handler's `terminate()` takes the same mutex. Not introduced here — this commit only widens the window by ~25 ms of flush work.

hal9000 added 1 commit 2026-10-22 16:58:44 -05:00

server: persist --cache-ram prompt cache to disk across restarts 2fc79ce5bc

Add --prompt-cache-file PATH (env: LLAMA_ARG_PROMPT_CACHE_FILE) to llama-server.
When set, the in-RAM prompt cache (--cache-ram) is saved to the given file on
shutdown (atomic write: .tmp + fsync + rename) and restored on the next startup.

The file header carries a fingerprint of the loaded model and context (n_ctx,
n_embd, n_layer, n_head, n_vocab and the model path); on any mismatch the whole
file is skipped with a warning. Per-entry sanity checks skip truncated or
invalid entries. Entries saved with a draft model are restored target-only (with
a warning) when no draft context is loaded, and entries with media tokens are
skipped when --mtmd is not active.

No-op when the flag is unset: no file I/O, no behavior change.

The slot save/restore byte-framing helpers are moved to a shared header
(server-ckpt.h) so both paths use the same code.

References upstream discussion: ggml-org/llama.cpp#27759

hal9000 added 1 commit 2026-10-22 17:56:38 -05:00

server: capture live slots when persisting the prompt cache 481c896c10

the RAM prompt cache only holds contexts that were saved when a slot was
about to be reused or cleared, so an active conversation is not in it at
shutdown time and was not persisted. capture the non-empty live slots
into the cache in prompt_cache_save() before writing the file, with the
size limit lifted for the capture so contexts larger than --cache-ram
are persisted too.

also: do not evict entries loaded from the file when they exceed the
--cache-ram limit (the file is an explicit request to persist them);
warn instead. the limit still applies to entries created at runtime,
which evict the oldest entries as needed.

hal9000 added 1 commit 2026-10-30 21:32:14 -05:00

server: rework prompt cache persistence into a dir with per-entry idle sync 31e7d04be0

Replace the single --prompt-cache-file with --prompt-cache-dir: each
--cache-ram entry becomes one self-describing, fingerprint-checked file
named by its content hash, written atomically (.tmp + fsync + rename).

A background sweep (GC-thread pattern) mirrors the cache into the dir
while the server runs: entries older than --prompt-cache-sync-idle
(default 300s) are written without stopping serving, and files that no
longer belong to the cache are pruned. Membership is re-checked under
the cache mutex before each unlink, and any write error aborts the
sweep (retry next pass, RAM stays the source of truth).

To make off-thread snapshotting safe, cache entries are now
refcounted and immutable after publish: two-phase alloc
(make_state/insert_state stamps content_hash + t_finalized_ms), and the
steal path erases from the list instead of clearing blobs in place.
The dir is serialized with a flock lock file; a second instance
disables persistence with an error instead of corrupting the mirror.
The cache is also flushed when entering sleep mode, which otherwise
drops the RAM cache on wake.

Co-Authored-By: Hermes <hermes@nousresearch.com>

hal9000 changed title from ~~server: persist --cache-ram prompt cache to disk across restarts~~ to server: persist the --cache-ram prompt cache to a dir with per-entry idle sync

2026-10-30 21:33:34 -05:00

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin prompt-cache-persistence:prompt-cache-persistence

git checkout prompt-cache-persistence

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: darthrubik/llama-cpp#1