Hermes Agent Memory Consistency: Prompt Snapshots, Recall Sidecars, and Asynchronous Writeback

Abstract: Hermes keeps dynamic memory out of a stable prompt prefix with exact recall sidecars.

Version note. This article is based on commit 3f9944b, fetched on July 23, 2026. The package reports version 0.19.0; the latest formal release is v2026.7.20, published three days earlier. Main was moving quickly, so stable installations may not share every behavior described below.

Consider a user who stores “The staging PostgreSQL port for this repository is 55432,” then asks the assistant to inspect staging on the next turn.

A memory-system diagram will usually draw a box called “vector database” between those turns. That box hides most of the engineering: when the fact becomes durable, whether it mutates the active system prompt, where recall enters the request, whether later turns replay identical prefix bytes, and what survives interruption or process exit.

Hermes represents this lifecycle boundary with api_content: a persisted sidecar containing the exact content sent to the model when that content differs from the clean transcript.

Scope and memory state model

Hermes runs the same AIAgent core behind a CLI, TUI, desktop application, messaging gateway, ACP adapter, and API. Its memory is correspondingly plural:

MEMORY.md and USER.md hold small, curated, human-readable facts.
SQLite stores sessions and searchable transcripts with FTS5.
One external MemoryProvider—Honcho, Mem0, Hindsight, OpenViking, or another plugin—may supply automatic recall, ingestion, and tools.
The model can call built-in or provider-specific memory tools explicitly.

A memory SDK decides how to extract, store, rank, and retrieve facts. Hermes must also decide where memory enters a model/tool loop, what is durable before the first API call, which work can happen after response delivery, and which failures are allowed to affect the user’s turn.

For a gateway message, the relevant path is:

GatewayRunner._handle_message()
  → resolve SessionStore entry
  → acquire lease for the resolved session_id
  → load transcript
  → reuse or construct AIAgent
  → AIAgent.run_conversation()
    → build_turn_context()
    → model → tool → result → model
    → finalize_turn()
  → persist/deliver gateway result

The gateway places a pending sentinel in _running_agents before its first relevant await; otherwise, image analysis, speech-to-text, or hooks could let a second message create another agent for the same session. Once routing resolves the final session_id, a turn lease serializes load history → run → flush, including when two routing keys alias one durable session.

Those concurrency mechanisms are part of memory correctness. If two turns load the same history base and append independently, no retrieval model can repair the resulting autobiography.

Four memory clocks.

The phrase “Hermes freezes memory for a session” is only partly correct. The implementation has at least four visibility clocks.

Curated file memory is snapshotted into the system prompt.

MemoryStore maintains two parallel states:

memory_entries and user_entries are live, tool-editable state persisted to disk.
_system_prompt_snapshot is captured during load_from_disk() and used for prompt injection.

build_system_prompt() combines stable identity and tool guidance, project context, the file-memory snapshot, a provider’s static block, and a date-level session line. The assembled string is cached on the agent. It is normally rebuilt for a new session or after context compression invalidates it; that invalidation also reloads memory from disk.

A memory tool can therefore write the port to MEMORY.md and return the new live state immediately, while the already-built system prompt continues to contain the old snapshot.

This protects prefix stability—a memory edit does not invalidate the cached system prefix—and interpretive stability—a tentative observation does not instantly become high-priority identity context.

The downside is an intentional staleness window. “Persisted” and “adopted into the system prompt” are different events.

External static context is also frozen.

MemoryProvider.system_prompt_block() is explicitly for static provider information. Hermes permits at most one external provider at a time, alongside built-in memory. The constraint prevents tool-schema growth and avoids two independent systems competing to own the same lifecycle callbacks and user model.

It is less composable than an arbitrary provider fan-out. It is also much easier to reason about. A team that genuinely needs a memory ensemble can aggregate several stores behind one provider and make conflict policy explicit there.

Query-specific recall is fetched for the current turn.

At turn start, build_turn_context() calls on_turn_start() and then MemoryManager.prefetch_all() once before entering the tool loop. Skill scaffolding is stripped first: a slash skill may expand into a large model-facing instruction body, but the memory backend should search for the user’s actual request, not embed the framework’s prompt.

External prefetch runs in a daemon thread, but the current turn joins it for up to eight seconds. If it times out, Hermes skips recall for that turn. While the stuck call remains alive, another prefetch for the same provider is not started.

That is more precise than saying prefetch is “non-blocking.” The prior turn can queue background warmup for the next turn, but cold current-turn retrieval has a bounded latency budget.

Completed turns are learned in the background.

After a non-interrupted turn has a real user message and final response, Hermes queues two operations:

sync_all(user, assistant, messages=...)
queue_prefetch_all(user)

They run on a single-worker executor. The worker preserves turn order and prevents a slow provider from keeping run_conversation() marked active after the visible response is ready. An interrupted turn is never synchronized: partial prose or an aborted tool chain is not durable conversational truth merely because it looked complete on screen.

Together they produce an influence gradient:

curated, confirmed facts  → frozen system prompt
query-relevant evidence   → current user-message sidecar
complete work history     → SQLite transcript
new observations          → asynchronous provider write

Replay and writeback semantics

Moving dynamic recall out of the system prompt solves one cache problem and creates another.

Assume turn N sends this user message:

Inspect staging.

<memory-context>
The staging PostgreSQL port is 55432.
</memory-context>

If Hermes stores only Inspect staging., then turn N+1 reconstructs a different history. The request prefix diverges at turn N’s user message; every assistant and tool message after that point may need to be prefetched again by the model provider.

Hermes stores two representations:

# Simplified pseudocode, not a verbatim source excerpt
recall = memory_manager.prefetch_all(clean_user_text)
wire_text = compose_user_api_content(clean_user_text, recall, plugin_context)

message["content"] = clean_user_text
message["api_content"] = wire_text
session_db.append_message(message)  # before the first model request

content is the clean semantic transcript. api_content is the exact API-bound string when injections or persistence overrides made it different.

When conversation_loop builds a request:

the current turn reuses its stamped api_content;
a historical user or assistant message substitutes its sidecar into content;
the bookkeeping field itself is removed before transmission;
all repeated model calls within the turn use the same composed bytes.

This is not merely documented intent. tests/agent/test_api_content_sidecar.py starts an in-process mock provider, makes two model calls in one turn, reloads history from SessionDB into a fresh AIAgent, and verifies that the serialized turn-N user message in turn N+1 equals the bytes sent in turn N.

If compaction, redaction, image cleanup, or sequence repair rewrites content, Hermes drops the stale sidecar. Replaying it could restore removed text; the design prefers a cache miss over incorrect content.

codex_app_server and mixture-of-agents paths also have explicit exceptions because their wire construction does not match the normal stamping invariant. Prefix stability is a scoped guarantee, not a slogan.

The transcript is part of the protocol.

The inbound user row is persisted after prefetch and plugin context are composed but before the first LLM request. Assistant tool calls are appended before their side effects execute; tool results are appended afterward and fed back to the model.

SessionDB uses SQLite in WAL mode and retries contended writes with jitter because gateway, CLI, cron, and worktree agents may share a database. It restores messages by autoincrement id, not wall-clock timestamp. Laptop sleep, VM clock changes, or NTP adjustment can make a later tool result carry an earlier timestamp than its assistant tool call. Timestamp ordering would break call/result adjacency and can produce a provider HTTP 400 during replay.

The transcript is therefore a replay protocol:

message insertion order preserves tool causality;
content preserves a clean human view;
api_content preserves historical model input;
tool-call rows preserve in-flight intent across a crash;
FTS indexes make prior sessions searchable without forcing all history into the prompt.

It defines what a recovered agent is allowed to claim happened, beyond “store embeddings and retrieve top-k.”

Asynchronous does not mean durable.

Hermes’s end-of-turn memory work is deliberately best-effort. MemoryManager drains the serialized executor for at most five seconds during shutdown. If a provider is wedged:

queued writes and prefetches may be cancelled and counted;
an already-running daemon task may remain detached until process exit;
the drain state records abandoned writes, abandoned prefetches, and active tasks;
teardown does not block forever.

Focused tests verify FIFO ordering, non-blocking turn completion, bounded shutdown, and explicit reporting of an abandoned queued write. Other tests verify that interrupted turns never reach the external provider and that skill expansion does not pollute recall or ingestion. I ran the four relevant test modules against this snapshot: 75 tests passed.

For a personal assistant, a broken memory SaaS should not freeze every chat surface. Regulated workflows, however, need a durable outbox, idempotency keys, retry ownership, and observable dead-letter handling; response delivery does not guarantee eventual memory persistence.

The provider boundary also expands the privacy boundary. The optional messages argument can include tool calls, file paths, command output, and other workspace material. The official plugin guide correctly tells cloud providers to document what leaves the device, but documentation is not enforcement.

Incident-driven constraints.

Two closed issues reveal why the current boundaries exist.

Issue #13631 described dynamic Honcho context rebuilding the cached system prompt and invalidating prefix caches. The current architecture keeps static provider text in the system prompt, moves per-turn recall into api_content, and replays historical sidecars.

Issue #4889 described a different form of pollution: an expanded skill body became the memory query. The fix now lives once in MemoryManager, before prefetch, queued warmup, and sync fan-out.

Good agent architecture often looks less like a pristine diagram and more like accumulated scar tissue. The valuable move is turning a failure into a reusable boundary instead of adding a provider-specific patch.

Governance and deployment fit

Compared with putting a mutable profile in every system prompt, Hermes sacrifices immediate high-priority adoption to preserve a stable prefix. Compared with plain vector RAG, it defines interrupted-turn semantics, byte-exact replay, session boundaries, and shutdown behavior. Compared with a memory SDK such as Mem0, it does not replace extraction or ranking; it specifies how such a backend participates in an agent lifecycle.

The costs are real:

built-in memory can be stale within a session;
current-turn recall may add up to eight seconds;
background learning can be lost during abnormal termination;
only one external provider is active;
dynamic recall has user-message priority, not system priority;
gateway/run.py, run_agent.py, and hermes_state.py remain large control-plane files;
byte-stable input does not guarantee an upstream cache hit if the provider, model, tools, or cache policy change.

Hermes fits researchers, creators, consultants, and independent developers who repeatedly work with the same assistant and want both continuity and inspectability. It is overbuilt for disposable question answering. It is under-specified as the sole persistence layer for systems requiring transactional memory guarantees.

Operational conclusion.

Long-term assistant deployments must explain who was allowed to define the current user model, rather than optimizing only for memory volume.

A stale port number causes a failed command. A stale health history, credit assumption, or workplace profile can quietly shape decisions for years. The failure rarely looks dramatic; the system simply keeps serving an obsolete person.

That requirement expands memory governance into provenance, confidence, expiration, review queues, identity scopes, retention, export, and proof of deletion. content versus api_content is already a small provenance primitive because it distinguishes what the human said from what the model actually heard.

The users who may resonate most are not necessarily AI enthusiasts. They are people exhausted by repeating context: caregivers, chronic-illness patients, cross-project advisors, research groups, and small-business operators. Continuity can remove real labor. It also concentrates interpretive power, so “remember more” must ship with “inspect, challenge, and retract more.”

My favorite choice in Hermes is the delay between recording and adopting a fact into the system prompt. Humans write notes without instantly turning every sentence into identity. A mature agent may also need separate stages for observation, recording, confirmation, recall, and adoption. In a long relationship, restraint is not lower intelligence. It is part of trust.

Source map and open questions

Gateway ownership: gateway/run.py:10284-10295, 11747-11769, 12628-12654
File-memory dual state: tools/memory_tool.py:123-215
Prompt snapshot and rebuild: agent/system_prompt.py:480-561
Provider lifecycle: agent/memory_provider.py:43-174
Recall, FIFO, bounded shutdown: agent/memory_manager.py:354-684, 1134-1212
Sidecar composition/persistence: agent/turn_context.py:49-126, 939-1028
API replay: agent/conversation_loop.py:973-1065
Completed-turn sync gate: run_agent.py:3661-3720
SQLite message/replay ordering: hermes_state.py:5454-5581, 6161-6216

Sources and verification.

Open questions.

Should api_content become a first-class, auditable, deletable record rather than an internal cache sidecar?
How can a system prove that compaction did not silently change the precedence of an old memory?
If memory must be transactionally durable, should the outbox belong to the agent runtime, the provider adapter, or the gateway?