DeerFlow's Long-Running Agent Runtime: Middleware, Recovery, and Bounded Concurrency

Abstract: DeerFlow governs long tasks with run ownership, checkpoints, budgets, and stop semantics.

Version note. The analyzed commit is a38b1daec392a335016e41f052ae42d90c9ceccd from July 23, 2026. The backend, harness, and frontend manifests say 2.1.0, while the latest formal release is v2.0.0, published June 25. This article therefore describes a development snapshot newer than the release. DeerFlow 2.0 is a ground-up rewrite and shares no code with the 1.x research pipeline.

One DeerFlow incident shows why long-running agents require a control plane: three subagents asked to write one sentence each consumed roughly 4.4 million input tokens.

That was a real DeerFlow incident. Issue #3875 describes a 16-minute run in which the lead made one correct batch of three delegations. The subagents accounted for 99.7% of the tokens because each fell into a degenerate loop. Issue #3857 records two other long-task failures: nine substantially redundant subagents in one paper-analysis run, and the same regional report section appended five times in another.

Adding workers did not add reliability. It increased the concurrency of waste.

The relevant unit of analysis is therefore not multi-agent orchestration in isolation, but the machinery that turns an invocation into an owned, bounded, observable, recoverable run.

Scope and runtime state model

The official documentation calls DeerFlow a runtime for long-horizon agents. More precisely, it is an agent harness plus a reference application.

The harness composes model calls, middleware, skills, tools, sandboxes, memory, subagents, and checkpointed state. The application adds a Gateway, threads, named agents, channels, scheduling, persistence, and a web UI.

It is no longer the fixed planner → researcher → reporter topology associated with DeerFlow 1.x. Nor is it merely a collection of LangChain decorators. A business graph answers, “What step should happen next?” DeerFlow’s surrounding control plane answers different questions:

• Is this run allowed to start?
• Who owns it?
• What state can be restored?
• Which side effects are safe to repeat?
• How much work may it consume?
• What does a disconnected observer replay?
• What proves the run is actually finished?

Those questions are where long-running agents stop looking like prompt demos and start looking like distributed systems.

Six kinds of state in one run.

A streamed thread run follows this path:

POST /threads/{thread_id}/runs/stream
  → start_run
  → RunManager.create_or_reject
  → background run_agent worker
  → ordered middleware
  → model ↔ sandbox tools / subagents
  → StreamBridge events
  → journal, workspace, and thread finalization
  → END_SENTINEL

At least six state surfaces participate:

1. RunRecord owns admission, worker identity, status, cancellation, and stop reason.
2. LangGraph state carries messages, todos, delegations, skill context, and other executable channels.
3. The checkpointer persists recoverable thread state.
4. RunJournal records what models and tools actually did, including token attribution.
5. The workspace contains reports, code, and other artifacts.
6. StreamBridge exposes a bounded, ordered event history to live or reconnecting observers.

They are intentionally not one database abstraction. A checkpoint existing does not prove finalization. A journal is an audit trail, not the graph’s resumable state. An SSE token is an observation, not a commit.

DeerFlow publishes the stream’s end marker at the tail of the worker’s finally block, after buffered subagent events, journal data, workspace changes, titles, thread status, and completion hooks have had a chance to flush. END_SENTINEL is therefore closer to a consumer-visible commit marker than “the last token I happened to receive.”

Request lifecycle and middleware ordering

Suppose a user asks DeerFlow to research a week of global AI news, organize it by region, and write a report. The run may search repeatedly, delegate research, and modify one file several times.

The HTTP connection does not own the task.

thread_runs.py::stream_run() calls start_run(), obtains a RunRecord, and then returns an SSE response subscribed to that run. The Content-Location header points to the canonical run resource.

Before admission, build_run_config() forces the URL’s thread_id, removes caller-supplied double-underscore private keys, and clamps the recursion limit to the server maximum. start_run() also checks model allowlists, thread ownership, and which context values are trusted.

RunManager.create_or_reject() then performs the active-run check while holding its local lock. With a persistent store, a partial unique index for pending/running runs resolves cross-worker contention. The supported multitask strategies—reject, interrupt, and rollback—encode different product behavior rather than collapsing everything into cancellation.

Initial persistence is part of visibility. If the store cannot create the run, DeerFlow should not leave a new run visible only in process memory.

This extra machinery is the price of decoupling business lifetime from socket lifetime. It also makes reconnection and cancellation meaningful rather than accidental effects of TCP.

Recovery semantics are installed before the model starts.

run_agent() checks the process-frozen checkpoint channel mode, initializes RunJournal, marks the run as running, and captures a pre-run workspace snapshot.

Journal construction deliberately happens inside the worker’s main try. A journal initialization failure must flow through the same error and finalization path as later failures; otherwise the SSE consumer could wait forever for an end marker.

The worker then:

• publishes run and thread metadata;
• installs trusted thread_id, run_id, user context, and journal references into the LangGraph runtime;
• builds the agent;
• binds graph, checkpointer, and full|delta mode through CheckpointStateAccessor;
• captures a materialized rollback point before any run mutation.

Raw checkpoint blobs are insufficient in delta mode because full channel values may not be stored there. Rollback captures materialized messages and pending writes. If that capture fails, rollback is disabled rather than restoring partial or empty history. This is a good example of choosing fail-closed for state destruction.

A model call crosses a directional middleware pipeline.

The current lead agent can instantiate more than thirty middleware slots, depending on model capability and configuration. A useful conceptual grouping is:

Input boundary
  InputSanitization
  ToolOutputBudget
  ToolResultSanitization

Execution environment
  ThreadData → Uploads → Sandbox
  SandboxAudit → ReadBeforeWrite

Context and capabilities
  DynamicContext
  SkillActivation → SkillToolPolicy
  DurableContext → Summarization
  Memory
  McpRouting → DeferredToolFilter
  SystemMessageCoalescing

Termination and protocol repair
  SubagentLimit
  LoopDetection
  TokenBudget
  TerminalResponse
  SafetyFinishReason
  Clarification

This is not a simple left-to-right execution log. LangChain runs before_* hooks in registration order and after_* hooks in reverse order. The first wrapper in the list is the outermost wrapper.

Registration order therefore encodes three different relationships:

• preprocessing dependency;
• postprocessing priority;
• wrapper nesting.

Consider provider safety termination. SafetyFinishReasonMiddleware is registered after LoopDetectionMiddleware, so reverse after_model dispatch gives Safety the raw response first. If OpenAI returns content_filter, Anthropic returns refusal, or Gemini returns SAFETY with a partially formed tool call, Safety strips structured and raw tool calls before LoopDetection accounts for the message.

Without that ordering, a truncated write_file call could be executed as valid input. The model might see the broken file, try to repair it, be safety-terminated again, and enter a loop created by the guard itself.

The repository’s middleware-execution-flow.md still documents 14 lead middlewares and four subagent middlewares. Its forward/reverse hook explanation remains useful, but the concrete inventory predates the current read-before-write, durable-context, token-budget, tool-progress, and provider-safety work. Fast-moving runtime code needs a pinned commit; an architecture document is not enough.

Representative implementation pattern.

Here is a reduced sketch of the builder. It is pseudocode, not a verbatim excerpt:

middlewares = [
    ReadBeforeWrite(),
    ToolProgress(),       # outer
    ToolErrorHandling(),  # inner
    DurableContext(),
    Summarization(),
    LoopDetection(),
    TerminalResponse(),
    SafetyFinishReason(),
    Clarification(),
]

if index(ToolProgress) > index(ToolErrorHandling):
    raise RuntimeError("invalid middleware order")

ToolErrorHandling normalizes successful results, exceptions, and provider-specific return shapes, then stamps structured deerflow_tool_meta. Control returns through the outer ToolProgress wrapper, which can now classify whether the call made progress.

If the order is reversed, progress tracking can silently no-op. DeerFlow turns that comment into a build-time assertion, and its tests pin the relationship.

That is the most reusable lesson in this codebase: when order affects correctness, promote prose to an invariant that can fail.

Recovery, limits, and deployment fit

Read-before-write protects the model's epistemic boundary.

The five-duplicate-sections incident was not caused by context compaction; the run still had full context. The failure was “append-only, never read back.”

ReadBeforeWriteMiddleware hashes the current complete file when read_file succeeds and stores that SHA-256 mark on the resulting ToolMessage. Modifying an existing file requires a matching mark for its current version. A successful write changes the hash, so the next modification requires another read.

The middleware locks the gate check and tool execution by (thread or sandbox, normalized path). LangGraph can execute multiple tool calls from one AI message concurrently. Without the critical section, two writes could both validate against the same stale mark before either mutation lands.

Because the mark lives in message state, summarization that removes the read result also removes authorization to write. The agent cannot keep editing a version it no longer has in context.

This is not a general transaction system, but it closely matches an agent’s knowledge boundary: the model must have observed the current version before it can change it.

ToolProgress detects low-value results.

ToolProgressMiddleware runs after execution and reads structured result metadata. It tracks an active/warned/blocked state per (thread, tool).

Repeated no-result responses, rate limits, authentication failures, or highly similar outputs mean the tool has stopped producing value. Recoverable failures can warn the model to change strategy; unrecoverable auth or configuration errors can block immediately.

LoopDetection detects repeated intent.

LoopDetectionMiddleware runs after the model and before execution. It hashes the tool-call set (name + args) and tracks tool-type frequency. The defaults warn after the third identical set and hard-stop after the fifth. A separate 30/50 frequency guard catches loops across different arguments, such as reading a succession of files forever.

The warning is queued in after_model but injected only on the next wrap_model_call. Injecting a human warning immediately after an assistant tool-call message would place it before the corresponding tool results, violating provider call/result adjacency. By the next model call, all ToolMessage responses are present and the warning can safely sit at the end.

One guard reasons about result quality; the other reasons about call patterns. Calling both “loop prevention” obscures why they need different hooks, state scope, and interventions.

Compaction and completed work.

Summarization is unavoidable in long tasks, but the harder problem is preserving operational state.

In the redundant-delegation incident, the lead repeatedly behaved as if earlier research batches had never been assigned. DurableContextMiddleware now extracts delegations and loaded skills into checkpointed channels before compaction removes their original tool messages.

At model-call time it temporarily projects:

• summary_text;
• a bounded delegation ledger;
• loaded skill context.

The projection is not written back to state, avoiding duplicate accumulation on every turn. Runtime-provided authority rules are a SystemMessage; user-, model-, tool-, and subagent-derived values are a hidden HumanMessage explicitly framed as data, not instructions.

Issue #4039 shows how subtle this contract is. After three parallel tool calls, a “keep four messages” policy can preserve:

assistant(tool_calls) → tool → tool → tool

The removed system and user context existed only in summary_text. The subagent chain did not project it back, so a strict OpenAI-compatible provider rejected the assistant-first history with HTTP 400.

The fix was not “keep one more message.” DeerFlow restored durable context before the preserved tool tail and then used SystemMessageCoalescingMiddleware to ensure strict providers saw exactly one leading system message.

Context compaction is therefore a protocol transformation. It must preserve both meaning and a provider-valid message grammar.

Concurrency and cost limits.

SubagentLimitMiddleware combines a per-response concurrent-call limit with the number of delegations already recorded for the current run. The default total is six; valid configuration ranges from one to fifty, and concurrent calls are clamped between two and four.

That still cannot stop three admitted workers from looping internally. After the 4.4M-token incident, the subagent runtime gained LoopDetection, TokenBudget, Summarization, and DurableContext. The default token ceiling is one million with summarization enabled and two million without it. Those are circuit breakers, not desirable spend targets.

Most hard guards do not raise. They remove tool calls so the graph can terminate naturally and put a reason into runtime context:

loop_capped
token_capped
safety_capped
subagent_limit_capped

The RunRecord can still be success while carrying stop_reason. That preserves a compact status enum and lets partial useful text survive, but every consumer must understand the compound semantic. An HTTP 200 or green “success” count alone will hide runs rescued by a fuse.

There is also a visible scaling limit: the guards currently write one shared reason key. If future behavior allows multiple caps in one run, the runtime will need first-wins, severity ordering, or a structured list.

Checkpoints and streams.

MemoryStreamBridge retains a bounded in-process event log. RedisStreamBridge uses one Redis Stream per run for cross-process subscriptions. Both provide monotonic IDs, heartbeat wakeups, Last-Event-ID replay, and an explicit end marker.

The checkpointer persists resumable thread state. It does not prove the run finished.

Issue #3265 exposed the distinction. The old non-streaming /wait path awaited the task directly. When a client or proxy disconnected during a long pip install, the handler could serialize whatever checkpoint already existed and make a half-finished run look normally complete.

The current /wait consumes the same StreamBridge as SSE. It serializes final checkpoint state only after observing END_SENTINEL; a disconnect follows the run’s cancellation policy and does not become a false success.

The snapshot’s HEAD, PR #4354, fixes another end-to-end reliability problem. Streaming write_file and str_replace argument deltas token by token made the browser repeatedly parse a growing JSON payload, approaching quadratic work. The worker now batches those file-tool deltas in groups of 32 while preserving token streaming for normal assistant text.

Backpressure can live all the way from provider wire chunks to the browser main thread.

Comparison with adjacent designs.

A conventional LangGraph business graph makes workflow topology explicit. DeerFlow moves cross-cutting concerns—sanitization, sandboxing, context, budgets, recovery—around the graph so different task shapes can reuse them. The gain is composition; the cost is that the graph alone no longer explains control flow.

Hermes Agent centralizes more lifecycle ordering in an explicit conversation loop. It is easier to trace persistence and memory timing through a few large files. DeerFlow decomposes each model turn into middleware and delegates global lifecycle to the Gateway worker. Replacement is easier; ordering bugs become a larger design surface.

OpenClaw also treats its Gateway, sessions, policies, and recovery as the product around an embedded agent runner. OpenClaw is oriented toward a persistent personal agent across many channels. DeerFlow currently emphasizes the App/Harness split, LangGraph-compatible run/thread resources, subagents, and artifact workspaces.

All three mechanisms move differentiation away from the model call itself and toward the surrounding control plane.

Costs and intended users.

DeerFlow makes sense when tasks take minutes, touch tools and files, produce artifacts, and genuinely need reconnect, cancellation, rollback, audit, or multiple execution surfaces. Research operations, content production, internal automation, and agent-platform teams are plausible users.

It is excessive for one- or two-call assistants, deterministic fixed DAGs, and side-effect-free question answering.

Operationally, adopters inherit:

• sandbox and workspace lifecycle;
• database-backed runs and checkpoints;
• Redis or PostgreSQL concerns in multi-worker deployments;
• compound success/stop semantics;
• a large middleware mutation surface;
• partial failure across checkpoint, journal, workspace, and stream;
• rapid development-line drift beyond formal releases.

The system also makes feature-specific fail-open/fail-closed choices. Read-before-write fails open if it cannot inspect a sandbox file, letting the real tool surface the error. Rollback fails closed if its snapshot is incomplete. Journal completion persistence is non-fatal. A buggy safety detector is treated as no match rather than taking down the run.

There is no universal answer. Each boundary chooses which error—false rejection or false permission—would be worse.

I ran focused middleware-order, read-before-write, stop-reason, RunManager, and StreamBridge tests. The result was 158 passed, five skipped, and one failed. The failing test replaces the input-sanitization module with an incomplete stub; a new transitive import from ReadBeforeWrite then expects neutralize_untrusted_tags and raises ImportError inside the fixture. That is best classified as a test-isolation gap in this snapshot—not omitted as “all green,” and not inflated into a runtime behavior regression. The exact command and traceback are in the evidence dossier.

Operational conclusion.

DeerFlow will resonate most with teams that have already been burned by automation, not necessarily with people chasing the newest model. They recognize the real cost of duplicated reports, runs that look successful but stopped halfway, loops that consume a budget, and side effects no one can attribute.

“Multi-agent” will gradually become an implementation detail. Buyers will ask plainer questions: Why did the task stop? How much did it spend? Which step changed the file? Where can it resume? Was this success clean or capped?

Long-running Agent deployments consequently require SRE practices, run ledgers, budget policy, replay debuggers, and side-effect audit systems.

The important design pattern is how incidents were translated into distinct mechanisms: repeated delegation became a ledger and total cap; duplicate output became a file-version gate; runaway behavior became result and call-pattern guards; partial completion became an end-of-run protocol. None was assigned only to a more admonishing system prompt.

My remaining concern is that a chain of twenty or thirty middlewares is becoming a small programming language. Today its type system consists mostly of comments, local assertions, and tests. A mature extension model may need machine-readable declarations such as requires_before, observes, and mutates, plus conflict detection at build time.

Otherwise the industry may replace prompt spaghetti with middleware spaghetti.

Source map and open questions

• Run API: backend/app/gateway/routers/thread_runs.py:486-548
• Config and admission: backend/app/gateway/services.py:441-559, 885-1047
• SSE and wait completion: backend/app/gateway/services.py:1105-1195
• Run ownership: backend/packages/harness/deerflow/runtime/runs/manager.py:920-1045
• Worker lifecycle: backend/packages/harness/deerflow/runtime/runs/worker.py:375-909
• Shared runtime middleware: backend/packages/harness/deerflow/agents/middlewares/tool_error_handling_middleware.py:154-264
• Lead middleware: backend/packages/harness/deerflow/agents/lead_agent/agent.py:265-449
• Read-before-write: backend/packages/harness/deerflow/agents/middlewares/read_before_write_middleware.py:1-268
• Progress and loop guards: tool_progress_middleware.py:1-578, loop_detection_middleware.py:1-735
• Durable context: durable_context_middleware.py:196-287
• Checkpoint accessor: backend/packages/harness/deerflow/runtime/checkpoint_state.py:1-187
• Stream bridges: backend/packages/harness/deerflow/runtime/stream_bridge/

Sources and verification.

• Full evidence dossier and focused test record
• Pinned source snapshot
• DeerFlow 2.0.0 release
• Official Core Concepts
• Official Agents and Threads
• Issue #3265: non-streaming partial completion
• Issue #3857: redundant delegation and duplicate output
• Issue #3875: unbounded subagent loops
• Issue #4039: durable context lost after compaction
• PR #4354: responsive large-file streaming

Open questions.

1. If two guards trigger in one run, should stop reasons be first-wins, severity-ordered, or an append-only structured list?
2. Can middleware declare machine-readable ordering and mutation contracts so the builder rejects incompatible compositions?
3. How should a replay debugger align checkpoint, journal, and stream timelines without treating transient events as durable truth?