llm-operations

79 beliefs (59 IN, 20 OUT)

This topic covers how the reasons system integrates with external LLM services and manages multi-agent belief import, addressing two fundamental challenges: how to safely incorporate LLM-generated content into a truth maintenance network, and how to import and isolate beliefs from external agent sources without corrupting local reasoning. These concerns matter because LLMs produce unbounded, potentially malformed output, and multi-agent scenarios introduce namespace collisions, lifecycle coordination, and cross-contamination risks.

The agent import subsystem is built around a two-node relay mechanism: each imported agent gets an active premise and a derived inactive node that function as a kill switch (active-inactive-relay-pair). The inactive node is placed in every imported belief's outlist rather than as an antecedent (kill-switch-uses-outlist-not-antecedent), a deliberate design choice that preserves per-belief retraction semantics — if the active node were an antecedent, it would create an alternative justification path that defeats individual retraction (active-not-in-antecedents). Namespace isolation uses a colon-separated prefix convention where every imported node gets an agent-name prefix (import-agent-namespace-prefix, namespace-prefix-is-colon-separated), and cascades from retracting one agent's active premise do not propagate to other agents (agent-cascades-are-isolated-by-namespace). The kill switch is fully reversible: re-asserting the active premise restores all agent beliefs via BFS propagation (kill-switch-cascade-is-reversible). Import handles mixed truth states carefully — beliefs marked OUT in the source receive empty justification lists to prevent resurrection by recompute (import-agent-out-beliefs-not-resurrected), and nodes are added before explicit retractions are applied to ensure the dependency graph is fully wired first (deferred-retraction-ordering).

LLM integration enforces safety at multiple boundaries. All LLM calls execute through subprocess isolation with CLAUDECODE environment stripping to prevent recursive invocation (all-external-execution-is-subprocess-isolated, invoke-model-strips-claudecode-env), and prompts are passed via stdin rather than command-line arguments to avoid shell injection (llm-prompt-always-via-stdin). The derive pipeline validates proposals with fail-soft filtering and Jaccard retraction guards at the LLM boundary, while the API layer enforces atomic persistence at the storage boundary (llm-driven-mutations-are-safely-bounded). The list-negative classification pipeline uses two-stage filtering — keyword pre-filtering followed by batched LLM classification (list-negative-uses-two-stage-classification) — with a resilient parser that extracts JSON from prose-laden responses and falls back to zero results on complete failure (list-negative-parser-is-fully-resilient). The review pipeline operates entirely on in-memory snapshots (review-has-no-storage-dependency), evaluates only derived beliefs with premises excluded (review-skips-premises), and defaults missing LLM response fields to passing to avoid false alarms (review-parse-defaults-safe). Auto-retraction from review is gated behind a dry-run flag (auto-retract-respects-dry-run).

A significant number of beliefs in this topic are OUT, predominantly higher-level synthesis nodes that attempted to compose multiple safety properties into unified claims. These include broad assertions about production hardening (llm-integration-is-production-hardened), defense-in-depth across layers (llm-integration-is-defense-in-depth-across-layers), and complete lifecycle coverage (review-completes-llm-quality-lifecycle). Their retraction suggests that while the individual safety mechanisms are well-established and verified, the composite claims that bundled them together were found to overreach or became stale as the underlying beliefs they depended on were revised. The granular, mechanistic beliefs about specific behaviors — how parsing works, what gets stripped, which batching size is used — remain solidly IN, reflecting that the system's safety story is better understood as a collection of specific, verified properties than as a single sweeping guarantee.

IN active-inactive-relay-pair
Each imported agent gets exactly two infrastructure nodes: `agent:active` (premise, starts IN) and `agent:inactive` (derived via SL with `outlist=[active_id]`, starts OUT); every imported belief includes `inactive_id` in its outlist
IN active-not-in-antecedents
The `active` premise is deliberately excluded from imported beliefs' antecedents; if it were an antecedent, it would provide a second always-valid justification path that defeats per-belief retraction semantics
IN active-premise-not-in-antecedents
Covered by existing `active-not-in-antecedents` and `kill-switch-uses-outlist-not-antecedent`
OUT agent-beliefs-undergo-full-revision
Agent-imported beliefs participate in the full revision system: the self-contained agent subsystem provides isolated lifecycle management and reversible defeat, while the comprehensive revision system ensures agent beliefs are subject to the same outlist defeat and contradiction resolution as locally-created beliefs — no revision exception exists for external provenance.
IN agent-cascades-are-isolated-by-namespace
Retracting one agent's active premise does not affect other agents' beliefs, because each agent's imported beliefs reference only their own `inactive` node in their outlist
IN agent-import-creates-two-control-nodes
Covered by existing `active-inactive-relay-pair` which captures the same two-node control structure
IN agent-isolation-spans-identity-and-authorization
Agent beliefs are isolated through two independent containment mechanisms at different system levels: namespace prefixing with relay pairs provides identity-level isolation (preventing ID collisions and enabling per-agent lifecycle control via kill-switch), while transitive subset-gated access tags provide authorization-level isolation (controlling per-caller visibility with tag inheritance).
IN agent-isolation-through-namespace-and-relay
Agent beliefs are doubly isolated: namespace prefixing prevents ID collisions, while the active/inactive relay pair provides per-agent kill-switch semantics without cross-agent interference
IN agent-subsystem-is-self-contained
The agent subsystem provides complete lifecycle management: import handles mixed truth states and topological cycles, namespace/relay pairs provide isolation and kill-switches, and all defeat operations are reversible for agent reactivation.
IN all-external-execution-is-subprocess-isolated
All LLM-facing operations execute through subprocess isolation with environment scrubbing: derive shells out to CLI binaries rather than importing SDKs (achieving provider agnosticism), and both derive and ask independently strip the CLAUDECODE environment variable (preventing recursive invocation) — two independent safety goals achieved through the same architectural choice.
OUT all-external-inputs-produce-correct-state
Both external input pathways (LLM derivation and multi-agent import) produce a fully correct persisted network state — bounded validation prevents invalid beliefs, namespace isolation prevents cross-contamination, and layered reconciliation handles all truth states — provided the agent count bug is fixed and missing source files are detected.
IN all-external-inputs-safely-integrated
Both LLM-derived beliefs and agent-imported beliefs are safely integrated into the network: defensive validation with retraction guards bounds LLM output, while complete reconciliation with dual modes and heterogeneous truth handling manages agent imports.
OUT all-llm-interactions-are-bounded-and-fail-soft
All LLM-facing operations apply consistent defensive patterns across both interactive (ask) and batch (derive) paths: bounded execution (iteration caps, timeout handling), fail-soft error recovery (fallback to raw results or skipped proposals), and environment isolation (stripping recursive invocation variables).
OUT all-llm-operations-achieve-coverage-and-fault-tolerance
All LLM-driven knowledge operations achieve both complete coverage and fault tolerance: the derive pipeline provides safe, complete, production-ready derivation with exhaustive exploration and accurate budget allocation, while interactive queries via ask are fault-tolerant, execution-bounded, and gracefully degrading — no LLM-facing operation can crash, hang, produce unbounded output, or corrupt the network.
OUT all-llm-operations-are-defensively-bounded
All three LLM-facing operations — interactive query (ask), batch derivation (derive), and belief classification (list_negative) — apply consistent defensive patterns: bounded execution, fail-soft error handling, hallucination filtering, and graceful degradation on LLM unavailability.
IN auto-retract-respects-dry-run
The `--auto-retract` flag in the `review-beliefs` CLI is gated by `--dry-run`: when dry-run is active, findings are displayed but no database mutation occurs, even for beliefs flagged as invalid.
IN automated-sync-achieves-full-lifecycle-coverage
Automated repeated sync safely reconciles external beliefs with complete lifecycle coverage including full source staleness detection — idempotent cascade-preserving sync combined with conservative staleness gating ensures no lifecycle gap between sync runs.
IN deferred-retraction-ordering
During agent import, nodes are added and truth values propagated before explicit retractions are applied, ensuring the dependency graph is fully wired before OUT transitions are forced.
IN estimate-tokens-chars-div-4
`estimate_tokens` uses `len(text) // 4` with a minimum return value of 1; it never returns 0, even for empty strings.
IN extract-tool-call-returns-first-match
When LLM output contains multiple JSON objects with a `"tool"` key, `extract_tool_call()` returns only the first valid match and ignores the rest; malformed JSON lines are silently skipped.
IN import-agent-inactive-always-in-outlist
Every imported belief unconditionally has `agent:inactive` in its outlist, enforced in `_build_justifications()` with no code path that omits it — this is the mechanism behind the per-agent kill switch.
IN import-agent-infra-nodes-excluded-from-removal
`_sync_claims()` filters out infrastructure nodes (active/inactive) via `infra_ids` before computing the removal set, so the kill-switch pair is never retracted by remote-wins sync.
IN import-agent-namespace-prefix
Every node imported from agent X gets the ID prefix `X:`, including infrastructure nodes `X:active` and `X:inactive`, ensuring zero collision with local or other-agent beliefs
IN import-agent-normalizers-share-intermediate-schema
Both `_normalize_markdown()` and `_normalize_json()` produce identical intermediate dicts (id, text, is_out, source, source_hash, date, metadata, raw_justifications) before hitting shared `_import_claims`/`_sync_claims` logic.
IN import-agent-out-beliefs-get-empty-justifications
Beliefs that are OUT or STALE in the source are imported with empty justification lists, preventing `recompute_all` from resurrecting them to IN.
IN import-agent-out-beliefs-not-resurrected
Beliefs marked OUT or STALE in the source are imported as OUT and are never resurrected by `recompute_all`, even when their justification antecedents are all IN in the local database.
IN import-agent-retraction-survives-recompute
After retracting an imported belief, `recompute_all()` must not resurrect it — the justification structure must not provide alternative paths to IN (issue #16 regression invariant).
IN invoke-claude-raises-when-binary-missing
`_invoke_claude()` raises `FileNotFoundError` if the `claude` CLI is not on `PATH`, rather than returning an error string — this is the one exception that `ask()` does not catch internally.
IN invoke-model-strips-claudecode-env
invoke_model() in llm.py strips the CLAUDECODE environment variable before all subprocess.run() calls, preventing recursive Claude Code entry from any module that uses the shared LLM interface (ask, derive, review).
IN kill-switch-cascade-is-reversible
Retracting `agent:active` cascades all agent beliefs to OUT via the inactive relay flipping IN; re-asserting `agent:active` reverses the cascade, restoring all beliefs to IN via the same BFS propagation
IN kill-switch-uses-outlist-not-antecedent
The `agent:inactive` node is placed in each imported belief's outlist (not antecedents) so that retracting `agent:active` cascades all imported beliefs to OUT, while per-belief retraction still works independently
IN list-negative-batches-at-40
`api.list_negative` splits candidates into batches of 40, so 120 keyword-matching nodes produce exactly 3 LLM calls.
IN list-negative-batches-at-50
`list_negative` splits candidate nodes into batches of approximately 50 for LLM classification, verified by the test suite asserting exactly 3 LLM calls for 120 candidates.
OUT list-negative-is-bounded-and-batch-scalable
The list-negative classification pipeline is both defensively bounded (two-stage keyword + LLM filtering with hallucination rejection and graceful malformed-output handling) and scalably partitioned (fixed batch size of ~50 candidates per LLM call), ensuring predictable resource usage and bounded LLM costs regardless of belief network size.
IN list-negative-is-defensively-bounded
The negative belief listing pipeline applies defense-in-depth: keyword pre-filtering narrows candidates before LLM classification, hallucinated node IDs are discarded against the actual network, and malformed LLM output falls back gracefully to zero count rather than raising.
IN list-negative-json-parser-tolerates-prose-preamble
The `list_negative` LLM classification response parser uses `re.finditer` to extract JSON objects from responses that include prose preamble, handling the common LLM pattern of prefacing structured output with natural language rather than requiring clean JSON.
IN list-negative-parser-is-fully-resilient
The list-negative LLM response parser handles all degradation levels: regex extraction recovers JSON objects from prose-laden responses, and completely unparseable output returns zero results gracefully rather than raising exceptions.
IN list-negative-uses-two-stage-classification
`list_negative` uses keyword pre-filtering against a hardcoded `NEGATIVE_TERMS` list (~50 words), then LLM classification via `ask._invoke_claude` to eliminate false positives.
OUT llm-belief-operations-span-creation-and-classification
All LLM-driven belief operations — creation via derive (with safety, completeness, and resource efficiency) and classification via list-negative (with defensive bounding and batch scaling) — share consistent defensive patterns across the complete belief quality lifecycle.
OUT llm-belief-pipeline-is-fully-quality-enforced
The system's complete LLM-driven belief pipeline — both generation (derive with safety, completeness, and coverage) and classification (list_negative with defensive bounding) — achieves fully code-enforced quality constraints at every stage, provided minimum-antecedent validation moves from prompt-only to code-enforced.
IN llm-driven-mutations-are-safely-bounded
LLM-driven belief derivation is safely bounded by defense in depth: the derive pipeline validates proposals with fail-soft filtering, Jaccard retraction guards, and environment stripping at the LLM boundary, while the API layer enforces atomic load/save with write-flag gating and dict-only returns at the persistence boundary — malformed or adversarial LLM output cannot corrupt the network.
OUT llm-fault-tolerance-is-multi-granular
LLM fault tolerance operates at two independent granularities: module-level fail-soft handling ensures entire operations degrade gracefully when the LLM is unavailable, while item-level batch fault isolation ensures individual failures within derive and review batches are contained without affecting other items in the same batch.
OUT llm-integration-fails-softly-across-modules
All LLM-facing modules apply consistent fail-soft error handling: the ask module always returns a string even when the LLM is unavailable, and the derive pipeline accumulates per-proposal errors rather than raising exceptions — no LLM failure path crashes the system.
OUT llm-integration-is-bounded-fail-soft-and-process-isolated
LLM integration achieves three independent safety properties across all modules: execution bounds (iteration caps and timeouts), fail-soft error handling (always returns usable results on failure), and process isolation (subprocess invocation with CLAUDECODE environment stripping to prevent recursive entry)
OUT llm-integration-is-defense-in-depth-across-layers
All LLM integration achieves defense-in-depth across two independent layers: application-level defensive bounding provides iteration caps, fail-soft error handling, Jaccard retraction guards, and hallucination filtering across all LLM-facing operations, while infrastructure-level process isolation executes all LLM calls through subprocess boundaries with CLAUDECODE environment scrubbing to prevent recursive invocation — ensuring safety at both the semantic and process boundaries.
OUT llm-integration-is-production-hardened
LLM integration achieves production-grade robustness across all dimensions: infrastructure-level safety through bounded execution, fail-soft error handling, and process isolation prevents runaway or recursive invocations, while operational-level coverage and fault tolerance ensures both derive (batch) and ask (interactive) paths complete successfully or degrade gracefully under all failure modes.
IN llm-mutations-are-bounded-end-to-end
LLM-driven belief derivation is bounded at every stage of the pipeline: input validation (fail-soft filtering, Jaccard retraction guard, environment isolation), atomic persistence (context-managed load/save), and output propagation (deterministic terminating BFS with lifecycle-aware traversal).
IN llm-prompt-always-via-stdin
Prompts are passed to LLM subprocesses via `subprocess.run(input=...)`, never as command-line arguments — avoiding shell injection and argument length limits.
IN llm-resolve-ollama-preserves-inner-colons
`resolve_model_cmd("ollama:model:tag")` splits on the first colon only, keeping `model:tag` intact as the Ollama model identifier.
IN llm-subprocess-isolation-prevents-recursion
All LLM subprocess invocations strip the CLAUDECODE environment variable to prevent recursive Claude Code entry, enforced centrally in invoke_model() and inherited by all LLM-facing modules (ask, derive, review).
IN llm-thinking-strip-ollama-only
Thinking-marker stripping (`Thinking...` / `...done thinking.`) is applied only to Ollama model output; Claude and Gemini output passes through unmodified even if it contains the same markers.
OUT multi-agent-reasoning-is-sound-and-scalable
The system provides both individually sound reasoning (deterministic, reversible, terminating truth maintenance) and safe multi-agent operation (isolated namespaces, reversible lifecycle control, clean architectural boundaries), enabling arbitrarily many agents without sacrificing correctness guarantees.
OUT multi-agent-revision-is-semantically-uniform
Multi-agent operation does not carve out exceptions to the universal revision semantics — agent beliefs undergo the same uniform revision (outlist defeat, contradiction backtracking, edge-case handling) as local beliefs because agent namespacing and relay pairs operate above the evaluation layer, not within it.
OUT multi-agent-safety-spans-all-layers
Multi-agent operation is safe across the full system: agent isolation prevents cross-contamination between namespaces with reversible lifecycle control, while data integrity is enforced across all three architectural layers through clean boundaries and snapshot persistence.
IN namespace-active-premise-invariant
When `namespace` is set, `add_node` auto-creates a `{namespace}:active` premise node and wires it as an antecedent into every namespaced justification; retracting that single premise cascades OUT every belief from that namespace.
IN namespace-is-colon-convention-with-auto-wiring
The namespace system is a colon-based convention with automatic infrastructure wiring: colon presence prevents double-prefixing, the `agent:id` format provides scoping, and node creation auto-wires a `{ns}:active` premise as an antecedent.
IN namespace-prefix-is-agent-colon
Covered by existing `namespace-prefix-is-colon-separated` and `import-agent-namespace-prefix`
IN namespace-prefix-is-colon-separated
Agent namespacing uses the format `agent_name:belief_id`; `_resolve_namespace()` skips prefixing any ID that already contains a colon, preventing double-prefixing of cross-namespace references
IN parse-review-defaults-to-passing
`parse_review_response` defaults missing fields to valid=True, sufficient=True, necessary=True, unnecessary_antecedents=[] — the LLM only needs to report failures explicitly; omission means the belief passed.
IN parse-review-response-never-raises
`parse_review_response` returns an empty list on any malformed input — bad JSON, non-list JSON, items missing `id` fields — rather than raising exceptions, with missing boolean fields defaulting to `True`.
OUT review-achieves-verified-fault-tolerance
The review pipeline's scoped mutation-safe operation combined with uniform fail-safe output achieves verified fault tolerance across all failure modes — batch failures, missing antecedent references, and malformed LLM responses.
IN review-and-contradictions-catch-orthogonal-errors
`review-beliefs` catches invalid reasoning within individual derivation steps (over-generalization, missing bridges, strength escalation) while `contradictions` catches incompatible facts across independently valid beliefs (absolute claims vs. documented exceptions) — the two commands are complementary with different cascade profiles (review: high cascade at depth, contradictions: low cascade at leaves).
IN review-batch-failure-is-silent-skip
When an LLM call fails for a review batch, the error is logged to stderr but the batch is skipped with no indication in the returned results; callers cannot distinguish "skipped due to error" from "no problems found."
OUT review-completes-llm-quality-lifecycle
The LLM-driven belief quality lifecycle is complete across all phases: creation via derive (safe, complete, efficient), classification via list-negative (bounded, batch-scalable), and quality evaluation via review (scoped to derived beliefs, mutation-safe, fault-tolerant) — covering belief genesis, categorization, and ongoing quality assessment with no unmonitored phase.
OUT review-driven-quality-lifecycle-is-fully-code-enforced
The complete LLM-driven quality lifecycle — creation via derive with structural validation and retraction guards, classification via list-negative with defensive bounding, and evaluation via review with scoped mutation safety — achieves full code enforcement of all quality constraints, but only when the derive pipeline's minimum antecedent requirement is enforced in code rather than prompt-only.
IN review-evaluates-direct-antecedents-only
`review-beliefs` presents each derived belief with only its direct antecedents to the LLM reviewer, not the transitive chain — producing occasional sufficiency false positives when support exists one level deeper, which must be triaged manually.
IN review-format-handles-missing-antecedents
`format_belief_for_review` renders `"(not found in network)"` for antecedent IDs that don't exist in the nodes dict rather than crashing, and returns an empty string for a nonexistent belief ID.
IN review-has-no-storage-dependency
The review module operates entirely on an in-memory `nodes` dict (from `export_network()`) and never reads from or writes to the database directly.
OUT review-is-read-only-and-fault-tolerant
The review module operates entirely on in-memory snapshots with no storage dependency, handles missing antecedent references with placeholder text rather than exceptions, and silently skips failed LLM batches — achieving fault-tolerant read-only operation across all failure modes.
IN review-multi-justification-disjunctive
The review prompt instructs the LLM that a belief is valid if ANY of its justifications is sound, matching Doyle's TMS disjunctive support semantics.
IN review-only-evaluates-derived-beliefs
`review_beliefs` filters out premises (nodes without justifications); only derived beliefs with at least one justification are sent for LLM review.
IN review-only-validates-derived-beliefs
The review pipeline filters out premises (nodes with empty justifications) before sending anything to the LLM; premises are never submitted for review validation.
IN review-output-is-uniform-and-fail-safe
Review response parsing defaults missing fields to passing, accepts only JSON arrays as valid input, and normalizes every result to a guaranteed six-key schema — producing uniform fail-safe structured output regardless of LLM response quality.
IN review-parse-defaults-fail-safe
`parse_review_response` defaults `valid`, `sufficient`, and `necessary` to `True`, so a missing or malformed field in LLM output never triggers a false alarm.
IN review-parse-defaults-safe
When the LLM response is missing fields, `parse_review_response` defaults to `valid=True`, `sufficient=True`, `necessary=True` — a parse error never generates a false-positive validity warning.
IN review-parse-requires-json-array
`parse_review_response` only accepts JSON arrays; a bare JSON object `{...}` is treated as unparseable and returns an empty list — the LLM must return a list even for single-item reviews.
IN review-pipeline-is-scoped-and-mutation-safe
The belief review pipeline restricts evaluation to derived beliefs only (premises excluded) and gates auto-retraction behind the dry-run flag, ensuring review operations are scope-limited and mutation-safe by default.
IN review-result-schema-is-normalized
Every result dict returned by `parse_review_response` is guaranteed to have exactly six keys (`id`, `valid`, `sufficient`, `necessary`, `unnecessary_antecedents`, `comment`) regardless of what the LLM returned, via normalization with safe defaults.
IN review-skips-premises
`review_beliefs` only sends beliefs with at least one justification to the LLM; premise nodes (empty justifications list) are excluded from review entirely.