persistence-backends
79 beliefs (64 IN, 15 OUT)
The persistence-backends topic covers how the reasons system stores and retrieves its belief network across two interchangeable storage backends: SQLite and PostgreSQL. This is a critical concern because the system's epistemic guarantees — atomic mutations, safe hypothetical reasoning, truth value preservation, and referential integrity — must hold regardless of which backend is in use. The core claim is that both backends are interchangeable (dual-storage-backends-are-interchangeable), providing equivalent safety through backend-appropriate mechanisms: SQLite uses context-managed load/save with write-flag gating, while PostgreSQL uses per-method transactions with composite-key multi-tenancy (atomicity-is-backend-independent, storage-layer-is-backend-agnostic-and-safe). This interchangeability extends to hypothetical reasoning, where SQLite discards speculative changes via write-flag gating and PgApi rolls back transactions (both-backends-support-safe-hypothetical-reasoning, pg-what-if-uses-transaction-rollback). The system is considered fully production-grade across both backends (storage-is-fully-production-grade-across-backends), with SQLite optimized via WAL mode and derived FTS5 indexes (storage-uses-wal-mode, storage-fts-is-derived-index) and PostgreSQL providing multi-tenant isolation via composite primary keys on every table (pg-uses-project-id-for-multi-tenancy, pgapi-multi-tenant-composite-keys).
The PostgreSQL backend, PgApi, is a SQL-native implementation that operates directly against the database with no in-memory Network object constructed (pgapi-no-in-memory-network, pgapi-is-sql-native-multi-tenant). It reimplements BFS propagation in application-level Python using JSONB containment queries against GIN indexes (pgapi-bfs-propagation-in-python, pg-uses-jsonb-with-gin-for-dependent-queries), and each public method constitutes a single committed transaction (pgapi-one-transaction-per-method). A notable design tension exists around referential integrity: antecedent and outlist references are stored as JSONB arrays without database-level foreign key constraints (pg-antecedent-refs-have-no-fk-constraints), so PgApi compensates with application-level validation that checks all referenced nodes exist before inserting justifications (pgapi-validates-refs-before-justification-insert, pgapi-enforces-referential-integrity-bidirectionally). The belief that database-level foreign keys would complete this picture is retracted (pgapi-referential-integrity-is-database-enforced), as is the claim of full defense-in-depth (pg-data-integrity-achieves-defense-in-depth), suggesting the system acknowledges this as an accepted gap rather than a resolved one. PostgreSQL routing uses a function-level early-return pattern where each API function checks for pg_conninfo and short-circuits to PgApi (pg-dispatch-is-function-level-early-return, all-api-functions-support-pg-dispatch), with psycopg imported lazily so the test suite works without Postgres dependencies (pg-psycopg-is-optional-dependency, pg-import-is-deferred).
The SQLite backend uses a full-snapshot persistence model: save replaces all rows and load constructs the complete network without incremental propagation (storage-save-is-full-replace, storage-load-bypasses-propagation, bootstrap-bypasses-incremental-propagation). Stored truth values are trusted on load without re-running propagation (storage-trusts-stored-truth-values), and justification ordering is preserved via AUTOINCREMENT rowid (justification-order-preserved-via-rowid, storage-justification-order-preserved). Lists like antecedents and outlist are stored as JSON text columns rather than normalized join tables (storage-uses-json-columns-for-lists), which is acceptable because SQLite always loads the full network. Schema evolution is handled pragmatically through try/except blocks that tolerate missing tables from older database versions (storage-handles-schema-evolution-via-try-except, storage-old-schema-compat), and the nogood counter is derived from existing data when the network_meta table is absent (storage-derives-counter-from-existing-nogoods). Each API function operates independently with no shared state or connection pooling (transaction-per-function).
Several retracted beliefs reveal the evolution of the system's self-understanding. The claim that PgApi is a complete SQL reimplementation achieving full behavioral parity is OUT (pgapi-is-complete-sql-reimplementation, pgapi-achieves-implementation-parity), as is the earlier claim of only partial API coverage (pgapi-partial-api-coverage), suggesting the reality lies between these extremes — substantial but not total parity. Similarly, broader claims that backend-independent guarantees extend to self-correction pipelines (backend-independent-self-correction), operational assurance (backend-agnostic-operational-assurance), and verified correctness (verified-correctness-extends-to-all-backends) are all retracted, indicating that while the core storage operations are backend-agnostic, higher-level system properties may still have backend-specific gaps. The retraction of persistence-is-snapshot-not-incremental and persistence-round-trip-is-lossless likely reflects refinement rather than fundamental disagreement, as their constituent claims remain active under more specific belief nodes. The testing infrastructure validates backend parity through a dedicated parity test suite (pg-test-suite-is-backend-parity) with per-test isolation via UUID-based project namespaces (pg-fixture-uses-uuid-isolation, pg-fixture-provides-per-test-isolation).
-
IN
all-api-functions-support-pg-dispatch
Every public API function (`export_markdown`, `import_json`, `import_beliefs`, `import_agent`, `sync_agent`, `hash_sources`, `check_stale`, `lookup`, `add_repo`, `list_repos`, `list_negative`) accepts `pg_conninfo`/`project_id` kwargs and routes to the corresponding `PgApi` method -
OUT
all-identification-is-deterministic-and-collision-free
All system-generated identifiers — dialectical artifact auto-IDs, unconditionally-recorded nogood records, and colon-based agent namespace prefixes — follow deterministic patterns that prevent collisions across all three identifier spaces: dialectical, contradiction, and multi-agent. -
OUT
all-identifiers-are-durable-across-persistence-boundaries
All system-generated identifiers are not only deterministic and collision-free at creation time but also durable across persistence boundaries — nogood IDs survive save/load cycles via the network_meta table's high-water mark, ensuring the collision-free property is maintained across the system's entire operational lifetime rather than just within a single session -
IN
atomicity-is-backend-independent
Both storage backends enforce atomic isolated operations through backend-appropriate mechanisms: the SQLite backend uses context-managed load/save with write-flag gating and per-function transaction scope, while PostgreSQL uses per-method transactions with composite-key multi-tenancy — achieving the same transactional guarantee at different architectural levels -
OUT
backend-agnostic-operational-assurance
The system's comprehensive operational assurance — spanning temporal self-correction, end-to-end reliability, and external control — holds identically across both storage backends with equivalent safety guarantees, provided PgApi achieves full API parity with the SQLite path. -
OUT
backend-independent-self-correction
The complete self-correction pipeline — contradiction resolution through dependency-directed backtracking, staleness detection through source hash comparison, and ongoing belief currency management — operates identically across all storage backends, making the system's self-maintaining properties a deployment-independent guarantee rather than a backend-specific capability. -
IN
bootstrap-bypasses-incremental-propagation
Both persistence loading and import construct the full node graph before truth maintenance — load trusts stored truth values and skips propagation entirely, while import adds all nodes then propagates via recompute_all — sharing a bulk-construction pattern that avoids per-node incremental propagation. -
IN
both-backends-support-safe-hypothetical-reasoning
Both storage backends enable hypothetical what-if reasoning without permanent mutation: PgApi performs real mutations inside a transaction then rolls back, while the in-memory backend uses write-flag gating to discard speculative changes after analysis -
IN
data-model-uses-string-enums
Both `Justification.type` ("SL"/"CP") and `Node.truth_value` ("IN"/"OUT") are plain strings, not Python enums; consumers must validate values themselves as invalid states like `"MAYBE"` or `"XYZ"` are representable. -
IN
dependents-survive-storage-roundtrip
After `Storage.save()` followed by `Storage.load()`, the loaded network's dependents index passes `verify_dependents()` with no errors. -
IN
dual-backend-atomicity-is-feature-complete
Both storage backends provide atomic isolated operations with complete API coverage — the SQLite backend's context-managed mutations and PgApi's per-method transactions cover the same full set of operations -
IN
dual-storage-backends-are-interchangeable
Both SQLite and PostgreSQL backends can be used interchangeably for any operation, with identical safety guarantees through backend-appropriate mechanisms and complete API surface coverage, so applications can switch backends without behavioral changes. -
IN
export-markdown-pg-reconstructs-network
The `export_markdown` PostgreSQL path reconstructs a full `Network` object from `export_network()` output — creating Node/Justification objects and wiring the dependents index — because the markdown exporter requires a wired dependency graph, not a flat dict. -
IN
initialization-is-safe-and-path-independent-across-backends
System initialization produces identical belief states regardless of both initialization path (stored-state bootstrap vs deterministic reasoning) and storage backend (SQLite vs PostgreSQL providing equivalent safety through backend-appropriate mechanisms), eliminating all bootstrap-time variation -
IN
justification-order-preserved-via-rowid
Justification insertion order is preserved across save/load cycles using `AUTOINCREMENT` rowid and `ORDER BY rowid` on read, which matters because justification priority affects truth maintenance. -
OUT
persistence-is-snapshot-not-incremental
The storage layer operates as a full snapshot: save replaces all rows, load trusts stored values without re-propagation, and the dependents index is rebuilt from scratch -
OUT
persistence-round-trip-is-lossless
The save/load round trip preserves all network state faithfully: snapshot persistence captures the full graph, stored truth values are trusted without re-propagation, justification insertion order is preserved via rowid, and outlist relationships survive serialization. -
IN
pg-antecedent-refs-have-no-fk-constraints
Antecedent and outlist references in `rms_justifications` are JSONB arrays without foreign key constraints; nonexistent referenced nodes default to truth value OUT via `truth_cache.get(a, "OUT")`. -
IN
pg-api-what-if-never-mutates
`PgApi.what_if_retract()` and `what_if_assert()` are read-only operations — they return cascade analysis (changed/went_in/went_out) without modifying database state, verified by checking `get_status()` before and after -
IN
pg-cleanup-covers-five-tables
Fixture teardown deletes from exactly `rms_propagation_log`, `rms_justifications`, `rms_nogoods`, `rms_network_meta`, and `rms_nodes` — adding a new `project_id`-scoped table without updating the fixture will leak test data. -
IN
pg-conninfo-accepts-flag-envvar-or-both
PostgreSQL connection is configured via `--pg`/`--project-id` CLI flags or `REASONS_PG_CONNINFO`/`REASONS_PROJECT_ID` environment variables, with CLI flags taking precedence over env vars — `_backend_kwargs(args)` handles the dispatch. -
OUT
pg-data-integrity-achieves-defense-in-depth
PgApi's data integrity achieves defense-in-depth through both application-level enforcement (write-time referential validation and outlist-aware dependent queries spanning antecedent and outlist references) and comprehensive input validation at all system boundaries, but full defense-in-depth requires database-level foreign key constraints to provide a second independent enforcement layer -
IN
pg-dispatch-is-function-level-early-return
PostgreSQL routing uses a function-level early-return pattern — each API function checks `pg_conninfo` and short-circuits to `_pg_dispatch`, which instantiates PgApi and calls the matching method via `getattr` — rather than using abstract base classes, subclassing, or factory patterns. -
IN
pg-dispatch-requires-project-id
When `pg_conninfo` is provided (via CLI `--pg` flag or `REASONS_PG_CONNINFO` env var) without a corresponding `project_id`, the system calls `sys.exit()` rather than proceeding with an incomplete configuration -
IN
pg-extra-required-for-postgres
PostgreSQL support requires installing the `pg` or `test-pg` optional extra (`psycopg[binary]>=3.1`); it is not available in a bare install. -
IN
pg-fixture-provides-per-test-isolation
Each pg test receives an isolated `PgApi` instance via the `pg_api` fixture, which creates a fresh project namespace (UUID) and tears down after each test, ensuring zero shared state between tests. -
IN
pg-fixture-uses-uuid-isolation
Each `pg_api` test fixture partitions its data by a unique UUID v4 `project_id` rather than using transactions or ephemeral databases, enabling safe concurrent test execution against a shared database. -
IN
pg-import-is-deferred
`PgApi` is imported inside the `pg_api` fixture body, not at module top-level, so the test suite loads without Postgres dependencies when running SQLite-only tests. -
IN
pg-keyerror-includes-missing-node-id
When `add_node` references a nonexistent node in `sl=` or `unless=`, PgApi raises `KeyError` with the missing node ID in the exception message — tested via `pytest.raises(KeyError, match="ghost")` — enabling callers to identify which reference is dangling. -
OUT
pg-multi-tenancy-is-referentially-complete
PgApi's multi-tenant isolation with composite primary keys and application-level BFS propagation prevents all cross-project data leakage and ensures consistent truth maintenance — unless antecedent references stored as JSONB arrays lack foreign key constraints, allowing phantom node references within a project. -
IN
pg-multi-tenancy-via-project-id
`PgApi` isolates beliefs by `project_id` so identical node IDs in different projects store independent data with no cross-contamination — each test gets a unique UUID project -
IN
pg-psycopg-is-optional-dependency
`psycopg` (v3) is soft-imported with fallback to `None`; `_require_psycopg()` raises `ImportError` with install instructions at `PgApi` construction time, mirroring the `mcp_client.py` lazy-guard pattern. -
OUT
pg-reimplements-network-in-sql
`PgApi` reimplements the in-memory `Network`'s algorithms (BFS propagation, entrenchment scoring, nogood resolution, dialectical operations) directly in SQL rather than delegating to the `Network` class. -
IN
pg-search-includes-one-hop-neighbors
PostgreSQL full-text search via `plainto_tsquery` includes 1-hop neighbor expansion — direct antecedents and dependents of matching nodes are included in results, providing richer context than exact-match-only search. -
IN
pg-teardown-rollback-before-delete
The `pg_api` fixture calls `conn.rollback()` before cleanup deletes to recover from any failed-transaction state left by the test, preventing cleanup failures from cascading. -
IN
pg-test-suite-is-backend-parity
`test_pg.py` is a parity test suite: every behavior tested has a corresponding expected behavior from the SQLite backend, validating that PgApi returns the same dict shapes and enforces the same invariants to ensure backend interchangeability. -
IN
pg-unsupported-params-raise-not-implemented
When PgApi-routed commands receive unsupported parameters (e.g., search with `depth != 1`, list with `challenged`/`min_depth`, add with `namespace`), the dispatch raises `NotImplementedError` with a clear message rather than silently ignoring the parameter. -
IN
pg-uses-jsonb-with-gin-for-dependent-queries
PgApi stores antecedents and outlist as JSONB arrays with GIN indexes, enabling per-operation dependent lookups via `@>` containment queries — unlike the SQLite backend which loads the full network, PgApi queries relationships incrementally. -
IN
pg-uses-project-id-for-multi-tenancy
Every `PgApi` query includes `project_id` in its WHERE clause, providing multi-tenant isolation with no cross-project queries and composite primary keys `(id, project_id)`. -
IN
pg-what-if-is-safely-simulated
PgApi's what-if operations achieve safe simulation: mutations are performed against real PostgreSQL data for accurate cascade analysis, then rolled back within a transaction to guarantee zero persistent side effects — combining fidelity with safety. -
IN
pg-what-if-uses-transaction-rollback
`what_if_retract`/`what_if_assert` perform real mutations inside a transaction, collect cascade effects, then rollback — reusing the propagation engine without duplicating logic. -
OUT
pgapi-achieves-implementation-parity
PgApi achieves full behavioral parity with the in-memory Network implementation: it reimplements the core algorithms (entrenchment scoring, nogood resolution, BFS propagation, dialectics) in SQL with BFS propagation executed in application-level Python. -
IN
pgapi-bfs-propagation-in-python
PgApi implements BFS propagation in application-level Python (not stored procedures), using JSONB containment queries (`@>`) against GIN indexes to find dependents -
IN
pgapi-enforces-referential-integrity-bidirectionally
PgApi enforces referential integrity in both directions: write-time validation checks all antecedent and outlist IDs exist before inserting justifications, while read-time dependent discovery queries both antecedent and outlist JSONB containment to find all affected nodes. -
IN
pgapi-find-dependents-queries-outlist
PgApi's `_find_dependents` queries both `antecedents @>` and `outlist @>` JSONB containment, correctly enqueuing outlist-dependent nodes for re-evaluation — a capability the in-memory Network historically lacked until PR #31. -
IN
pgapi-has-no-retry-logic
`PgApi` uses one connection per instance with no retry on transient database errors; callers are expected to handle `psycopg` exceptions from connection failures or serialization conflicts. -
OUT
pgapi-is-complete-sql-reimplementation
PgApi is a complete SQL-native reimplementation of the in-memory Network: it re-implements core algorithms (BFS propagation, entrenchment scoring, nogood resolution) directly in SQL with multi-tenant composite keys and per-method transaction semantics, achieving full behavioral parity. -
IN
pgapi-is-sql-native-multi-tenant
PgApi operates as a SQL-native multi-tenant implementation: all operations execute directly against PostgreSQL with no in-memory Network object constructed, composite primary keys on all tables provide project-level isolation, and each public method is a single committed transaction. -
IN
pgapi-multi-tenant-composite-keys
All PgApi tables use composite primary keys `(id, project_id)` for multi-tenancy, with JSONB columns (not TEXT) for antecedents, outlist, and metadata -
IN
pgapi-no-in-memory-network
`PgApi` executes all operations as direct SQL against PostgreSQL — no in-memory `Network` object is ever constructed, enabling concurrent writers -
IN
pgapi-one-transaction-per-method
Each PgApi public method is a single PostgreSQL transaction that commits or rolls back at the end; __exit__ rolls back on exception -
OUT
pgapi-partial-api-coverage
PgApi implements core operations (add/retract/assert/search/nogood/explain) but defers simulation, dialectics, namespace support, import/export, and maintenance operations to future work -
OUT
pgapi-referential-integrity-is-database-enforced
PgApi achieves complete referential integrity: application-level bidirectional enforcement (write-time validation and outlist-aware dependent querying) is complemented by database-level foreign key constraints, preventing orphaned justification references even under direct SQL manipulation outside the application layer. -
IN
pgapi-validates-refs-before-justification-insert
PgApi's `_validate_refs` checks all antecedent and outlist node IDs exist in `rms_nodes` before inserting a justification, providing application-level referential integrity that compensates for JSONB arrays' inability to enforce foreign key constraints. -
IN
pgapi-what-if-uses-transaction-rollback
PgApi's `what_if_retract` and `what_if_assert` wrap BFS cascade analysis inside a database transaction with `try/finally` guaranteed ROLLBACK, achieving read-only simulation by never committing the exploratory mutations. -
OUT
references-are-durable-across-persistence-and-evolution
All system-generated identifiers survive both persistence boundaries (save/load cycles, cross-session durability via high-water marks) and format evolution boundaries (parser versioning, schema migration, forward-compatible metadata) — references remain valid and resolvable across time and system versions. -
IN
require-sqlite-blocks-pg-for-guarded-commands
`_require_sqlite` enforces that certain subcommands (e.g., `hash-sources`) cannot run against a Postgres backend, checking both CLI flags and environment variables and calling `sys.exit()` if PG is detected -
IN
storage-derives-counter-from-existing-nogoods
When loading a database without the `network_meta` table (pre-fix schema), `Storage.load()` derives `_next_nogood_id` from the max ID of existing nogoods rather than defaulting to 1, enabling backward-compatible upgrades -
IN
storage-fts-is-derived-index
The FTS5 full-text search index is rebuilt from scratch during every `save()`; it is a derived index, never the source of truth. -
IN
storage-fts-rebuilt-on-every-save
The `nodes_fts` full-text index is deleted and rebuilt from scratch on each `save()`, guaranteeing consistency with the `nodes` table at the cost of full-reindex overhead. -
IN
storage-handles-schema-evolution-via-try-except
Missing tables from older database versions (`repos`, `network_meta`) are handled by swallowing exceptions during `load()` rather than formal migrations — a backward-compatibility substitute that silently degrades. -
IN
storage-is-fully-production-grade-across-backends
Both storage backends achieve fully production-grade operation — concurrent access optimization (WAL mode, derived FTS5 indexes), equivalent safety guarantees (atomic isolated mutations through backend-appropriate mechanisms), and multi-tenant isolation — when both backends provide complete API coverage. -
IN
storage-justification-order-preserved
Justifications are inserted in list order and loaded via `ORDER BY rowid`, preserving the ordering that determines which justification is evaluated first during truth computation. -
IN
storage-layer-is-backend-agnostic-and-safe
Both storage backends provide equivalent safety guarantees through backend-appropriate mechanisms: atomic isolated mutations (context-managed load/save for SQLite, per-method transactions for PgApi) and safe hypothetical reasoning (write-flag gating for SQLite, transaction rollback for PgApi), making safety properties independent of backend choice. -
IN
storage-lists-as-json-columns
Antecedents, outlist, nogood node sets, and metadata dicts are stored as JSON text columns rather than normalized join tables, simplifying the schema at the cost of individual field queryability. -
IN
storage-load-bypasses-add-node
`load()` assigns nodes directly to `network.nodes` and calls `_rebuild_dependents()` afterward, deliberately skipping `add_node()` to avoid triggering truth maintenance propagation during state restoration. -
IN
storage-load-bypasses-propagation
`load()` constructs nodes directly into `network.nodes` rather than calling `add_node`, so truth maintenance propagation does not fire during deserialization. -
IN
storage-no-partial-load
`Storage.load()` either returns a complete `Network` with all state or fails; there is no streaming, lazy-loading, or partial deserialization. -
IN
storage-old-schema-compat
`load()` tolerates missing `network_meta` and `repos` tables via silent `try/except`, supporting databases created before those tables were added; `next_nogood_id` is derived from existing IDs as a fallback. -
IN
storage-optimizes-concurrent-access-and-search
The storage layer optimizes for both concurrent access (WAL mode enables non-blocking reads during writes) and full-text search (derived FTS5 index rebuilt from scratch on every save guarantees consistency), making the persistence layer production-ready for multi-reader workloads with search capability. -
IN
storage-round-trip-preserves-dependents
Covered by existing belief `persistence-round-trip-is-lossless` — dependents are part of the persisted state -
IN
storage-save-is-atomic-snapshot
Duplicate of existing `storage-save-is-full-replace` and `mutation-pipeline-is-atomic-snapshot` -
IN
storage-save-is-full-replace
`Storage.save()` deletes all rows from every table before re-inserting the entire network; there is no incremental or differential update path. -
IN
storage-trusts-stored-truth-values
`load()` trusts the stored `truth_value` without re-running propagation, making the database the source of truth for node status. -
IN
storage-uses-json-columns-for-lists
Antecedent IDs, outlist IDs, nogood node sets, and node metadata are stored as JSON text columns rather than junction tables — acceptable because the code always loads the full network, never queries relationships via SQL joins. -
IN
storage-uses-wal-mode
SQLite connections enable WAL mode on initialization, allowing concurrent readers without blocking writes. -
IN
transaction-per-function
Every API function opens the database, does its work, and closes — no shared state, no connection pooling, no long-lived sessions; each invocation is fully independent. -
OUT
verified-correctness-extends-to-all-backends
Verified production correctness — spanning all belief origins with deterministic state trajectories and full maintenance loop observability for indefinite independent verification — extends identically across both SQLite and PostgreSQL storage backends with no backend-specific safety gaps. -
IN
write-false-prevents-persistence
Functions using `_with_network(write=False)` can mutate the in-memory network (as `what_if_retract` does) but changes are never saved to SQLite; write-or-not is declared upfront and never conditional.