Changelog¶

What changed in each release, newest first. The notes are written for the person deciding whether to upgrade; the rules are in docs/launch/changelog-style.md. Lorebase follows semantic versioning. Before 1.0, any minor version may change an interface, and the notes say so.

0.3.1 - 2026-11-06¶

A patch for the first contact with the project: the published images run on arm64, the PyPI packages carry their license and links, a Scribe worker keeps all its slots when the database fails while it records a job and gets a liveness check, the console scripts answer --help, and the quickstart commands point at the Keeper they started. No migration and no breaking change. Three new settings, all optional: scribe.worker.outcome_retry_delay_s, scribe.worker.liveness_file and scribe.worker.liveness_interval_s. The lines link the documentation of each change: the work landed directly on main, without pull requests to link.

Added¶

Scribe liveness (Scribe): with scribe.worker.liveness_file set, the job loop rewrites that file every scribe.worker.liveness_interval_s (10 s) with its live slots, the jobs running and the time it last reached the store, and python -m lorebase_scribe.liveness FILE reads it, with the standard library only, and exits 1 with the reason (no file, a stale file, no live slot or, with --max-store-age, a store not reached). The compose scribe service writes /tmp/scribe-liveness.json and runs that command as its healthcheck; the scribe values of the chart show the exec probe
Metric lorebase.scribe.worker.slots (gauge, state live or lost) and the status unrecorded of lorebase.scribe.jobs (metrics)
lorebase-herald --help prints the variables an MCP host sets (LOREBASE_URL, LOREBASE_API_KEY, LOREBASE_DOMAIN, LOREBASE_BRANCH, LOREBASE__HERALD__TRANSPORT) and the documentation link, --version the version; at start Herald logs the transport, the Keeper it calls and the default Domain to stderr, so a host pointed at the wrong Keeper sees it before the first call (MCP)
make up prints the export LOREBASE_URL=... LOREBASE_DOMAIN=supply line for the CLI and the env entry for an MCP host over stdio, with the port set in .env; make shell-env prints the export line again, for eval "$(make shell-env)" (quickstart)
make up checks every host port the chosen profiles publish before it starts anything and names the variable to change, and refuses to start while a *_BIND value in .env would publish the Keeper of the stack, which runs with authentication off, beyond this machine; KEEPER_PUBLISH_INSECURE=true, in .env or the shell, starts it anyway with a warning (quickstart)
make env sets HERALD_DOMAIN=supply while it is unset, so the first call of an MCP host against make up PROFILES=mcp needs no Domain of its own
server.json at the repository root describes Herald for the official MCP registry as io.github.aantenore/lorebase-herald, with the PyPI package lorebase-herald (run with uvx) and the image ghcr.io/aantenore/lorebase, which now carries the io.modelcontextprotocol.server.name label the registry checks; the entry is submitted after a release (releasing)
MCP server: host configurations that run Herald from PyPI with uvx lorebase-herald, as one claude mcp add command, a Cursor .cursor/mcp.json entry and a VS Code code --add-mcp command
Comparison: DataHub, the Fabric IQ ontology MCP server, Semantica, Doltgres with the Dolt MCP server, TypeDB, mnem and RDF triple stores, each with a linked source checked on 2026-11-06, and a Standards section: LinkML import and export are supported; RDF, OWL, SHACL and JSON-LD are not native and are reached through LinkML's generators
Images ghcr.io/aantenore/lorebase, lorebase-ml and lorebase-web for linux/arm64 as well as linux/amd64: each platform is built on a runner of its own architecture, and the release joins them into one multi-arch index, which it signs and attests with the SBOM (releasing)
PyPI metadata of the ten packages: license Apache-2.0 with the license text in every sdist and wheel, a README, author, keywords, classifiers and links to the repository, documentation, changelog and issues; scripts/release_tool.py check fails when a package lacks them, and also checks server.json (its versions, its name and the image label)

Changed¶

Scribe: an outcome the store refuses to take (a dropped connection, a statement timeout) is written once more after scribe.worker.outcome_retry_delay_s (1 s); when that fails too the run counts unrecorded and the job runs again once its lock goes stale, as a lost job does (Scribe)
README and the documentation home open with who Lorebase is for and one MCP call with the JSON answer and its Revision; the status note follows, and the 30-second demo stays the second screen
Quickstart: the commands take the Keeper URL from the line make up prints (eval "$(make shell-env)") instead of a fixed http://localhost:8080; the port table and the port troubleshooting row list GUIDE_HOST_PORT (8095), HERALD_HOST_PORT (8091) and the scale-out ports; the MCP section sets HERALD_DOMAIN next to HERALD_ALLOW_ANONYMOUS; the semantic search example writes query.json under ${TMPDIR:-/tmp}
CONTRIBUTING.md, the README and the validation and demo pages state the CI policy as it is: the checks run on demand and for release tags, not on every pull request
Kubernetes and deployment say that the 0.3.0 images are linux/amd64 only and that 0.3.1 and later are published for linux/amd64 and linux/arm64
Releasing: a section's heading reads X.Y.Z - unreleased until the tag, whose date then replaces unreleased; the page follows the workflow's jobs and adds the MCP registry step
The Release workflow runs the CI workflow on the tagged commit first; images, the GitHub release and PyPI wait for it, so a tag whose checks fail publishes nothing. After the release it publishes the documentation site built from the tag (the github-pages environment must allow v* tags to deploy)
The compose defaults of Postgres (pgvector/pgvector:0.8.7-pg16), Ollama, LiteLLM, the OpenTelemetry collector, Jaeger, Prometheus, Grafana and the OIDC mock, and the Helm default of Ollama, are pinned by version and digest instead of latest, main-stable or pg16; the *_IMAGE variables still override them, and tests/tooling/test_dockerfile_pins.py checks the pins
CI: the integration job may run 35 minutes instead of 20, and the BM25 integration job is part of the aggregate CI check
The release jobs no longer keep the checkout token on the runner
.env.example sets POSTGRES_IMAGE, OLLAMA_IMAGE, LITELLM_IMAGE, OTEL_COLLECTOR_IMAGE, JAEGER_IMAGE, PROMETHEUS_IMAGE and GRAFANA_IMAGE to the version and digest pins of deploy/compose/docker-compose.yaml instead of floating tags, so a .env created by make env keeps the pinned images.

Fixed¶

A Scribe worker slot ended for good, and nothing reported it, when the database failed while the worker recorded a job's outcome: with the default concurrency the process stayed up with one slot, or none, while the queue grew. The slot now stays on, a slot task that ends with an error is logged and started again after scribe.worker.poll_interval_s, and lorebase.scribe.worker.slots shows it (Scribe)
lorebase with no server configured said pass base_url or set LOREBASE_URL, an argument of the Python client the command line does not have; it now names the three ways a person has (--url, LOREBASE_URL, or url in a profile of the config file) and the path of that file
lorebase-herald --help and --version started the stdio server, printed nothing and exited 0 when the host closed the pipe
The audit chain heads and the merge attempts queries used a DISTINCT ON spelling that SQLAlchemy 2.1 deprecates (an SADeprecationWarning in every integration run); they use the dialect's distinct_on, and the test suite fails on any such warning from now on
The 30-second demo excerpts in the README, the documentation home and manual setup quoted Revision ids that examples/merge_demo.py no longer prints; they are regenerated, and tests/tooling/test_merge_demo.py now fails when a quoted line differs from the output
Stale documentation: the comparison listed LinkML import and export as roadmap work, the FAQ named the 1M-Record baseline as missing and left out the MySQL/MariaDB and SQL Server Sources, and examples/README.md linked a quickstart anchor that moved; tests/tooling/test_docs_links.py now checks the links and anchors of the README, CONTRIBUTING.md, SECURITY.md and examples/README.md
scripts/quickstart.sh up resolves the variables of its exposure and port checks the way docker compose does: a variable set in the shell wins over the environment file, then the default. KEEPER_BIND=0.0.0.0 KEEPER_ALLOW_INSECURE=true make up with a loopback .env no longer starts an exposed Keeper with authentication off, and KEEPER_PUBLISH_INSECURE set in the shell (true or false) wins over the file.

Not done yet¶

The chart wires no probe for Scribe by default; the scribe values show the exec probe to set.
The MCP registry entry (server.json) is submitted by hand after the release; the registry lists nothing until then.

Upgrade notes¶

No migration. Rebuild or pull the images: the compose scribe service gains a healthcheck that reads the liveness file, with nothing to configure. The new settings have defaults; set scribe.worker.liveness_file and an exec probe to give another deployment the same check.

0.3.0 - 2026-10-30¶

Knowledge that stays correct: a reconciliation queue for contradictions between Sources, corrections that survive Source syncs, document versions, ingestion modes and stewards, Traits and lenses in search and context packs, the merge of text values, and the resolution of words to schema and values with consolidation from use and coverage signals; the web app gains a Reconciliation screen and a passage-by-passage Conflict resolver. Pre-1.0: this minor version changes interfaces; read the breaking changes before upgrading.

Ten migrations (0038 to 0047). New configuration sections keeper.reconciliation, keeper.stewards, scribe.reconciliation, scribe.consolidation, store.values, store.resolve_misses, store.coverage, search.resolve and read.coverage, and new keys store.resolve_misses.caller_key, store.storage.hnsw_m, store.storage.hnsw_ef_construction, search.semantic.iterative_scan and search.semantic.max_scan_tuples. New endpoints POST /domains/{d}/resolve and /domains/{d}/reconciliation; responses add coverage to POST /query and POST /resolve, semantic_truncated to POST /search, can.settings to GET /domains/{d}, and source_key, version, replaces and status to documents. A Source sync no longer overwrites a value someone corrected on the target, and automatic merges into a protected branch wait for steward and role approvals (below, Breaking changes). The lines link the documentation of each change: the work landed directly on main, without pull requests to link.

Breaking changes¶

Source syncs no longer put back a value someone corrected on the target: an auto_when_clean Source that merged silently stops at an open Proposal with a DV1 or DV2 Conflict when upstream changes such a value, and on its first run after migration 0040. Set local_edits: source_wins on the Source to keep the old behavior (below, Changed and Upgrade notes).
Source syncs with auto_when_clean or auto_with_authority and extraction in direct mode no longer merge into a protected branch whose rules include required_approvals_from_stewards or required_roles; the Proposal stays open (merge_blocked: approvals, direct_blocked: approvals). Installations without those rules see no change (below, Changed).
A text Attribute edited in different sentences on two Branches merges automatically (resolver policy:text_merge) instead of raising a Conflict. Set text_merge: false in the Domain's merge_policy setting to keep the Conflict (below, Changed).
Semantic search reads the HNSW index at any size: unfiltered and broad-filter lists are approximate where they were exact (recall 0.92 to 0.99 at 300,000 vectors); a one-Kind filter stays exact (below, Changed).

Added¶

Resolution of words to schema and values (design 8.9, AC-39). POST /domains/{d}/resolve {text, kinds?, levels?, query_vector?, limit?, branch?, at?} returns ranked candidates the words of a text may stand for: Kinds, Attributes (by name or alias, with their type) and stored values with a ready-to-use where predicate and the count of Records holding them, each with its level (exact, synonym, trigram, embedding), score and the words matched, plus the words nothing matched. It never interprets the question, runs a query or calls a model; the embedding level only uses a caller's query_vector when no other level matched. Candidates never name a Kind or Attribute the caller may not read. Herald lorebase_resolve, CLI lorebase resolve TEXT, Python client records.resolve (sync and async). Settings search.resolve (agents).
Value index: mat_values holds per branch the distinct short values of selected Attributes, kept by triggers on mat_records in the writing transaction (migration 0045, additive). Enums are selected always, plain strings while their distinct values stay few (store.values); the Scribe value_index pass (scribe.scheduler.value_index_interval_s, default 300 s) and lorebase-scribe value-index DOMAIN [--rebuild] decide and backfill the selection. Sensitive and vault values are never indexed. A pass that changes the selection fences only its own Domain's writers (a transaction advisory lock taken shared by the triggers), for at most store.values.pass_timeout_ms; a schema change that makes an Attribute ineligible removes its values in the same commit. At 100,000 Records resolution answers in 11.2 ms at p95 (levels 1-2); the triggers cost about 90 µs per statement where nothing is held and 0.3 ms per updated Record whose held values change (performance, operations).
Value index (design 8.9): an Attribute's distinct-value limit now holds between value_index passes. The limit is max_values: hard_max_distinct for enums and resolvable: true, max_distinct × hysteresis for plain strings. The write that takes a branch past it marks the Attribute overflowed in the same transaction: resolution stops returning its values at once, later writes stop indexing it, and the next pass drops it. A schema commit or merge on the default branch applies a new limit in the same commit. Reads also skip a branch past its limit that is not yet marked.
Migration 0047 adds mat_value_sizes, the per-branch count of held values kept by the value triggers, and max_values and overflowed to mat_value_attributes. Passes read the sizes instead of counting mat_values. On upgrade, existing selections get hard_max_distinct and their sizes are counted; the first pass of each Domain counts its sizes again and sets its own limits. Trigger cost: about 90 µs per statement where nothing is held, as before; 0.3 to 0.4 ms per updated Record whose held values change, against 0.29 ms before (measured with tests/perf/resolve_bench.py on a loaded machine; performance).
Schema: aliases on Kinds and Attributes, value_aliases (stored value to other words) and resolvable: true|false on Attributes; absent by default, so existing schemas keep their hashes. Validation refuses resolvable and value_aliases on types other than string and enum (alone or in arrays), on sensitive and on link Attributes, and value aliases of an enum that name no value (schema.attribute_flag_invalid). The web Modeler keeps them on round trips.
Herald lorebase_query now documents counting with "aggregate": {"group_by": attr} and points to lorebase_resolve.
Consolidation from use (design 8.10, AC-40): with store.resolve_misses.enabled (off by default), Keeper keeps the phrases POST /resolve matched to nothing per Domain, branch and phrase (counts, first and last seen, digests of distinct callers; migration 0046), off the request path and only for head reads of callers holding search.resolve.misses.permission Domain-wide; requests opt out with record_misses: false, and searches narrowed by kinds or levels are not recorded. One caller adds and holds a bounded number of phrases; a full Domain evicts its least missed phrase. Phrases expire after retention_days (30) and go with their Domain, and so do the vocabulary suggestions quoting them (their share removed, a suggestion left without phrases deleted, decided or not). A Scribe vocabulary_consolidation pass (scribe.scheduler.consolidation_interval_s, CLI lorebase-scribe consolidate DOMAIN) turns frequent misses into schema suggestions of the new types value_alias, attribute_alias, kind_alias (near values and names; never for sensitive, vault, derived or non-resolvable Attributes, never a phrase that already names something) and, with scribe.consolidation.llm.enabled, attribute_description (task schema_consolidation; the model sees names and phrases only). They are reviewed like other suggestions; accepting one opens a schema change Proposal. Vocabulary suggestions have their own cap (schema.suggestions.max_vocabulary_tracked) so they cannot crowd out extraction's; they are withheld from reviewers who may not read their Kind's Records or Attribute's values; an alias that would name two things is refused at acceptance. lorebase-scribe misses purge DOMAIN deletes a Domain's misses and vocabulary suggestions.
Coverage signals (design 8.11, AC-41): POST /query and POST /resolve answer with a coverage block (coverage: false leaves it out; default read.coverage.enabled): the share of Records missing each Attribute used, Source freshness (last successful sync, stale_after exceeded, Records unseen), the phrases unknown in the Domain, and one note per signal. It reads per-Kind statistics of the default branch head (mat_coverage, migration 0046) counted by a Scribe coverage pass (scribe.scheduler.coverage_interval_s, CLI lorebase-scribe coverage DOMAIN) from one snapshot per Kind, Records unseen by Sources recounted every pass; the last successful run of a Source is one probe of the partial index ix_source_runs_last_succeeded; masked Attributes and Kinds outside the caller's scope are never described, and other branches and Revisions get no statistics. The two lookups take p95 3 ms at 100,000 Records and 600,000 Source runs.
Python client Coverage on QueryResult and ResolveResult, coverage on QueryRequest and ResolveRequest, record_misses on ResolveRequest, the vocabulary suggestion types and attribute, value, description on SchemaSuggestion; CLI lorebase query and lorebase resolve print the coverage notes; Herald lorebase_query and lorebase_resolve end with coverage: lines.
Web app, Review > Schema suggestions: vocabulary suggestions with the alias, the Attribute, the value they stand for or the proposed description, and the missed phrases as evidence; their Documents cell stays empty, as they come from no document.
store.storage.hnsw_m (default null: pgvector's 16) and store.storage.hnsw_ef_construction (default 128; pgvector's own default is 64) set the build parameters of the HNSW index of embeddings, for a new installation (migration 0001) and for indexes storage apply builds later. storage apply does not rebuild an existing index: installations from before this release keep ef_construction 64 until they rebuild it with ALTER INDEX ... SET and REINDEX INDEX CONCURRENTLY (operations). Measured at 300,000 vectors, ef_construction 128 raises mean recall@20 from 0.910 to 0.988 unfiltered and from 0.964 to 0.989 with a 40% filter, same index size, build 66 s instead of 50 s. Documentation now says the semantic list is approximate and gives these figures (search).
Incremental Sources and upstream deletions (design 9.1). Each run of an incremental Source records the Records it read, changed or not (source_written.last_seen_run and last_seen_at, migration 0043, additive). The new Source setting stale_after (ISO 8601 duration, default null: off; incremental Sources only) makes the reconciliation sweep raise a stale candidate with signal source_unseen for each Record the Source owns that no run has read for longer; nothing is deleted. Such a candidate takes keep_current or a dismissal (another choice answers 422) and goes obsolete once a run sees the Record again (source_seen). scribe.reconciliation.stale.source_unseen (default true, Domain override allowed) turns the check off (Scribe).
Web app, Modeler: the Source editor sets stale_after; the Reconciliation screen explains the source_unseen signal.
Document versions and lifecycle (design 9.2, AC-35). An upload may name a source_key (multipart field or JSON member; at most keeper.documents.max_source_key_chars, 1024): the bytes of the key's active version return it (200), other bytes become version n + 1 with replaces the previous one, now superseded, audited document.version. Keyless uploads keep their content addressing. GET .../documents?source_key= and GET .../documents/{id}/versions list versions; documents show source_key, version, replaces and status (active, superseded, retired, purged), and a document's details open_stale_candidates, can_retire and can_purge. Migration 0041, metadata only (operations).
The document_version_check job: for a new version, each value whose current provenance is extraction from an earlier version, and that the new version does not propose again, has its old evidence quote looked for verbatim (white space aside) in the new text; a gone or purged quote raises a stale reconciliation candidate with signal evidence_gone ({document, from_version, to_version, quote}), masked like evidence for sensitive Attributes. Nothing is retracted and no model is called. Bounded by scribe.extraction.versions.max_values_per_job (5000) with continuations; a Domain turns it off with reconciliation.triggers.document_versions: false (extraction).
Retirement and purge of documents: POST .../documents/{id}/retirement {reason} (new permission document.retire, held by maintainer and admin) runs the check with empty text and stops extraction for good (409 document-retired when an upload asks to extract it; extraction jobs naming it fail). POST .../documents/{id}/purge {reason, confirm} (new permission document.purge, admin only; confirm is the document id) deletes the bytes, parsed text, chunks and cached answers of the document and replaces its evidence quotes in Proposal items, candidates and schema suggestions by [erased], keeping provenance rows; irreversible, audited document.purge in the same transaction (operations).
Python client client.documents.versions|iterate_versions|retire|purge and source_key on upload, upload_url and list; CLI lorebase documents upload FILE --key KEY, lorebase documents list --key KEY, lorebase documents versions ID, lorebase documents retire ID --reason TEXT, lorebase documents purge ID --reason TEXT --confirm ID.
Web app, Documents: key, version and status in the library and on the document page, the versions of a document, its open stale candidates linked to the Reconciliation queue, a source key on upload, and Retire and Purge (typing the document id to confirm) for those who hold the permissions.
Read presets (design 8.8). POST /search and the search and query seeds of POST /context take traits: a Trait stands for the Kinds implementing it that the caller may read, and kinds with traits search the union. Herald's lorebase_search and lorebase_context take traits and lens, the CLI has lorebase search --trait and --lens and lorebase context --trait and --lens.
Lenses: lenses.{name} in the schema document is a named read preset (description, kinds, traits, relations, depth, include_schema, include_recent_changes, include_lineage, budget_tokens), a schema element with the id prefix lns_ that merges and is validated like the others (schema.lens_reference_unknown for a name the schema does not hold, schema.lens_invalid for a depth below 0 or a budget below 1). lens on POST /search and POST /context supplies the defaults: members of the request win, a lens over the caps is served at them (budget.capped), and the lens's Kinds and Traits limit neighbors and search and query seeds while refs seeds are always kept. A lens is not a permission: Kinds, Traits and Relations the caller may not read are dropped from it, and an unknown lens, or one whose Kinds are all unreadable, is 422 lens_unknown; with keeper.auth.authz.invisible_kinds, a Trait none of whose implementing Kinds is readable is 422 unknown_trait like an unknown one. GET /schema lists the lenses; Herald adds the resource lorebase://{domain}/lenses/{lens} and a LENSES part to lorebase_schema (agents).
Vector search that keeps its limit under filters: with pgvector 0.8 or later and search.semantic.iterative_scan: auto (default) the semantic list runs under hnsw.iterative_scan = relaxed_order bounded by search.semantic.max_scan_tuples (20000) and is ordered again by exact distance; otherwise the index returns search.candidates * search.semantic.overfetch_factor (4) nearest rows before the filters. The response says semantic_truncated when the scan bound left a list short while more Records match. /readyz reports the case in a vector_search check (search).
Python client: traits and lens on SearchRequest and ContextRequest, semantic_truncated on SearchResult, and the lenses section of schema.element.
Reconciliation queue (design 10.6): values on which Sources disagree no longer pass unnoticed when a one-sided write replaces another Source's value. Scribe raises candidates of type contradiction, superseded, stale and duplicate while extraction items are assembled, before a Source sync merges, and in the reconcile_sweep job (scribe.reconciliation.sweep_interval_s, off by default); nothing is changed, hidden or decided because a candidate exists. Stages run cheapest first: identity, values, normalization (units of a shipped table, numeric tolerance, case and punctuation, negation words) and an optional semantic stage (off by default) that marks paraphrases equivalent and compares stored Record vectors for duplicates. Candidates are fingerprinted by their values, so a dismissal holds until a new value appears (Scribe).
Reconciliation decisions in Keeper under /api/v1/domains/{d}/reconciliation: list with filters, get, POST .../{id}/resolution (take a claim, keep the current value, a custom value, or a policy authority, newest_wins, prefer_current, prefer_incoming; dry_run), POST .../{id}/dismissal, POST .../{id}/reopen, POST .../reconciliation:resolve in bulk (at most keeper.reconciliation.bulk_max) and POST .../reconciliation/sweeps (202). Values are written by the Kind's update_{kind} with a reconciliation provenance; when the branch needs review the write goes on a new reconcile/{candidate}-{random} branch and Proposal. Duplicates merge into a surviving Record through a Proposal. A candidate whose decision waits in an open Proposal refuses new decisions (409), and a claim or policy decision is refused when a newer value was written since detection. Decisions need conflict.resolve plus what the write needs, and clearance for sensitive or vault values, and are audited (reconciliation.resolve, reconciliation.dismiss, reconciliation.reopen).
Proposal check reconciliation: warn (default) lists the open candidates of a Proposal, block stops the merge while contradictions are open (409 reconciliation-open). Set by keeper.reconciliation.merge_check, scribe.reconciliation.merge_check for the merges Scribe runs, and the Domain setting reconciliation_check; the stricter one applies.
Valid time: an Attribute may declare valid_time: true (not on derived Attributes). Its values carry a validity period on their provenance, never in content: from Source mappings (attribute_mapping.{attribute}.valid_from, valid_to), from tier 2 extraction when the dates are in the quote, and from the valid parameter of update_{kind}. Disjoint periods make a superseded candidate instead of a contradiction.
Python client client.reconciliation (sync and async) and CLI lorebase reconcile list|show|resolve|dismiss|reopen.
Web app: a Reconciliation screen (/{domain}/reconcile) with a tab and count per type, filters by status, Kind, Attribute, source and signal, the claims of a candidate side by side with source, time, period and evidence, decisions with a preview and a required reason, duplicate merges, bulk decisions and sweeps. The Record page marks values with open candidates, the Proposal page links the candidates of its check, and the Source editor names the validity columns of valid_time Attributes. New settings VITE_RECONCILIATION_PAGE_SIZE and VITE_RECONCILIATION_COUNT_LIMIT (tour).
Metrics lorebase.scribe.reconciliation.candidates, lorebase.scribe.reconciliation.sweep.duration and lorebase.scribe.reconciliation.open (metrics).
Ingestion modes of extraction (design 9.2, AC-36): a Domain's settings.extraction chooses review_all (nothing accepted at once), doubts_only (the default, unchanged behaviour) or direct, with auto_accept_threshold, min_confidence and rules by source_key prefix (longest wins). Tightening always applies; loosening only for the members in scribe.extraction.domain_may_loosen (default [mode]), others are ignored and listed in ignored_settings of GET /domains/{d}. PATCH /domains/{d} takes settings.extraction (domain.settings, audited domain.update with before and after values). New settings scribe.extraction.mode (direct refused at startup), domain_may_loosen, direct_merge_protected (extraction).
direct ingestion: Scribe merges an extraction Proposal itself only when every item came from tier 0 and none is pending, the target accepts it and the merge is clean and not blocked by contradictions; any tier 1 or 2 item caps the job at doubts_only (mode_capped: tier). The job result says mode, direct_blocked (tier, pending_items, protected, reconciliation, conflicts), direct_merged and merged_revision; the merge is audited proposal.merge with mode: direct.
Stewards (design 10.7, AC-37): stewards: [role, ...] on domain and per Kind in the schema (an undefined role is the warning schema.steward_role_unknown, returned in warnings by schema:validate). Proposals and reconciliation candidates carry steward_roles and, for callers with principal.read, computed assignees (keeper.stewards.max_assignees_listed, 50); GET .../proposals and GET .../reconciliation take assigned=me|any|none. A principal who cannot read a Kind the work touches is never an assignee. Proposals record the Kinds they touch (proposals.kinds, migration 0042).
Protection rule required_approvals_from_stewards (keeper.proposals, Domain setting of the same name; default 0): approvals that must come from assignees, the Proposal needing the larger of it and required_approvals; too few assignees answers 409 stewards-unavailable. The approval block shows required_from_stewards, steward_approvals and stewards_unavailable.
Protection rule required_roles (keeper.proposals, Domain setting of the same name; default empty) (design 10.3): each listed RBAC role must have approved a Proposal into a protected branch, through a principal holding it who may review the Proposal. A merge missing one answers 409 approvals-required naming the roles; Proposal approvals show required_roles and missing_roles.
Python client assigned on proposals.list|iterate and reconciliation.list|iterate, steward fields on Proposals and candidates, settings on domains.update; CLI lorebase proposal list --mine and lorebase reconcile list --mine. Web app: an "Assigned to me" filter and the steward roles and assignees on Review and Reconciliation, the steward approvals on the Proposal page, and the ingestion mode and ignored settings on the Domain settings screen.
A purge no longer races a running extraction job of the same document: the purge and every write of the job (text, chunks, items, candidates, suggestions) lock the document row, the job stops (cancelled) when the document was retired or purged meanwhile, the purge cancels the queued and running extraction jobs of the document (jobs_cancelled in its audit entry), and a model answer is cached only while its chunk is stored.
GET /domains/{d} and PATCH /domains/{d} report can.settings: whether the caller holds domain.settings, computed by the authorizer (null in the Domain list).
Web app, Settings: the ingestion settings of a Domain (mode, auto-accept threshold, minimum confidence, rules by source key prefix) are editable for callers with domain.settings, with Keeper's validation errors at the fields they name and ignored values marked; others see the settings read only.
Web app: the Explorer's Kind filter offers the Traits of the schema, and a lens selector (Explorer search, and the Retrieval lab's searches and context pack) appears when the schema declares lenses.

Changed¶

Automatic merges no longer skip the rules that name who must approve (design 9.1). A Source sync with auto_when_clean or auto_with_authority and an extraction job in direct mode merged into a protected branch, with scribe.sources.auto_merge_protected and scribe.extraction.direct_merge_protected on (the default), whatever its approval rules said. Those settings still waive the plain required_approvals count, but when the target's effective rules include steward approvals (required_approvals_from_stewards above 0) or required roles (a non-empty required_roles), from keeper.proposals or the Domain setting of the same name, the merge is not performed and the Proposal stays open: the Source run's stats say merge_blocked: approvals, the extraction job's result direct_blocked: approvals, both with blocking_rules. The rules are evaluated by one implementation that Keeper and Scribe share (lorebase_store.approval_rules); Scribe reads keeper.proposals from the shared configuration. Installations without steward or role rules see no change (operations).
Semantic search reads the HNSW index at any size (design 8.8). The nearest vectors of the searched Kinds are taken from embeddings alone and joined to the Records, instead of joining first (which sorted every vector of the model): 25 ms against 329 ms p50 unfiltered at 300,000 vectors, 34.5 ms p95 against 513, inside the 200 ms budget. Unfiltered and broad-filter lists are approximate (HNSW recall 0.92 to 0.99 here) where they were exact; a one-Kind filter is exact. Over-fetch (iterative_scan: off) restricts to the Kinds' vectors before its limit.
Web app: a dialog opens with focus on its first control (a confirmation on its cancel button), or on its title when it has none, instead of on its Close button.
The include_schema, include_recent_changes and include_lineage members of a context request are unset by default (a lens may set them) and behave as true without one; the Python client's ContextRequest no longer sends them unless given.
Object type code 10 (schema.lens) joins the stored object types; no migration is needed.
Source syncs: corrections stick (design 9.1, AC-34). A Source now writes only the values that changed upstream since it last wrote them, kept per Source as hashes in source_written (migration 0040, advanced in the transaction that merges the Source's Proposal). A person's correction on the target is no longer put back by the next sync while upstream did not change; when upstream does change that value, the Source's Proposal holds a DV1 Conflict whose base is the Source's previous value, and a snapshot run that would delete a Record holding someone else's values holds a DV2, so an auto_when_clean Source waits for a person in those cases. The new Source setting local_edits chooses: review (default), keep (resolved at once for the person's value, resolver policy:local_edits) or source_wins. To get the old behavior, set local_edits: source_wins on the Source. Under auto_with_authority, Attributes whose authority is the Source are overwritten as before. Links a person removed or added are left until upstream changes them. A Record kept by a DV2 keeps its Links (the merge restores Links the deleting side removed with it), and a Record someone else deleted that comes back from upstream comes back with its Links. Under keep, each run also decides again the Conflicts a person reopened by editing a kept value while the Proposal was open. Each local edit is audited as source.local_edit_conflict, and run stats count local_edits and seeded. The first run of each existing Source seeds the state; a value it did not write that differs from upstream counts as a local edit that once (operations, Scribe).
Extraction: a value item over a value a person set last (an Operation, or a resolution or reconciliation decision by a principal) stays pending whatever its confidence, with hold reason local_edit (held_local_edit in the job stats) (extraction).
Web app, Modeler: the Source editor sets local_edits.
Merge: a text Attribute edited in different places on two Branches now merges automatically instead of being a Conflict. The values are compared by sentence (or paragraph) against the base; edits to different sentences are both kept, the result is recorded as an automatic resolution with resolver policy:text_merge, and the Review screen lists it under "Resolved by policy". Edits to the same sentence, or two insertions at the same place, are still a Conflict (neighbouring sentences rewritten on each side merge), which now carries details.text_hunks (the whole text in order, with the merged clean parts and the conflicting ones); its fingerprint is unchanged. Other types (string, uri, enums, numbers, json, lists) and values added on both sides are not affected. To get the old behavior set text_merge to false in the Domain's merge_policy setting, or only for a Kind, Relation or Attribute with text_merge_elements or text_merge_attributes; text_merge_granularity (sentence or paragraph) and text_merge_max_chars (characters over the three values, default 200000; fixed caps on the number of sentences and on the cost of repetitive text also apply) tune it (design 6.8, 6.10).
Web app, Conflict resolver: a text value that changed in the same sentences on both Branches (a DV1 with text_hunks) opens as passages: unchanged text folded away, each conflicting passage with Ours, Theirs and Base and a choice (ours, theirs, both in either order, base, or edit in place), a live preview with its length against the Attribute's limits, and "Use merged text" once every passage has a choice. [ and ] move between passages and 1 to 4 and 0 choose.
Web app, Conflict resolver: string values that moved from the base are shown word by word (added and removed words marked, with text for screen readers), and the Ours and Theirs columns carry the names of the target and source Branches. In the custom editor, "Copy base", "Copy ours" and "Copy theirs" load a side into the field. Saving a resolution opens the next open Conflict. Outside text fields o, t and b pick a side, e opens the custom editor, s or Enter saves and n moves to the next open Conflict. Leaving a Conflict with unsaved changes asks first, and "Apply resolutions" asks when Conflicts are still open or the one on screen is unsaved. The bulk choice uses the same names as the resolver ("Take ours", "Restore the Record") instead of the raw choice.
The release workflow can be run by hand (workflow_dispatch, input tag) on an existing tag, and publishes the ten packages to PyPI through trusted publishing, each from its own environment (pypi-<package>), when the repository variable PYPI_TRUSTED_PUBLISHING is true (release process).
Documentation: the release verification commands name 0.2.3 and the identity of a run started by hand; the README, the quickstart and the MCP guide say how to install lorebase-cli and lorebase-herald from PyPI.

Fixed¶

Web app, Modeler: a save also dropped members the editor does not model inside nested objects: a Kind's search, the Trait implementations of a Kind, Attribute constraints, Operation parameters and preconditions, and a Source's relation_mapping entries and target_branch_policy. They are now kept as they were.
Semantic search under iterative scans (pgvector 0.8 or later, the default) left hnsw.ef_search at pgvector's 40, so each pass of the index scan offered 40 candidates and recall@20 was about 0.92 at 300,000 vectors. The read transaction now raises it to search.candidates * search.semantic.overfetch_factor (800 by default, capped at 1000) with and without iterative scans: at 300,000 vectors the 40% filter goes from 0.929 to 0.964, p95 61 ms. The unfiltered list stays at 0.910 with the default index (a query in a region the graph does not reach); see the new HNSW build settings (Added) (performance).
docs/reference/configuration.md did not follow the changed description of search.semantic.overfetch_factor: the page is generated from the comments of config/lorebase.example.yaml, which now carries it.
Web app, Modeler: in live mode, saving the schema dropped the lenses, the stewards of the Domain and of Kinds, and any member or section of the document the editor does not model. The editor now carries them through untouched and writes them back as they were.
Semantic search without iterative scans (pgvector before 0.8, or search.semantic.iterative_scan: off) over-fetched less than configured: an HNSW scan ends after hnsw.ef_search rows (40), so search.candidates * search.semantic.overfetch_factor nearest rows never came back, and a filter keeping 1% of the Records left about 0.2 results of 20. The read transaction now raises hnsw.ef_search to the over-fetch count, up to pgvector's limit of 1000: 7.8 results of 20 on 100,000 vectors, flagged semantic_truncated (performance).
Web app, mock mode: accepting an attribute schema suggestion reports an update of its Kind, as Keeper does.

Security¶

Resolution misses: caller digests are HMAC-SHA256 under store.resolve_misses.caller_key, required when store.resolve_misses.enabled is on (at least 32 characters, a secret:// reference), so a reader of resolve_misses can no longer confirm a guessed principal id. Changing the key restarts the count of distinct callers; a caller seen under both keys counts twice until its phrases expire, and lorebase-scribe misses purge DOMAIN starts clean. Migration 0047 clears the digests stored before it.

Not done yet¶

Corrections that stick: the last-written state of a Source is not shown in the web app or the API.
Reconciliation: no queries by valid time (periods classify claims, reads stay at Revisions), no key aliases after a duplicate merge (references to the absorbed key are not redirected), and no MCP tool for the queue. In the web app, tab counts stop at VITE_RECONCILIATION_COUNT_LIMIT and the signal filter and ranking apply to the pages loaded.
Allowed merge origins per protected branch (roadmap).
Resolution misses are not linked to the next query of the same agent session, and context packs and search carry no coverage block (roadmap).
The performance targets of the design at 10M Records on the reference hardware, and the 1M-Record neighborhood on branches other than main, just over budget when the working set does not fit in memory (performance).
The stability guarantees for /api/v1 bind from 1.0 (API stability).

Upgrade notes¶

Where steward approvals or required roles apply to a protected branch, Source and direct-extraction merges into it now wait for review (above, Changed); give Scribe the same configuration file as Keeper so it sees keeper.proposals. Existing embedding indexes keep ef_construction 64 until rebuilt (operations).
stale_after suits incremental feeds that deliver their live rows again within the window; a feed that sends only changed rows flags every unchanged Record once stale_after passes (false positives): leave it null there (Scribe).
Migration 0044 builds ix_mat_records_object_hash on mat_records (domain_id, branch, object_hash) per partition with CREATE INDEX CONCURRENTLY (writes continue; no restart; extra space about 40 bytes per row). See operations.
Migration 0038 creates reconciliation_candidates and adds two nullable columns to provenance (no backfill); 0039 adds indexes to the candidates. Run the migrations to 0039, then restart Keeper and Scribe. Detection is on by default (scribe.reconciliation.enabled) and only raises candidates; the merge check defaults to warn, which blocks nothing.
Migration 0040 creates source_written and source_written_pending, empty. The first run of each Source after it is compare-only: every owned Attribute whose upstream value differs from the target's and was not written by that Source (an import, a hand edit, another Source) is a DV1 Conflict under the default local_edits: review, and a snapshot deletion of a Record holding such values a DV2, so an auto_when_clean Source that merged silently stops at an open Proposal until someone reviews it. Run each Source with dry_run: true first (local_edits, seeded). When the target's values are stale copies of upstream, set local_edits: source_wins for that first run (the old behavior, each overwrite audited) and back to review after; otherwise review in bulk with lorebase proposal conflicts PROPOSAL --status open --json and lorebase proposal resolve PROPOSAL --file resolutions.json (or POST .../proposals/{p}/conflicts:resolve), then merge (operations).
Migration 0042 adds proposals.kinds (null for existing Proposals until their next merge preview; meanwhile their Kinds are computed when steward roles, assignees or the steward approval rule are read). Without settings.extraction and without stewards in a schema nothing changes; scribe.extraction.mode: direct is refused at startup (operations).
Enabling store.resolve_misses.enabled now requires store.resolve_misses.caller_key (at least 32 characters, a secret:// reference); the settings fail validation without it.
From 0.2.3: replace the images (or the packages) with 0.3.0, run the migrations to 0047 (0041, 0043, 0045, 0046 and 0047 are additive), then restart Keeper and Scribe. Read the notes on 0040 and 0042 above before the first Source runs.
From 0.2.0, 0.2.1 or 0.2.2: as from 0.2.3; the migrations include 0037 of 0.2.2.
From 0.1.0: follow the upgrade notes of 0.2.0 below, with the images of 0.3.0.

0.2.3 - 2026-10-24¶

A governed bulk delete: delete_records removes many Records of one Kind in one Operation execution, one Revision and one rate-limit token. No migration, no breaking change.

Added¶

Built-in bulk delete delete_records (kind, keys, optional skip_missing): deletes many Records of one Kind in one Operation execution, one Revision and one writes rate-limit token, instead of one delete_{kind} call (and Revision) per Record. It is governed like the other built-ins: protected branches and review, dry_run, expected_head, Idempotency-Key, and Kind-scoped grants as for delete_{kind}; it needs the Kind's delete built-in enabled. Links of the deleted Records follow each Relation's on_delete (cascade or restrict) as a single delete does. Unknown keys fail the call (record.not_found per key) unless skip_missing leaves them out with warnings. At most keeper.operations.max_bulk_delete_keys keys per call (default 1000). Python client operations.delete_records, CLI lorebase record delete KIND [KEY]... [--keys-file FILE] [--skip-missing], and Herald lists it among the generic Operations (design 5.7, operations).

0.2.2 - 2026-10-23¶

Kind-scoped invisibility completes full RBAC: a caller whose grants are narrowed to Kinds no longer learns of the other Kinds, and gets 404 or 422 where it got 403 (keeper.auth.authz.invisible_kinds: false keeps the earlier answers). Lexical search can rank with BM25 through ParadeDB pg_search, opt-in and off by default. Seven fixes for running Lorebase locally: compose rebuilds its images and migrate refuses a database ahead of the code, Keeper settings forwarded from .env, a clear message for reserved Attribute names, herald.http.allow_anonymous, Herald as the compose profile mcp on 127.0.0.1:8091, CORS expose_headers and an additive allow_headers, and the web app README for live mode. Migration 0037 adds functions only.

This release was built and published locally, without the release workflow: the images on GHCR, the checksums and the release assets are not signed with cosign, and the images carry no SBOM attestation (the CycloneDX SBOMs are attached to the release as files). cosign verify does not apply to 0.2.2.

Breaking changes¶

Kind-scoped invisibility is on by default: principals whose grants are narrowed to Kinds get 404 or 422 instead of 403 for Kinds they may not read, and no longer see those Kinds in the schema, counts, search or context packs. Clients that told a forbidden Kind from an unknown one by the 403 must stop relying on it, or set keeper.auth.authz.invisible_kinds: false to keep the earlier answers (design 13.3).
Docker compose builds the backend, backend-ml and web images from the checkout on every up; set LOREBASE_PULL_POLICY=missing to keep using prebuilt images (deployment).

Added¶

Kind-scoped invisibility, the last part of full RBAC. A caller whose grants are narrowed to Kinds no longer learns of the other Kinds: GET /schema (JSON and YAML) and schema elements return the schema without them and without the Relations, path Derivations, derived and ref Attributes, Traits, Operations and Sources that would tell about them, and every request naming one of them answers with the 404 or 422 of a name that does not exist, never a 403. Counts, Link summaries, neighborhoods, search hits and typo corrections, health reports, context packs and agent trace refs cover the visible Kinds only. Such callers can now read the Domain, its branches and tags, so the web app, Herald and the Guide work for them. On by default; keeper.auth.authz.invisible_kinds: false keeps the 403 naming the Kind (design 13.3, threat model).
Optional BM25 ranking of lexical search through ParadeDB pg_search, for searches of very common terms on large Domains: search.backend: bm25 ranks with a BM25 index of the existing search vectors, built by python -m lorebase_store.storage apply with store.storage.bm25_index: true on a server that preloads pg_search. Matches, typo corrections, partial matching, Kind filters and masking are those of the default ranking; scores are relative to the best hit. Without the extension or a valid index search keeps the default ranking and /readyz reports search_backend: degraded. pg_search is licensed under AGPL-3.0 and runs only where you choose its image: the compose overlay deploy/compose/search-bm25.yaml and the Helm values file ci/search-bm25-values.yaml set it up; the default stack, chart and images are unchanged. At 1M Records very common words are 20 to 35% faster at p50 and never truncated; rare words are about 3 ms slower, overlays slower, and scans slow down after writes until VACUUM (search, performance).
Migration 0037: lb_search_terms, lb_bm25_query_valid and the security definer functions lb_search_bm25 and lb_search_bm25_overlay, granted to the application role like the other read functions. They need nothing of the extension until it is installed.
make perf-up PERF_SEARCH=bm25 runs the perf Postgres on the ParadeDB image, and tests/perf/bm25.py compares both rankings, the index build and the commit cost on a seeded database.
herald.http.allow_anonymous (default false): Herald over Streamable HTTP forwards calls without a credential to Keeper without one instead of refusing them, for local stacks whose Keeper runs with auth mode none. Keeper still authenticates each call, so under any other mode they keep failing with [unauthorized] (MCP).
Herald in docker compose: the optional profile mcp runs it over Streamable HTTP on port 8090 in the container, published on 127.0.0.1:8091 (HERALD_HOST_PORT; 8090 is the web app's), wired to the stack's Keeper, with DNS rebinding protection for 127.0.0.1, localhost and herald. make up PROFILES=mcp starts it with the default services (MCP).
Docker compose forwards documented Keeper settings from .env to the Keeper services when they are set there: LOREBASE__KEEPER__AUTH__MODE, ...__PROPOSALS__ALLOW_SELF_APPROVAL, ...__PROPOSALS__REQUIRED_APPROVALS, ...__OPERATIONS__DIRECT_COMMIT_ON_PROTECTED and the ...__CORS__ origins, headers and credentials. Unset, the YAML file decides as before (deployment).

Changed¶

Kind-narrowed callers get 404 or 422 instead of 403 for Kinds they may not read, see the cache entries and ETags of their own projection, and no longer see a Kind whose keys reference an invisible Kind or whose titles or full-text index are computed from one (a searchable count of its Records); derived Attributes reading an invisible Kind, such as supplier_count and single_sourced for a reader of Parts only, are left out of their Records and of Derivation previews. Agent trace steps of other principals no longer carry result counts, query plans or a truncated flag for them, Proposal extraction checks no longer count items for callers who do not see every Conflict, and resolving a Conflict outside their Kinds answers 404 like an unknown one. Branch responses leave out the progress of background copies for them. Proposal Conflicts about a reference to an invisible Kind are left out for them, and Record history, lineage, explanations and context packs leave out origins, messages and Proposal titles naming something invisible and the changes that touched only what is left out.
Documentation: the validation record gains the five-day scenario's CI figures, the rounds and findings of the PII vault and outbox security reviews; the README and the documentation landing page describe v0.2.1; the roadmap keeps open only the reference-hardware benchmark and the API stability guarantees (validation, roadmap).
The standby of the compose profile scale-out passes the arguments of its service command to Postgres, so an overlay can add settings to it.
Documentation: the web app README explains live mode against a Keeper on another port or host, through the dev proxy (LOREBASE_KEEPER_PROXY) or cross-origin (VITE_KEEPER_BASE_URL), and how Keeper allows the app's origin with LOREBASE__KEEPER__CORS__ALLOW_ORIGINS as a JSON list (apps/web).

Fixed¶

Keeper CORS: keeper.cors.expose_headers (default ETag, Lorebase-Revision) makes conditional reads work for a web app on another origin, which could not read the ETag and so never got a 304. keeper.cors.allow_headers now adds to the headers Keeper always allows (Authorization, Content-Type, Idempotency-Key, If-None-Match, the request id, session CSRF and agent trace headers) instead of replacing them, and the trace headers are allowed even with tracing off, since clients send them anyway (configuration).
A Kind Attribute named after a reserved api name (key, kind, ref, revision) is refused with a message that says why and proposes another name, such as "`key` is a reserved name, choose another (e.g. `code`)", instead of api name 'key' is reserved; the schema documentation lists the reserved names (design 5.4).
Docker compose no longer starts a stale local image: the backend, backend-ml and web images are rebuilt from the checkout on every up (pull_policy: build; LOREBASE_PULL_POLICY=missing keeps prebuilt images), and migrate refuses a database that a newer Lorebase already migrated with an error naming the database revision, the newest revision of the code and the fix, instead of Alembic's Can't locate revision and a Keeper that restarts in a loop (deployment).

Upgrade notes¶

keeper.cors.allow_headers lists further headers only; its default is now empty. A configuration that repeats the earlier default keeps working, and one that listed fewer headers now allows the required ones as well.
Clients of Kind-narrowed principals that relied on 403 to tell a forbidden Kind from an unknown one now get 404 or 422 for both. Set keeper.auth.authz.invisible_kinds: false to keep the earlier answers.
Migration 0037 creates functions only and runs on any Postgres; apply it as usual. Search is unchanged unless you set search.backend: bm25 and build the index.
From 0.2.1: replace the images (or the packages) with 0.2.2, run the migration to 0037, then restart.
From 0.2.0: as from 0.2.1; the notes of 0.2.1 add nothing to do.
From 0.1.0: follow the upgrade notes of 0.2.0 below, with the images of 0.2.2.

Not done yet¶

The performance baseline on the 1M-Record fixture: the neighborhood on branches other than main is just over budget when the working set does not fit in memory (performance).
The performance targets of the design at 10M Records: the 2-hop neighborhood, full-text search (searches of very common words above all), Record history and lineage are over budget, and a commit of 1,000 changes takes more than 2 s (performance).
The stability guarantees for /api/v1 bind from 1.0 (API stability).
Packages on PyPI: the wheels and sdists are attached to the GitHub release.
Signed images and assets: 0.2.2 was published without the release workflow (see above).
BM25 search at 10M Records is not measured (make perf-up PERF_VM=16g PERF_SEARCH=bm25, then tests/perf/bm25.py --database perf_seed_10m_zipf), and storage apply does not tune autovacuum for it: with BM25 on and frequent writes, set a lower autovacuum_vacuum_scale_factor on the mat_records partitions yourself (performance).

0.2.1 - 2026-10-23¶

Patch release for the storage profiles of 0.2.0. python -m lorebase_store.storage apply now converges when commits run during the lineage rewrite, the downgrade of migration 0036 no longer deadlocks against commits, and a repeated apply skips the full lineage scan once a pass is recorded complete (--verify forces it). The performance documentation replaces the 10M-Record storage profile estimate with measured numbers. No migration, configuration key or API response changes.

Added¶

make perf-up starts the Postgres of the performance harness without running a benchmark; PERF_VM=16g sizes its memory for a Docker VM of 16 GiB (shared_buffers 6GB, effective_cache_size 12GB) instead of the default PERF_VM=8g. Each PERF_PG_* value can still be set on its own; these are harness settings, not product defaults (performance).
python -m lorebase_store.storage apply --verify scans every lineage map row again and resets what is recorded (performance).

Changed¶

storage apply no longer scans every lineage map row again once a rewrite has finished: it records the pass per partition in storage_tasks with a watermark, and a later run with the same profile only looks at the rows written since. At 10M Records the full scan took 13.5 minutes even with nothing to rewrite. Rows written while the triggers were turned off out of band, or frozen by VACUUM before a run looked at them, are only found by --verify (performance).
The storage profiles at 10M Records are measured instead of estimated (Docker VM of 16 GiB): the hot set goes from 57 GB to 44.1 GB in balanced and 43.3 GB in compact, the cold 2-hop neighborhood from 122 to 100 ms at p50 (96 in compact), and a commit of 1,000 Part updates takes 2,711 ms at p50 in balanced and 2,332 in compact under the same load (performance, validation).

Fixed¶

storage apply no longer leaves lineage map rows a commit held during the rewrite in the other form: the page ranges they are in are rewritten again after the pass, and while rows stay held the step is deferred (exit status 3) and the next run finishes them (performance).
The downgrade of migration 0036 takes the locks of mat_records and mat_links up front, in the order commits write them, each wait shorter than deadlock_timeout: a commit waiting for it is never chosen as a deadlock victim, and neither is the downgrade.

Not done yet¶

The performance baseline on the 1M-Record fixture: the neighborhood on branches other than main is just over budget when the working set does not fit in memory (performance).
The performance targets of the design at 10M Records: the 2-hop neighborhood, full-text search (searches of very common words above all), Record history and lineage are over budget, and a commit of 1,000 changes takes more than 2 s (performance).
Full RBAC: Kind-scoped invisibility in queries (roadmap).
The stability guarantees for /api/v1 bind from 1.0 (API stability).
Packages on PyPI: the wheels and sdists are attached to the GitHub release.

Upgrade notes¶

From 0.2.0: replace the images (or the packages) with 0.2.1 and restart; there is no migration and no configuration to change. If an earlier storage apply stopped or left lineage map rows in the other form, run it again as the maintenance role: it finishes them, and it exits with status 3 while commits still hold rows, in which case run it again later. The first run of 0.2.1 makes the full lineage pass once and records it; later runs skip it.
From 0.1.0: follow the upgrade notes of 0.2.0 below, with the images of 0.2.1.

0.2.0 - 2026-10-23¶

Second pre-alpha release. Records now carry their history and lineage, values tagged pii, confidential or restricted are masked and vault values can be erased, Scribe computes embeddings for semantic and hybrid search, path Derivations work, storage profiles decide what the database keeps besides the rows, and Guide, an optional exploration assistant outside Keeper, reads a Domain with the caller's own credential. The web app signs in through Keeper and its Modeler edits Operations, Derivations and Sources, and its copy no longer names the data of any one Domain. Pre-1.0: this minor version changes interfaces; read the breaking changes before upgrading. The database schema moves from migration 0012 to 0036, read.history.lineage_max_changes is removed, Scribe and Guide run on a new image ghcr.io/aantenore/lorebase-ml, and POST /search adds truncated_candidates. The lines link the documentation of each change: the work landed directly on main, without pull requests to link.

Breaking changes¶

Webhook signing keys are sealed by default: Keeper refuses to create a webhook unless webhooks.encryption_key is set (webhooks.allow_unsealed_keys allows it for local development only), make env generates LOREBASE_WEBHOOK_KEY for the compose stack, and the Helm chart requires secrets.values.webhookEncryptionKey or an existing Secret entry.
The Helm chart no longer runs Keeper and Scribe as the owner of the tables. As on compose, it takes two DSNs: db-maintenance-dsn (secrets.keys.dbMaintenanceDsn, the owner) for the migration Job and for database.maintenance_dsn of Scribe and Keeper, and db-dsn for the application role (database.appRole.name, default lorebase_app) that Keeper and Scribe connect as, so row-level security, the append-only audit_log and the ban on DDL apply to them. A db-roles init container of the migration Job creates the role when missing (database.appRole.create, needs CREATEROLE) or waits for one created elsewhere, refuses SUPERUSER, BYPASSRLS or the owner, and applies the grants of the compose script; the migration runs with database.application_role. With the in-cluster Postgres the operator manages the role (postgresql.cluster.roles) with the password of secrets.values.dbAppPassword. The chart refuses one user for both DSNs unless database.allowSharedRole: true (development). Upgrading: add db-maintenance-dsn with the old DSN and point db-dsn at the application role; an in-cluster release upgrades in one step (next line) (Kubernetes).
Kubernetes, in-cluster database: one helm upgrade now adds the application role (kubernetes). A pg-roles pre-upgrade hook Job declares the roles of postgresql.cluster.roles on the CloudNativePG Cluster and waits for the operator before the migration Job runs; upgrading a release that used one DSN no longer takes two upgrades. It needs access to the API server when egress is restricted (networkPolicy.egress.kubernetesApi); database.setup.clusterRoles.enabled: false turns it off.
Scribe refuses to start when scribe.worker.heartbeat_interval_s is more than half of store.jobs.stale_after_seconds. Jobs run at least once: a job is run again after its worker is lost, so handlers must tolerate a second run (the built-in ones do).
Reading values of Attributes tagged pii, confidential or restricted needs pii.read, confidential.read or restricted.read; other callers see them masked (design 10.2). Feeding vault values through imports and Sources needs the new Domain permission pii.import (only admin among the built-in roles), and vault key rotation needs vault.manage (installation administrators only). Give the read permissions through a custom role in keeper.auth.authz.custom_roles.
Sensitive Attributes are left out of full-text search for every caller (see Changed): a search that matched a sensitive word no longer finds the Record.
read.history.lineage_max_changes is removed (lineage no longer reads a window of changes). Configuration files refuse unknown settings: delete it.
lorebase admin erase --value V refuses a literal value, which stayed in the shell history: use --value - or --value @FILE. A value erased before is stored erased when written again; vault.erased_values: allow keeps the old behaviour.
Lexical search ranks a different set of matches by default: within search.common_term_docs (2,000) every match is ranked, beyond it the first search.common_term_candidates (1,000) in index order, and POST /search answers truncated_candidates: true when that cut applies. Before, the first store.reads.search_candidates (1,000) matches were ranked for every query. search.common_term_docs: 0 keeps the old behaviour. Keeper needs migration 0035.
Compose and Helm run Scribe and Guide on the new image ghcr.io/aantenore/lorebase-ml (844 MB against 453 MB; fastembed and a baked embedding model). Deployments that mirror or pin images need it too; Helm imageMl.repository: "" keeps Scribe and Guide on image (kubernetes).
Guide needs a writable state directory while its root file system is read-only: compose mounts the volume guide-data at /var/lib/lorebase/guide and Helm guide.persistence (an emptyDir by default); elsewhere the default is ~/.local/state/lorebase/guide, so mount a volume there or set guide.mode_memory_file: null (guide).
Compose: make up, make logs, make down, make clean-stack and scripts/quickstart.sh add the files listed in COMPOSE_OVERLAYS after their own and stop before compose runs when one does not exist; make down stops every profile make up can start (quickstart).
Migration 0036 (shared lineage maps) leaves the lineage maps of an existing install inline until python -m lorebase_store.storage apply runs as the maintenance role, so processes of 0.1.0 keep reading them during a rolling upgrade; the space comes back only with apply --compact --reindex (store schema notes). A new install shares maps from the start.
store.storage.profile (compact, balanced, fast; compose LOREBASE_STORAGE_PROFILE, Helm config.store.storage.profile) decides which secondary indexes exist. The default builds the same indexes as before; compact drops the GIN index on Record values, the indexes of indexed-only Attributes and the lexicon's trigram index when storage apply runs (performance).
Web app: the supply presets and dataset-specific placeholders are gone, example questions come only from domain.examples of the schema, and the "planned screen" placeholders (Explorer, Modeler, Review, Sources, Settings, the Record page's Run Operation and the Canvas diff overlay tab) are removed. The Retrieval lab no longer embeds the Explore panel; it links to Explore. Every user-facing string moved to apps/web/src/messages, so local text patches must move with it.

Added¶

Search ranks every match only when it can afford to (design 8.4, performance). Lexical search without filters reads the word lexicon's counts of the query's lexemes, which bound its matches: within search.common_term_docs (2,000) every match is ranked; when every term is more common, only the first search.common_term_candidates (1,000) matches in index order are, and POST /search answers truncated_candidates: true so that callers narrow the query (Herald and Guide say so to the agent). Before, at most store.reads.search_candidates (1,000) matches were ranked for every query, without saying when that cut the ranking. search.common_term_docs: 0 keeps that behaviour. Measured at 1M and 10M Records with realistic text, 20,000 and 5,000 raised the p95 of title-word searches 3.4 times at 1M and 4.4 times at 10M, hence the lower defaults. Migration 0035 adds the security definer functions lb_search_bounded and lb_search_bounded_overlay, which report the cap.
values: false on POST /neighborhood (agents): nodes come back as Record headers (kind, key, title) and edges as Link endpoints, for callers that need the structure only. The Python client takes values on NeighborhoodRequest; Herald's lorebase_neighborhood asks Keeper for it unless its values is true (its default leaves values out, as before, now Link values too); Guide's neighborhood tool takes values (default true).
--vocabulary zipf for examples/supply/scripts/make_supply_data.py: names, titles, summaries and reasons drawn from 40,000 terms with Zipf frequencies plus product codes, the rest of the dataset unchanged. The 1M and 10M benchmarks use it (seed databases perf_seed_1m_zipf and perf_seed_10m_zipf; PERF_VOCABULARY=demo measures the earlier ones); tests/perf/focus.py compares search with and without the lexicon-sized ranking and the neighborhood with and without values on a seeded database.
Benchmarks at 1M and 10M Records (performance): make perf-1m and make perf-10m start a Postgres of their own (tests/perf/compose.yaml, compose project lorebase-perf), seed the dataset once into a kept database and measure it in place; make perf-down deletes it. The 10M fixture loads a tenth through the commit pipeline and copies its materialized rows with renumbered keys (--replicas); an interrupted load resumes. The harness also times the agent trace through Keeper (--trace-iterations) and can bound the measured branch copy (--copy-batches, --no-pair-copy).
COMPOSE_OVERLAYS (in .env or the environment): extra compose files, space separated, that make up, make logs, make down, make clean-stack and scripts/quickstart.sh add after their own, so make up PROFILES=assistant can start Scribe with a hosted model and its key file (deploy/compose/hosted-model.yaml) and an overlay of your own (quickstart, inference providers). A listed file that does not exist stops the command before compose runs. make logs (SERVICES="scribe guide" narrows it) follows the logs of the stack.
Job metrics lorebase.jobs.reclaimed (by type and outcome requeued or failed) and lorebase.jobs.released (by type), and the alert LorebaseJobsReclaimed (operations) when running jobs lose their worker.
Guide and Scribe embed text out of the box in compose and Helm. They run on a new image, lorebase-ml (target backend-ml of deploy/docker/backend.Dockerfile; ghcr.io/aantenore/lorebase-ml in releases): the backend image plus fastembed (ONNX Runtime, no PyTorch) and the embedding model EMBEDDING_MODEL (default BAAI/bge-small-en-v1.5) downloaded at build time, so POST /embed and Scribe's embedding jobs work offline with the read-only root file system. It is 844 MB against 453 MB for the backend image, so Keeper, the migrations and Herald stay on the small one. Helm: imageMl (empty imageMl.repository puts Scribe and Guide on image). The supply example schema gives Part and Supplier an embedding template, so semantic search over the sample returns hits once Scribe has run its first pass (quickstart).
make up PROFILES=assistant (or make up-assistant) starts Guide and Scribe with the quickstart stack through the new overlay deploy/compose/assistant.yaml, which points the web app at /guide and lets Guide accept calls while Keeper has no authentication. make down now stops every profile make up can start.
make example-documents (scripts/quickstart.sh documents) extracts the contract C-3099 and the four example documents with the rules tier and lists the schema suggestions. One document supports each suggestion once, below schema.suggestions.min_support (2), so a single upload listed nothing; five documents list ends, sole_source_parts and buyer (extraction).
Record lineage names the Revision behind every value, however many changes came after it (agents): materialized Records keep that Revision per Attribute (migration 0032, mat_records.lineage_revisions), and rows written earlier fall back to the branch history. Lineage at a branch head is one statement, about 25% faster at 100k Records (performance). GET .../records/{key}/lineage takes at and answers from a snapshot of that Revision, like Record history; so do lorebase record lineage --at, the Python client and the MCP tool lorebase_lineage.
The Modeler previews Derivations and Source expression mappings before they are saved (tour). POST /domains/{d}/derivations:preview checks the edited schema like a schema change and evaluates a candidate expression or aggregate Derivation on sample Records of its Kind, with Record-read permissions and masking (read.derivation_preview); POST /domains/{d}/sources:preview (schema.read) evaluates a Source's expression mappings on the rows sent (read.source_preview_max_rows). Nothing is written. Scribe now maps Attributes by expression (CEL over the row, row["name"] for any column) instead of refusing them. Python client: preview_derivation, preview_source.
Path Derivations are now followed wherever Relations are: neighborhoods (when named in relations), Link counts of a Record, Operation links(), links.<path> in Derivation expressions and aggregate Derivations, which are updated incrementally when a Link of any step changes. Paths crossing a Kind the caller may not read are left out of neighborhoods and Link counts. mat_schema.path_fanout records the store.traverse.path_fanout each branch was computed with; a commit or a starting job worker that finds a different value rewrites the materialized paths and the derived values reading a path (design 5.8).
Structured output modes are remembered per chat endpoint across restarts (inference providers). Scribe keeps them in the table inference_endpoint_modes; Guide keeps them in a locked JSON file (guide.data_dir, guide.mode_memory_file). Entries are keyed by an HMAC of the endpoint, model and output settings (inference.mode_memory_key is mixed in when set) and expire after inference.mode_memory_ttl_s (7 days). lorebase-scribe modes clear and lorebase-guide modes clear forget them at once.
Schema suggestions from the CLI: lorebase schema suggestions list|accept|reject|merge (extraction). merge (also POST /domains/{d}/schema/suggestions:merge, schema.write, client merge_suggestions) folds near-duplicate Attribute suggestions already stored, with the same similarity as new ones. A Kind suggestion is no longer hidden by a same-named Attribute unless that Attribute is free text no Relation models and the evidence shows no independent entity (schema.suggestions.kind_independent_instances, kind_independent_attributes). An optional dedicated discovery pass (scribe.extraction.discovery.llm_pass, off by default) proposes Kinds and Attributes through the inference.llm.tasks.schema_discovery route.
Reads at a past Revision (at) support filters, ordering, includes, aggregates, Link and lineage summaries, traversal, neighborhoods, search, Record history, health reports, context packs and explanations (design 8.6). They are served from a snapshot of the Revision, materialized on first use with the code of a branch rebuild and shared by later reads, so they answer what the head answered when that Revision was the head. read.time_travel.snapshot_max_entries (100000) bounds the size of a snapshot, larger Revisions are refused with 422 time_travel, and read.time_travel.snapshot_max_per_domain (4) how many are kept. History, explain and the health routes take at as a query parameter, traverse, neighborhood, search and context in the body.
An overlay for trying browser sign-in on the compose stack (quickstart): deploy/compose/oidc-mock.yaml starts a mock OpenID Connect provider in Keeper's network namespace, turns on keeper.auth.sessions and puts the web app in session mode (WEB_AUTH_MODE=session); set LOREBASE_SESSION_KEY in .env. The provider signs in anyone: local use only. One sign-in, session, CSRF-protected sign-out and expiry were checked through the web service (validation).
guide.http.default_origins, the browser origins Guide accepts while allowed_origins is empty. The compose file sets it to the origins of its web service, so GUIDE_ALLOWED_ORIGINS=[] now behaves as unset (Guide logs a warning) instead of refusing the web app.
A values schema for the Helm chart (deploy/helm/lorebase/values.schema.json): every value of values.yaml is typed, settings with a fixed set of values are enumerated, and unknown keys are refused except under config and postgresql, so helm lint, template, install and upgrade reject a misspelled key or a wrong type. CI lints and renders the guide profile and checks that the schema refuses bad values.
Connectors: SQL Server query Sources can end in ORDER BY (Scribe appends OFFSET 0 ROWS inside the derived table), and rowversion watermarks are kept as hex in the run checkpoint and bound as bytes. POST /domains/{d}/schema:infer accepts named MySQL, MariaDB and SQL Server sources (mysql, mssql), read through a bounded sample under the same allowed-source rules and profiled like records (new lorebase-keeper[mysql] and lorebase-keeper[mssql] extras, store.profiling.sql settings). JSON uploads can hold their records under a key with the records_path form field (Python client infer_files(records_path=...), web "Start from data"). The new POST /domains/{d}/schema:preview (schema.write, nothing stored; client preview_files) returns the first rows of uploaded files, and the web Source preview uses it for Parquet files.
Explore in the web app (guide): a screen next to the Retrieval lab, and the same panel under the lab, that asks Guide a question about the Domain and shows each event of POST /explore as it streams in (reasoning notes, tool calls with compact arguments, result summaries and counts, errors, the answer and the totals). The request carries the caller's own credential (Bearer token, or the session cookie and CSRF header when Guide is served from the app's origin; guideUrl may be a path such as /guide). Stop aborts the request, a stream longer than VITE_EXPLORE_TIMEOUT_MS is stopped, and a 429 waits for Retry-After. Model output is rendered as plain text only; citations become links to Record pages when their Kind and key are valid, "View trace" opens the session in Agent activity, and the cited Records can be drawn on the Canvas. The supply Domain has five preset questions. Against the mock API the panel plays scripted explorations. New settings VITE_EXPLORE_MAX_STEPS, VITE_EXPLORE_TIMEOUT_MS and VITE_EXPLORE_MOCK_STEP_MS.
Guide, an optional exploration assistant outside Keeper (guide). The new service lorebase-guide (package lorebase_guide) answers POST /explore with a stream of server-sent events: a small read-only agent reads the Domain through Keeper (schema, search, query DSL, Record, links, neighborhood, traversal, history, lineage and context packs), streams each tool call, a short result summary and the model's reasoning notes, and ends with an answer citing the Records it was shown, or "Not found in the graph." Every Keeper request carries the caller's own credential (Bearer token or API key, or the web session cookie with its CSRF token, checked against Keeper), so RBAC and masking apply exactly as for the user; Guide holds no credential and has no write tool. Each exploration is an agent trace session (X-Lorebase-Session, tool header, reasoning posted as notes). Budgets for steps, tokens per call, tokens per exploration and wall-clock time, per-credential rate and concurrency limits, and an Origin allow-list are in the new guide settings. POST /embed returns the vector of a query or document text with the configured embedding model and prefixes, the same vector Scribe stores for the same document text. Besides Scribe, Guide is the only package that may use lorebase_inference; the import contracts keep Keeper and Herald model-free and forbid them to import Guide. It runs from the backend image (compose profile assistant, Helm guide.enabled, off by default), and the web app reads its address from the runtime configuration guideUrl (VITE_GUIDE_URL).
The Python client refuses empty, . and .. path segments (Domain ids, Kind names, Record keys, relation names) and branch names with such parts, raising ValueError before any request: an HTTP client resolves them, which sent the request to another route.
A quickstart that works from a fresh clone (quickstart). make env writes .env with generated database passwords, make up builds the image from the checkout and waits until Keeper is healthy, make example-data loads the supply sample and merges it into main, make down and make clean-stack stop the stack and delete its data. The quickstart adds an agent connected over MCP, a real model (Ollama in compose or on the host, or a hosted endpoint whose key is mounted from a file by deploy/compose/hosted-model.yaml), cleanup, a troubleshooting table and measured times. .env.example has COMPOSE_PROJECT_NAME, so two stacks on one machine no longer share containers and volumes, and Scribe and inference-check resolve host.docker.internal on Linux too.
Ranked partial matching in lexical and hybrid search (design 8.4). When q as written (every term) finds fewer than search.partial.min_results Records (1), Records holding some of its terms follow, after every full match: the words and quoted phrases of q joined by OR, negations kept for each, stop words of search.partial.stop_words (english) left out, scored weight * (ts_rank_cd + term_bonus * terms held) with typo corrections applied per term. Each hit reports match (all or partial, absent for hits found by vector only); the field is new and optional. Descriptive queries such as "ring that stops oil leaking" found nothing before, so hybrid search ranked them by the vector alone; the O-rings now come first lexically too (validation). Measured at 100k Records in performance.
Chat endpoint options for hosted providers (inference providers): inference.llm.headers sends extra HTTP headers (values accept secret:// references, for example an anthropic-workspace-id), extra_body merges extra request fields (reasoning_effort, thinking, ...), json_schema_strict sets response_format.json_schema.strict, and temperature: null leaves the temperature out for models that refuse it. All four also work per task in inference.llm.tasks; a task on another base_url never inherits the headers. Header values are scrubbed from error text like the key. Anthropic's OpenAI-compatible endpoint and local Ollama models were run against the example documents; setup and results are in validation.
Agent trace (agent trace, keeper.trace, off by default). Requests that opt in (X-Lorebase-Trace: on, or a listed principal or API key, or trace_all, which leaves out the web app's own requests) become steps of the session they name (X-Lorebase-Session) or, without one, of an inferred session (same principal, credential and user agent, calls less than inferred_session_idle_s apart, marked inferred) in the new table agent_trace_steps (migration 0023): tool (X-Lorebase-Tool), route, status, latency, response cache use, the query as structured data (filter values kept only on Attributes that are not sensitive, capture_filter_values), result counts, search corrections, context pack budget and the Records and Links the answer touched, never values; with X-Lorebase-Trace: debug and the new permission trace.explain, query plan summaries. Agents attach notes with POST /api/v1/trace/sessions/{id}/notes; GET /api/v1/trace/sessions and GET /api/v1/trace/sessions/{id} list sessions and replay one. A principal sees its own sessions; the new permission trace.read (held by admin) shows other principals' steps in a Domain, with their query and note text only to readers holding every sensitivity clearance there (withheld otherwise). Steps are written off the request path in batches, capped per session and deleted after retention_days (7). Herald forwards the session id of the MCP client or session (none when there is none), the tool name and a traceparent on every Keeper call; the Python client takes session_id, trace and tool and has a trace resource; the CLI has lorebase trace list|show; the web app has an Agent activity screen with the timeline and the touched subgraph on the Canvas, step by step.
Retrieval lab in the web app (retrieval lab): one question, two parameter sets side by side (mode, weights, thresholds, limits), lexical results with scores and corrections, semantic and hybrid ones when a query vector is pasted or an embedding service is configured (guideUrl in the web app's runtime configuration; the browser calls it, never Keeper), the context pack an agent would get with why each entry is in it and its tokens against the budget, and the involved Records on the Canvas. POST /search takes tuning (lexical_weight, semantic_weight, rrf_k, trigram_weight, and the work bounds typo_candidates and candidates, which may only be lowered, and trigram_threshold, which may only be raised) in place of the search settings for one request.
Schema suggestions from document extraction (extraction). Extraction jobs report what they see outside the schema: **Label:** value fields and table columns no Attribute matches, headings naming a thing by a code no Kind knows, spans and relations of the new scribe.extraction.discovery.span_labels and relation_labels, and, at tier 2, an unlisted list in the answer schema plus records, values and links outside the schema, all with verified evidence. Observations are matched against the schema (normalized singular names, synonyms, similarity), never kept when they match a sensitive Attribute, kept without evidence when the quote holds a sensitive value, and aggregated per Domain by fingerprint into suggestions with support, documents, examples, samples and, for Attributes, the type schema inference gives the samples; extracting a document again does not count it twice. GET /api/v1/domains/{domain}/schema/suggestions lists them (permission document.read; pending ones from schema.suggestions.min_support, at most max_suggestions) and POST .../schema/suggestions:decide rejects, resets or accepts them: accepted suggestions become one schema change set on a new branch (keeper.schema_suggestions.branch_prefix) proposed into the default branch, new Kinds and Relations as drafts. The web app has a Schema suggestions tab in Review and the Python client schema.suggestions, iterate_suggestions and decide_suggestions. New settings schema.suggestions, scribe.extraction.discovery and keeper.schema_suggestions; migration 0021 adds the schema_suggestions table. The tier 2 cache key changes when discovery is on, so the first run after upgrading calls the model again.
lorebase schema import --from linkml --apply merges into the current schema of the Domain instead of replacing it: Operations, Derivations, Sources and the fields LinkML cannot carry (derivation, authority, indexed, ...) are kept, elements match by id or name, and Kinds, Attributes, Relations and Traits missing from the LinkML file are kept with a warning unless the new --prune flag removes them; --dry-run shows the resulting change set (LinkML).
LinkML schema import and export (LinkML). lorebase schema export --format linkml translates the schema of a Domain, or a local document with --file, into a LinkML schema: Kinds become classes, Traits mixin classes, Attributes class attributes with their range, required, identifier, constraints and ifabsent, enums LinkML enums, and each Relation a pair of slots linked by inverse whose multivalued gives the cardinality. Element ids, sensitivity tags, search settings and the types LinkML lacks travel as lorebase.* annotations, Operations, Derivations and Sources as schema annotations, so importing the export gives back the same schema. lorebase schema import FILE --from linkml prints the Lorebase document (or writes it with -o), --apply and --dry-run send it to a branch like schema apply; what it leaves out (Operations, Derivations and Sources, is_a inheritance, which it flattens, abstract and tree_root classes, inlined objects, rules, any_of, enum meanings, custom type constraints) is reported as warnings with their path, and classes without slot descriptions or a title attribute are imported as draft Kinds. The translation needs PyYAML, in the new optional extra lorebase-cli[linkml].
Path Derivations work (design 5.8). A path Derivation is a virtual Relation (A -r1-> B -r2-> C read as A -> C) usable by its api_name or inverse_api_name wherever a Relation name is accepted from a Kind: Links of a Record and of a Relation, via and count predicates, include and traversal. By default its Links are composed at read time from the steps' Links (overlays and row-level security included); every later step follows at most store.traverse.path_fanout Links (100) per intermediate Record, smallest far keys first. With materialized: true the Links are stored in mat_links under the Derivation id and kept current in the commit transaction: only the start Records a changeset can affect are recomputed (above store.materialize.derivation_max_incremental_records, the whole path), schema changes and rebuilds rewrite them, and verify(..., derived=True) checks them. Both modes give the same Links, also at past Revisions, where they are composed from the tree. Following a path needs read access to every Kind it crosses. Path Links carry no values.
Trait queries at a past Revision (at): POST /query with trait lists the Records of every implementing Kind from the tree, in the head's order and with the head's cursors; filters, ordering by other fields, includes and aggregates stay refused there (422 time_travel), as for Kinds.
MySQL/MariaDB and SQL Server Sources: the types mysql_table, mysql_query, mssql_table and mssql_query read a table or a query like the Postgres ones (snapshot or incremental with a watermark compared in SQL, batches of fetch_rows, connection by allowed secret:// reference, statement_timeout_ms and connect_timeout_s in the new scribe.sources.mysql and scribe.sources.mssql settings) and can be profiled for schema inference. The drivers are optional extras and a run without one fails naming it: lorebase-scribe[mysql] (asyncmy; the session is read-only) and lorebase-scribe[mssql] (pymssql in worker threads, no ODBC driver to install; use a login that can only read, since SQL Server has no read-only transaction). Schema validation asks the new query types for a query, and the web Modeler lists the types and keeps them on a round trip (scribe).
Operation side effects (design 10.5, agents). A stored Operation's side_effects (unique snake_case names of at most 64 characters, now checked by schema validation as schema.api_name_invalid or schema.api_name_duplicate) make each committed execution write one operation.<name> outbox event in the transaction of its Revision, invocation record and audit entry, so the event exists exactly when the execution committed; dry runs, refusals, executions that need review (agent task writes that open a Proposal included) and idempotent replays write none. The payload names the execution, carries the parameters as recorded (possibly sensitive ones ***) and up to keeper.operations.side_effect_max_edits (100) edits. Webhooks subscribe with operation.<name> or operation.*; a webhook created without events now gets revision.committed and operation.*. Sensitive values in events reach a webhook only when it was created with matching clearances (pii, confidential, restricted; creating one needs that clearance permission on the Domain), others get webhooks.masked_value; vault values are stored as references and opened when the delivery is sent, so an erased one arrives as webhooks.erased_value. Every attempt of an event carries Idempotency-Key: evt_<sequence> (webhooks.idempotency_header): delivery is at least once and unordered. Each consumer of the outbox now keeps its own delivery state (outbox_pending, filled by a trigger in the writing transaction), so a second consumer no longer loses events to the webhook worker, and an event whose transaction commits late is not skipped. Outbox events are pruned once every consumer handed them off and they are older than the new store.outbox.retention_s (7 days; webhooks.retention_s now applies to deliveries only), or after store.outbox.max_age_s (30 days) in any case, store.outbox.prune_batch (10,000) at a time. New metrics lorebase.outbox.handoffs, lorebase.outbox.pruned and lorebase.operations.events. Migration 0020 adds outbox_consumers, outbox_pending and webhooks.clearances, moves undelivered events to the webhooks backlog and drops outbox.delivered_at and outbox.attempts.
Rate limits shared by every Keeper: keeper.limits.rate.backend: postgres keeps the token buckets in the application database (the UNLOGGED table rate_limit_buckets), so several Keepers enforce one limit instead of each its own. A Keeper leases a few tokens per query (postgres.lease_tokens, at most a quarter of a bucket) and serves requests from them, so most requests cost no query and the Keepers together never exceed a limit. Failed-authentication buckets are written on every failure and fail closed (429) when the database does not answer within postgres.timeout_s; the other buckets follow postgres.on_error (local, the default, enforces them in the process meanwhile; allow; deny). The default backend is now called local (memory still works). The Helm chart sets postgres when keeper.replicaCount is above 1 or the HPA is on (keeper.rateLimitBackend overrides it); compose reads KEEPER_RATE_LIMIT_BACKEND, to set with the scale-out profile. New metric lorebase.ratelimit.shared.queries (operations).
Auditing of reads (keeper.auth.audit.reads, off by default). Successful reads of authenticated callers are recorded in audit_read_log, a table with hash chains of its own (per Domain and installation-wide) so that sampled, droppable read records never touch the chains of actions. sample_rate, principal_types and kinds choose which reads are kept, except that a read that showed or masked a sensitive value (pii, confidential, restricted, vault values included) is always recorded, with sensitive: [shown|masked]. Records are queued and written in batches off the request path; a full queue or a failed batch drops records, counted in lorebase.audit.reads.dropped. python -m lorebase_keeper audit verify --reads verifies the read stream and the daily audit_verify job checks both streams (operations).
Signed checkpoints of the audit chains outside the database (scribe.audit_checkpoints, off by default). An audit_checkpoint job, every interval_s (one hour), signs the newest entry of every chain with an Ed25519 key and stores it in a directory (never replacing a file), at an HTTP endpoint or in an S3-compatible bucket (PUT signed with Signature Version 4), with the address checks and pinning of webhook targets. python -m lorebase_keeper audit verify --checkpoints PATH --checkpoint-key PUB reports checkpointed entries that are missing or changed, so a table owner who cuts the end of a chain or recomputes one is detected. New metric lorebase.audit.checkpoints (operations).
The five-day proof scenario runs in CI. The End-to-end workflow builds the backend image from the checkout (Docker layers cached), starts the demo stack with the deterministic fake inference provider and runs examples/five_day_demo/run.sh on pushes to main, nightly, on demand and on pull requests that change examples/, packages/, deploy/compose/ or deploy/docker/; on failure it uploads the stack logs and the run transcript. Every day of the demo now ends with assertions (demo.py assert-*: schema applied, import counts, merge with resolved Conflicts, Source runs and webhook, agent writes through Proposals and the audit log, restore equals live), a failed one exits non-zero, and the run prints the seconds each day took. Day 2 gained a document: a contract extracted with the deterministic tiers into a Proposal of items that is reviewed and merged, and the agent of day 5 now also opens a Proposal of its own with task and intent (its key gets the editor role on agent/*). DEMO_NO_BUILD=1 makes stack.sh up use a prebuilt image (demo).
Personal data vault and erasure (design 13.5). An Attribute tagged pii with vault: true stores its values encrypted (AES-256-GCM) in the vault table, and Record objects hold a token {"$vault": "vlt_..."} instead; hashes, diffs and merges compare tokens, which stay the same for the same value of the same Attribute of the same Record. POST /api/v1/domains/{domain}/vault/erase (new permission pii.erase, admin only among the built-in roles), lorebase admin erase KIND --key K or --attribute A --value V, and vault.erase_record / erase_value in the Python client erase a data subject's values on every branch and Revision, in one transaction with an audit entry that never holds the value; erasing again changes nothing, and reads show keeper.reads.sensitivity.erased_value ([erased]). Keys come from the new vault settings (token_key, encryption_key, key_id, previous_keys), the same for every process that commits; without them, writes of vault values are refused (value.vault_unavailable). Vault Attributes cannot be unique, indexed, derived, references, titles, primary keys or foreign keys, and vault storage cannot be turned on or off on an existing Attribute (schema.vault_invalid). References are accepted in writes only for the Record they belong to, and Conflicts over vault values are resolved by picking a side. Bundles carry tokens, never vault values, and their import refuses plain values in vault Attributes. lorebase admin erase --value - reads the value from standard input. Migration 0019 adds the columns that bind a vault row to its Record. The Modeler keeps vault when it edits a Kind.
Sensitivity masking on reads (design 10.2). Values of Attributes tagged pii, confidential or restricted are shown only to callers holding the new permissions pii.read, confidential.read or restricted.read (Domain-wide, on the branch read, or narrowed to the Kind; admin holds them, custom roles grant them); other callers get keeper.reads.sensitivity.masked_value (***), or nothing with omit: true. This covers Records, Links, query includes, traversals, neighborhoods, history and lineage values, diffs and changesets, explain and context packs, Conflicts and stored resolutions, and Operation edits, so the CLI, the Python client and the MCP server see the same. Filtering, ordering or grouping by a sensitive Attribute needs the clearance (403 sensitive_attribute), and by a vault Attribute is refused (422 vault_attribute). Responses are cached and tagged per clearance. Clearances come from Domain-wide grants or grants narrowed to the Kind (by its id on the default branch), not from grants narrowed to a branch; the tags of the default branch, carried through Derivations, apply to every Revision and branch read; removing a tag on any branch, by a schema change or a merge, needs its clearance (403 sensitivity_lowered), and writers without a clearance check (agent auto-merge, Source merges) cannot. Derived Attributes and the step values of Operations must carry the tags of what they read, preconditions cannot read sensitive Attributes (schema.sensitivity_inherited), sensitive Attributes cannot be primary or foreign keys, and uniqueness violations on them no longer name the other Record. Extraction items are masked too, Operation invocation records and Proposal descriptions no longer keep parameter values that may be sensitive, and Source run errors are shown only to callers cleared for every tag of the schema.
Copies and deletions of large branches run in the background, so they no longer hit database.statement_timeout_ms (at 1M Records a copy took 85 to 95 s and failed through the API). A copy branch whose source holds more than store.materialize.async_copy_min_rows Records and Links (10,000) is created at once as an overlay of its source, readable and writable right away, and a store.materialize job copies its rows in batches of job_batch_rows, one transaction each, resuming from its cursor after a failure, then switches the branch to a full copy. Commits on the branch and on its source stay correct during the copy and wait for at most one batch; a schema change on the branch is refused with 409 branch-materializing and Retry-After until it is done. Deleting a branch with more than async_delete_min_rows rows hides it at once and a store.branch_delete job deletes its rows in batches. At most job_concurrency of these jobs run at once across workers. Branch responses have materialization_status (pending, running, done, failed) and materialization_progress; lorebase branch create --mode copy --wait waits for the copy, and lorebase branch list|get show it. Overlays promoted because their base moved too far are copied the same way. New metrics lorebase.store.branch_jobs.rows, .runs and .batch.duration. Migration 0017 adds refs.deleted_at and the branch_materializations table (operations, store schema notes).
The embedding model of a Domain is advertised to agents: GET /api/v1/domains/{domain}/schema has a search block with the search modes and, when a Kind has search.embedding_template, embedding (model, dims) read from search.semantic.model and store.embeddings.dims; lorebase_schema shows it as a SEARCH line, and the Python client has SchemaDocument.search.
inference.embeddings.query_prefix and document_prefix (empty by default) for models trained with prefixes, such as E5: lorebase-inference embed --query TEXT puts the query prefix before the text, and Scribe puts the document prefix before every embedded Record text. Changing the document prefix makes the next pass embed every text again. lorebase-inference embed TEXT without --query no longer adds anything, which is what it did before prefixes existed (inference providers).
Garbage collection deletes the embeddings no branch head references (for example the vectors of Records that only a deleted branch held); python -m lorebase_store.gc --dry-run and the gc job report their count as embeddings (operations).
The Modeler edits search.embedding_template and search.embedding_sensitive on the Kind form, checks the placeholders against the Kind's Attributes (unknown, derived and sensitive ones) before a change is sent, and follows an Attribute rename in the template.
Embedding jobs for semantic and hybrid search. A Kind opts in with search.embedding_template in the schema ({attribute} placeholders by api name, plus {_kind} and {_key}; sensitive Attributes only with search.embedding_sensitive: true; invalid templates are reported as schema.embedding_template_invalid). Scribe's scheduler queues an embeddings job per Domain every scribe.scheduler.embeddings_interval_s (60 s): it embeds, branch by branch, the Records that lack a vector of the configured inference.embeddings model, re-renders the Kinds whose template changed, skips texts whose hash did not change and copies the vector of an identical text instead of calling the model. Vectors stay keyed by content hash, so overlay and copied branches only add work for the Records they change. Batching, a row budget per job with resumable continuations, call pacing and retries come from the new scribe.embeddings settings; the model's dimension is checked against store.embeddings.dims before the first call. lorebase-scribe embeddings DOMAIN [--branch B] [--reindex] [--prune] backfills, reindexes and, after a model switch, deletes the vectors of other models. New metrics lorebase.scribe.embeddings.records and lorebase.scribe.embeddings.call.duration. Migration 0014 adds embeddings.text_hash and the embedding_cursors table (scribe).
Query vectors on the caller's side, since Keeper and Herald never call a model: lorebase-inference embed TEXT prints the vector of the configured embedding model, and lorebase search TEXT --vector-file FILE (- for standard input) sends it for hybrid search (inference providers).
Per-Record history and lineage: GET /api/v1/domains/{domain}/kinds/{kind}/records/{key}/history lists the Revisions that changed a Record on a branch, newest first, with the changed Attributes, their provenance and, with values=true, the values before and after; GET .../records/{key}/lineage tells per Attribute where the current value came from (provenance, Revision, Source, Operation, extraction evidence, Proposal). Both read one index range of the branch history, are authorized and cached like Record reads, and are bounded by the new read.history settings. Also in the Python client (records.history, records.lineage), the CLI (lorebase record history|lineage), the MCP server (lorebase_history, lorebase_lineage) and the History tab of the Explorer Record page.
Merge policies authority and newest_wins (design 6.10). authority (globally or per Attribute in authority_attributes) is a ranked list of origins such as source:<id>, principal:<id>, actor:<id> or origin:extraction: on an attribute value Conflict the side whose last change comes from the higher ranked origin wins, and schema_authority: true ranks an Attribute's schema authority Source first. newest_wins (or newest_wins_attributes) takes the side whose last change has the later provenance time, equal times going to the greater provenance id. Both read the Lineage of both branches, leave the Conflict to a person when they cannot decide, and record why they decided in the attempt's auto_resolutions[].details, which the Review screen shows. Set them in the Domain's merge_policy setting; Source merges apply them too.
Sign-in for the web app with Keeper as backend-for-frontend (keeper.auth.sessions, off by default; needs an oidc auth mode). GET /api/v1/auth/login runs the OpenID Connect authorization code flow with PKCE against the configured provider, GET /auth/callback checks state, nonce and the ID token and sets an opaque HttpOnly, SameSite=Lax session cookie (__Host- prefixed and Secure unless cookie_secure: false), GET /auth/session returns the user and the session's CSRF token, and POST /auth/logout ends the session and returns the provider's end-session URL. Sessions live in the new web_sessions table (migration 0013; only a SHA-256 of the session id is stored, provider tokens are encrypted with sessions.encryption_key) with idle and absolute timeouts, a new id on every sign-in and refresh-token checks at the provider. A session maps to the same principal, groups and grants as the user's bearer tokens; POST, PUT, PATCH and DELETE with the cookie need the CSRF header, a trusted Origin and no cross-site Sec-Fetch-Site. API keys, bearer tokens and auth.mode: none work as before. The web app gets VITE_AUTH_MODE=session (authMode in /config.js): a sign-in screen at /login, a sign-out button, cookie requests with the CSRF header, and a redirect to sign-in on 401 (deployment).
Operation, Derivation and Source editors in the web app Modeler (tour). Operations: typed parameters, preconditions, steps with their Kind or Relation and values, review policy, permissions, side effects, rate limit, and a dry run of the saved Operation on the Branch. Derivations: expression, aggregate and path variants, with the computed Attribute picked from the schema. Sources: the fields of each type (records_path, flatten_separator, http headers and pagination for http_json), secret:// references only for connections and credential headers, key, Attribute and Relation mapping, sync mode with watermark, schedule and merge policy. Expressions are edited in a CEL editor, and the findings of Keeper's dry run, the CEL type check included when schema.expressions_enabled is on, appear next to the field. Edits go through the same change set, Branch and Proposal flow as the other editors, and keys a form does not edit stay as the YAML has them.
Helm chart values for the web sign-in: config.keeper.auth.sessions settings, the OIDC client secret and the session encryption key read from the Secret (secrets.keys.oidcClientSecret, secrets.keys.sessionEncryptionKey, or secrets.values.* for a chart-managed one) and wired as secret://env references, web.config.authMode: session, render-time checks, and the ci/sessions-values.yaml overlay (Kubernetes).
Audit chain verification (operations). python -m lorebase_keeper audit verify DOMAIN|--installation|--all recomputes the hash chains of the audit log with the maintenance DSN, prints the first entry that does not verify (or JSON with --json) and exits 1 when there is one; --after-seq resumes from the last_seq of an earlier run. GET /api/v1/domains/{domain}/audit/verify (permission audit.read) and lorebase audit verify (exit code 2 on a broken chain) check one Domain's chain, and the Python client has audit.verify. Entries are read in batches, so memory does not grow with the log. Scribe queues an audit_verify job of every chain each scribe.scheduler.audit_verify_interval_s (one day; 0 disables it), counted in the new metric lorebase.audit.verifications by outcome, and the LorebaseAuditChainBroken alert fires when a chain is broken.
A threat model: STRIDE per trust boundary (web sign-in, API keys and OIDC, authorization in Keeper, Scribe connectors and document intake, Herald, Postgres roles and row-level security, webhooks, inference providers, the release pipeline), each mitigation with the file that implements it, and the residual risks in order of priority.
An API stability policy for /api/v1: what is stable, which changes are compatible or breaking, the deprecation process with Deprecation and Sunset headers (none is deprecated yet, so Keeper sends neither), client compatibility, MCP tool stability and the lorebase.schema/v1 format. It binds from 1.0; before it, breaking changes are named in the changelog.
Webhook clearances can be changed: PATCH /domains/{domain}/webhooks/{id} with {"clearances": [...]} replaces them, audited as webhook.update with the clearances before and after. Each clearance added needs its clearance permission on the whole Domain, as at creation; removing one needs only domain.settings. The Python clients have webhooks.set_clearances (agents).
Storage profiles (performance): store.storage.profile (compact, balanced, fast; compose LOREBASE_STORAGE_PROFILE, Helm config.store.storage.profile and ci/storage-<profile>-values.yaml with matching Postgres settings) decides what the database keeps besides the rows: shared or inline lineage maps, the GIN index on Record values, the per-Attribute indexes of indexed Attributes, the trigram index of the typo lexicon, half or full precision for the HNSW index, TOAST compression and the fill factor of mat_records; every other key of store.storage overrides one choice. Every feature works in every profile. python -m lorebase_store.storage status|apply|prewarm shows, applies (batched, resumable, reads go on; --compact --reindex moves rows into free space and gives pages back) and prewarms (pg_prewarm, autoprewarm with shared_preload_libraries). The sizing guide gives the memory that keeps the agent read set resident at 1M, 10M and 100M Records per profile. At 1M Records the hot set of the agent read path went from 12.5 GB to 5.5 GB (balanced, after apply --compact --reindex); at 10M Records the hot set is estimated at 49 GB against 57 GB measured before (the 10M after-run is pending).
Shared lineage maps (migration 0036, store schema notes): mat_records.lineage, lineage_revisions and mat_links.lineage hold a reference to a map stored once per Domain (mat_lineage_maps) instead of the map; readers resolve it with lb_lineage. Record rows are about 55% smaller. tests/perf/working_set.py measures relation sizes, pages per read and buffer residency; tests/perf/halfvec_recall.py the recall of a half-precision HNSW index.
Example questions in the schema: domain.examples of lorebase.schema/v1 lists optional {label, question} pairs (at most 20, unique labels; design 5.13). They are part of the Domain element, so changing them is a schema change, and GET /domains/{d}/schema returns them in its domain block. The web app offers them in Explore and the Retrieval lab only when present. Schemas without them load and hash as before. examples/supply/schema.yaml declares the five questions of the example agent.
VITE_DOCS_URL (docsUrl in /config.js): the documentation site behind the web app's "Learn more" links (default https://aantenore.github.io/lorebase, none hides them). LOREBASE_GUIDE_PROXY makes pnpm dev and pnpm preview proxy /guide to a guide service.
pnpm screenshots --live --keeper <url> --domain <id> --guide <url> takes the tour pictures from a running stack read only: the existing Domain as it is, a real Explore answer and its trace (apps/web).

Changed¶

.env.example no longer sets GUIDE_ALLOW_ANONYMOUS=false (the compose default is still false); with make up PROFILES=assistant it defaults to true. A value set in an existing .env is kept. The UV_SYNC_ARGS comment no longer lists fastembed, which the lorebase-ml image carries.
Agent trace: trace_all no longer traces the web app's own requests (agent trace). The web app sends X-Lorebase-Client: web on every Keeper request; Keeper leaves out of trace_all the requests carrying it (keeper.trace.client_header, web_client) and those authenticated by a browser session cookie that name no session. keeper.trace.include_web: true traces them again. Guide and Herald sessions, header opt-ins and listed principals or keys are traced as before. The header is added to the CORS allowed headers when tracing is enabled.
Agent trace: values compared by filters are kept on Attributes that are not sensitive (agent trace), so a timeline shows single_sourced = true instead of {"redacted": "boolean"}. The new keeper.trace.capture_filter_values (non_sensitive by default, or none) decides; a field is kept only when the read's own field check finds it not sensitive under the effective tags used for masking (through Traits and Derivations; title and key by their title and primary key Attributes). Kept values are scalars or lists of scalars, cut to max_text_chars and the new max_filter_value_items (20); they are shown like the rest of the query, to the caller and to readers cleared in the Domain only. Set capture_filter_values: none for the earlier behavior.
Snapshot builds for reads at a past Revision are bounded: one build per Domain at a time, read.time_travel.snapshot_max_concurrent_builds per process with snapshot_build_wait_ms of waiting (503 with Retry-After), snapshot_builds_per_minute and snapshot_build_burst per principal (429), snapshot_max_history_revisions of history (422), and a snapshot_min_age_s grace period before a snapshot may be dropped for another; requests are validated before a build, and snapshots use the sensitivity tags of their Revision's schema.
Large branch copies never run in one transaction any more (store schema notes). A copy above store.materialize.async_copy_min_rows that cannot start as an overlay (a copy of an overlay, from an older Revision, or with the overlay cap reached) is created building (migration 0031) and a store.materialize job writes its rows from the tree of its start Revision in batches; meanwhile its head is read like a past Revision and commits into it answer 409 branch-materializing. Overlays over a base above that size are no longer promoted inline: their limits queue the batched copy, and a rebuild of the base detaches them and builds them from their trees.
Schema changes on an overlay whose background copy is queued or running are accepted instead of answering 409 branch-materializing: the change applies to the rows the commit writes and the copy job reconciles derived values, path Links, titles and search vectors in batches after its switch. A schema change on an overlay over a large base queues that copy instead of promoting the overlay inline.
Background copy and deletion batches size themselves by the time they hold the branch lock: from store.materialize.job_batch_min_rows (100) up to job_batch_rows, aiming at job_batch_target_ms (200 ms) and at job_batch_contended_ms (50 ms) once a commit waited for the lock, so a commit on a branch being copied no longer waits up to 0.5 s for a batch. New metric lorebase.store.branch_jobs.batch.rows.
Schema suggestions no longer keep evidence or samples that hold a vaulted value of any vault Attribute of the Domain (extraction). Extraction compares keyed fingerprints of the quote's word sequences with the live vault rows and opens nothing. It fails closed: a quote past discovery.vault_match_max_text_words, vault_match_max_text_chars or vault_match_max_candidates, or any quote of a Domain with vaulted values when Scribe has no vault keys, loses its evidence. A suggestion without evidence looks like one that never had any; the job report no longer carries suggestion_evidence_withheld, and operators get the counter lorebase.extraction.evidence_withheld.
mypy no longer carries a project-wide override for pyarrow (reported as an unused section when only some packages are checked): the three optional pyarrow imports ignore its missing type information where they are.
Sensitive Attributes are left out of full-text search for every caller: their words are no longer in search vectors, the lexicon, corrections or snippets (a searchable sensitive Attribute gets the warning schema.sensitive_attribute), and a Kind whose title Attribute is sensitive has Records without a title. Semantic and hybrid search use the vectors of a Kind that embeds sensitive Attributes only for callers cleared to read them. Rows materialized before keep their titles and search vectors until the Kind's definition changes on that branch (any edit of the Kind, such as its description, recomputes them); materialize.verify reports such rows until then.
Validation messages of schema migrations no longer quote values of sensitive Attributes.
Deleting a branch above store.materialize.async_delete_min_rows rows returns 204 at once while its rows are deleted in the background; until then its name stays taken, and creating a branch with it answers 409 already-exists with Retry-After. Copies above async_copy_min_rows answer with materialization: overlay until their copy is done. Set both thresholds very high to keep copies and deletions in the request transaction.
Typo-tolerant search matches words instead of whole titles. Each branch keeps, per Kind, the distinct words of its search vectors (mat_lexicon, kept by triggers on mat_records in the writing transaction; an overlay adds its base's), and a query word the searched Kinds do not hold is corrected to their closest words by trigram similarity (negated words are left alone), then searched as (word | correction ...). Hits of the query as written rank first, the others after them at search.trigram_weight times their rank. POST /search responses have corrections (word, replacements; empty when nothing was corrected), also in the Python client (SearchResult.corrections) and in lorebase_search. search.trigram_threshold is now the least similarity of a correction (default unchanged, 0.3), and search.typo_candidates (3; 0 turns it off) and search.typo_min_length (4) are new. Words are lexemes of the text search configuration, so a stemming configuration reports stems. Migration 0018 adds the table, its triggers and backfill (writes to mat_records wait while it reads every search vector), the read functions lb_search_corrections and lb_search_corrected (grant them to an application role created later with grant_read_functions, as before), and drops the title trigram index of 0011. At 1M Records typo-tolerant search takes 7.6 to 14 ms at p95 instead of 116 to 123 ms; the triggers add about 5% to a commit of 1,000 changes at 1M Records and about 12% at 100k (performance, design 8.4).
The performance harness (tests/perf/bench.py) reports p99 next to p50 and p95, times Record history and lineage, and with --seed-database NAME keeps the loaded dataset in its own database and measures a copy of it, so a run at 1M Records loads once. The 1M-Record run is in performance: every read is within budget except typo-tolerant title search (about 120 ms at p95 against 50 ms), the 2-hop neighborhood on branches other than main (29 to 34 ms against 30 ms) and lineage against the 10 ms proposed for it; creating or deleting a copy branch at that size takes longer than the default database.statement_timeout_ms.
The web app Canvas lays large graphs out in a web worker instead of on the main thread: the graph is drawn at once and refined while it is on screen, the layout stops when it settles or at a time budget and can be paused, and large graphs use Barnes-Hut, hide edge labels and most node labels, and hide edges while panning. On a generated graph of 5,000 nodes and 10,000 edges the first render takes about 0.6 s instead of 1.7 s (performance). The limits are configurable (VITE_CANVAS_*), and pnpm perf:canvas with VITE_MOCK_PERF_NODES measures a generated graph.
Herald gives each Keeper request its own span: a request hook on the Keeper client sends a traceparent with a fresh span id whose parent is the span of the MCP tool call, in the caller's trace, so the requests of one tool call are told apart in the agent trace and in tracing backends. When an OpenTelemetry span is active in Herald, each request is a child of that span instead; with herald.trace.enabled off no traceparent is sent. The Python clients take event_hooks, passed to httpx (agent trace).
Operation executions that need review now emit their operation.* events when their Proposal merges (agents). An agent task write keeps its events in operation_deferred_events (migration 0028); the merge of a Proposal from the task branch writes those whose Revision the merged source head contains to the outbox and deletes them, in the merge transaction, so each is written once. They name the target as branch, the merged head, source_branch and proposal, and are counted in lorebase.operations.events when the merge commits. Events of a task branch deleted unmerged are dropped, and so are those older than store.outbox.max_age_s.
The web app is domain-agnostic and its copy is shorter. No screen names the Kinds, Records or questions of a dataset (the supply presets, placeholders such as a question about bearings in Torino, and example keys are gone; mock data stays in the mock fixtures), and no description puts the Domain's name into its text. Every user-facing string lives in apps/web/src/messages: a heading and at most one short line per screen, with "Learn more" links into the docs, consistent terms and no internal jargon. src/test/domain-agnostic.test.ts fails on dataset identifiers or an interpolated Domain name outside the mock fixtures and tests. The "planned screen" placeholders (Explorer, Modeler, Review, Sources, Settings, the Record page's Run Operation and the Canvas diff overlay tab) are removed.
Web app layout: shared primitives (PageHeader, Section, Card, CardGrid, FormRow, FormAction) give every screen the same spacing, equal-height cards and buttons on the line of their inputs; the navigation becomes a scrolling row below 768px, tab lists scroll instead of overflowing, and tables, long keys and JSON no longer push screens wider than a phone. The Explore panel is no longer repeated in the Retrieval lab: the lab links to Explore ("Ask the guide") when a guide service is configured, and its parameter fields have readable labels.
The tour pictures, the README gallery and docs/tour.md show the current app on a demo stack with real data, a real Explore answer and its Agent activity trace, Record history, the Derivation preview, typo-tolerant and hybrid search, extraction items and schema suggestions; four pictures come from the mock and say so.

Fixed¶

run_index_jobs (lorebase_store.indexes) sends job heartbeats while a build runs, so a long index build is no longer reclaimed as lost after store.jobs.stale_after_seconds. Releasing a job on SIGTERM and reclaiming one after SIGKILL were checked in containers (validation).
Jobs of a Scribe that is killed or loses the database no longer stay running for an hour: a job whose worker sent no heartbeat for store.jobs.stale_after_seconds, now 90 seconds (was 3600, measured from the last heartbeat; Scribe sends one every scribe.worker.heartbeat_interval_s, now 30 seconds, was 60), is reclaimed by the scheduler leader at its next tick: queued again, or failed once it used store.jobs.max_attempts, so a job that keeps killing its worker is not run forever (scribe). A worker that lost a job this way no longer overwrites the outcome of the run that replaced it (its run counts as lost in lorebase.scribe.jobs), and python -m lorebase_store.jobs work now sends heartbeats too.
A stopping Scribe (SIGTERM) leaves no job running even when a handler does not stop when cancelled: shutdown_cancel_s (scribe.worker, 5 seconds) after shutdown_grace_s its jobs are put back in the queue anyway, without counting the attempt.
Agent trace: calls without a session id of one client no longer split into many one-call inferred sessions when they run concurrently (agent trace). A step carries the time its request started but is written when the request finishes, and the grouping only looked back from each step, in write order: a call that started before the last one written found no session and opened its own (the web app's parallel reads opened a new session every few milliseconds). Inferred steps of a batch are now grouped in time order and join the session of the nearest step of their client within inferred_session_idle_s on either side.
Guide in the compose stack and the Helm chart has a writable state directory (/var/lib/lorebase/guide, volume guide-data; Helm guide.persistence, an emptyDir by default) while its root file system stays read-only, so it remembers structured output modes across restarts and no longer logs a warning at each start. The example agent reports the keys of the edits and the Proposal ids from the response to the committing call, not those of its dry run.
Hybrid search ranks its results by fused score again: Records found only by the semantic list were listed before every lexical match.
Guide's query tool told the model a where and order_by shape Keeper's query DSL refuses ({attr: value}, {attr: asc|desc}), so in live runs most failed tool calls were refused queries and many explorations ran out of steps. The description now gives the DSL's predicate forms and a valid example (checked against the DSL in tests), and the history, neighborhood and traverse descriptions state their bounds (validation).
A Guide exploration whose token budget ran out after several useful reads answered Not found in the graph.: only the step budget left a turn to answer. A tool step is now taken only when the budget still covers it, the next call with a full tool result and an answer; otherwise that turn is the final one (guide). The input estimate takes its characters-per-token ratio from the endpoint's count of the previous call (JSON tool results ran at about 2.2 on Claude models, not 4), and the answer of the final turn is kept when the endpoint counted more than estimated. The instructions now ask for a partial answer with citations rather than found: false when part of the question is answered, and treat a checked negative result (no Record meets the conditions) as an answer. The tokens of a model answer that does not parse now count towards the exploration's total, and the model call failed log line names the schema problems (paths and keywords, never answer text).
Guide's neighborhood and traverse tools dropped the Link Attributes Keeper returned (a stocked quantity, a lead_time_days), so the model answered that the graph held no quantities. Edges now carry their values.
The example process agent sent temperature: 0 on every request, which current Claude models refuse with HTTP 400. --temperature none (AGENT_TEMPERATURE) leaves it out and --extra-body (AGENT_EXTRA_BODY) passes options such as thinking (process agent).
Herald no longer logs every Keeper request URL, Record keys included, at INFO through the HTTP client's logger; over stdio those lines went to the agent's standard error.
Record lineage stays bounded on Records with long histories: changes are counted up to read.history.lineage_count_max (10000; changes_truncated reports more), and rows without lineage revisions search only the newest read.history.lineage_scan_changes changes (1000). The new store.lineage_backfill job fills lineage_revisions for rows materialized before migration 0032. The creating change falls back to the history when the Revision stored for it has no history row, maps that are not JSON objects read as empty, and a merge attributes only the values it changes, so values it leaves keep the Revision that set them (agents).
Record lineage after a fast-forward names the Revisions of the source branch that set each value, as a rebuild does, instead of the source head; rebuilds and backfills attribute a merge only the values it changed, and python -m lorebase_store.jobs lineage-backfill --recompute (job payload "recompute": true) corrects rows written by merges before that fix. Workers starting together queue the lineage backfill once.
After a fast-forward, the branch history holds one row per change of each fast-forwarded Revision, as a rebuild writes it, so Record history and the lineage change count match a rebuild (store.materialize.fast_forward_history_max_revisions, 1000; a longer range keeps one row per element at the source head). lineage-backfill --recompute rewrites the history, Records and Link lineage of earlier fast-forwards the same way.
A background copy that keeps failing no longer holds up the deletion of its base for good: each failed job is retried after a delay doubling from job_retry_seconds (at most copy_retry_max_seconds), and after copy_max_failures failures the overlay is detached from its base and built from its tree instead. New outcomes retried, fallback and given_up of lorebase.store.branch_jobs.runs and alerts LorebaseBranchCopyFallback and LorebaseBranchBuildFailed (operations).
Derivation and Source previews (tour). A previewed value computed from sensitive Attributes of another Kind is masked unless the caller is cleared on the Derivation's Kind and on every Kind the values come from, and a masked item reports a failure without its error code (error.code is now nullable). keys are bounded by read.derivation_preview_max_key_length (1024). POST /sources:preview accepts a schema.read grant narrowed to the Source's target Kind and answers 403 for other Kind-narrowed callers before validating the change set; it refuses rows of more than read.source_preview_max_columns columns (200) and requests over read.source_preview_max_cells evaluations (1000, rows times expression mappings) with 422, and evaluates off the event loop. Expressions over rows are compiled once over every column of the Source, so a column a row lacks is null in Scribe and in the preview alike (it was an unknown-variable error), and very large decimals keep their exponent form.
Path Derivations: values over the inverse direction stay current when a Link pushes another far key out of an intermediate Record's store.traverse.path_fanout bound. Job workers rewrite branches computed with another fan-out in the background next to their job loop instead of before it, and a branch that fails or is being copied in the background no longer stops or blocks the sweep. Operation links() over a path reads its first step no further than store.reads.max_page_rows, and neighborhoods following a composed path report truncated only when Links were left out.
Embedding vectors are keyed by the fingerprint of the embedding template as well as by content hash and model (migration 0024). Branches whose templates differ for the same content keep a vector each instead of overwriting one another, and Keeper's semantic and hybrid search read the vectors of the searched branch's template. Vectors of identical texts are still copied without a model call, now across templates too. The upgrade sets the fingerprint wherever every branch holding the content agrees on one template, and the next pass re-renders each text once to fill in the rest. Garbage collection also deletes vectors of templates no branch uses any more. Scribe resolves embedding templates once per schema hash and keeps the last scribe.embeddings.template_cache_size schemas.
Database sources of schema inference are bounded further: at most store.profiling.sql.max_tables tables (now 50), total_timeout_s for a whole request, and max_value_chars per value, with unbounded text columns cut and binary or unknown-type columns not read. A schema must be a plain name found in the catalog, and keeper.schema_inference.allowed_schemas can restrict a source to some of them. A failed or misconfigured source answers a generic 503. Previews cut long text (preview_max_cell_chars).
LorebaseGcNotRun no longer fires for a gc schedule set to dry runs (scribe.scheduler.gc_dry_run: true). Scribe counts the collections it schedules in lorebase_gc_scheduled_total{dry_run}, and the alert needs real scheduled collections (dry_run="false") that never complete within 3 days. The Scribe dashboard shows scheduled runs next to completed ones.
lorebase_jobs_queue_depth reports jobs waiting for a retry delay (queued with a run_after still ahead) as status="retrying", so queued counts only claimable jobs and LorebaseJobQueueStalled no longer counts jobs no worker could claim yet.
LinkML: path Derivations are exported as slots carrying their steps, inverse name and materialized as annotations, and schema import --apply reads them back, so a round trip keeps them. Applying to a Domain no longer warns that Operations, Derivations and Sources are not imported when the merge keeps them, and --prune without --apply or --dry-run is refused instead of doing nothing.
Large commits no longer slow down with the number of Links they write. Looking existing Links up by (from, to) pairs planned one index scan per pair (about half a millisecond of planning per pair, so 0.5 s for a chunk of 1,000) and now runs as one semi-join on two arrays (13 ms for the same chunk); cardinality and required-Link checks count the batch's Links per key once instead of once per key; canonical JSON writes strings without characters to escape in one step. Loading 28,000 Records and 84,000 Links in 15 commits went from 67 to 48 s on the same machine.
Commits no longer slow down with the size of the Kinds they write. The search vectors of a commit's Records were refreshed by filtering every Record of the Kind on the commit's Revision (7 s per statement for a Kind of 200,000 Records) and are now refreshed by key; unique values were checked with an expression the Attribute's unique index does not match (a minute per thousand values at 160,000 Suppliers) and now go through the index.
The background copy of a branch no longer fails at 10M Records before copying anything: its first step counted the base branch's rows for the progress report (85 million rows, past the statement timeout). rows_total is now the planner's estimate until the copy switches, then the rows read.
Schema suggestions no longer list one fact twice under two names: Attribute suggestions of the same Kind whose names share at least schema.suggestions.merge_similarity of their words (0.6) and whose samples share a value type are folded into the better supported one (renewal_notice_days and renewal_proposal_notice_days, termination_notice_days and termination_for_cause_notice_days). Kinds named by a generic document or workflow word (schema.suggestions.kind_stoplist) or by an Attribute (Buyer beside buyer) are not suggested, and a proposed Kind is listed once it has kind_min_attributes (2) Attribute suggestions or as many observations.
Tier 2 extraction asks for link Attributes: a Relation with link Attributes has its own branch in the answer schema with the link's values (evidence for each, the required ones required), and the prompt asks for the required ones. Link values are verified and grounded like record values and reach the Link item; before, stocked_at Links came without quantity. A Link item still missing a required link Attribute stays pending, with the names in match.missing.
A structured output mode an endpoint refuses is remembered for the process, per base_url, model and output settings, instead of being tried again at the start of every extraction job (two refused requests per job with a strict-only endpoint before).
A tier 2 chunk whose answer stays invalid after the repair attempts no longer fails the whole extraction job: the chunk is skipped with a warning, counted as model_invalid_answers, and its tokens still count toward the job budget and usage. An answer cut off before any output says that a reasoning model may have spent the token limit on reasoning.
Tier 2 timestamp values written without an offset (2024-01-23, 2024-01-23T10:15) are read as UTC instead of being discarded as invalid, and timestamps are grounded by their day in the quote ("approved on 23 January 2024").
A webhook receiver that disappears no longer holds a Scribe worker. Each delivery attempt, host lookup and every checked address included, is now cut off after webhooks.timeout_s; before, the budget applied to each address in turn, the lookup had none, and one pass sent up to webhooks.batch deliveries one after the other, so a receiver that accepted connections and never answered, or dropped packets, kept a webhook_delivery job running for many minutes while its heartbeat kept it from being requeued. Connections give up after the new webhooks.connect_timeout_s (5 s) and try the next checked address; a pass sends webhooks.concurrency (8) deliveries at a time, starts none after webhooks.pass_timeout_s (60 s) and releases the rest to the next pass without counting an attempt; a pass still running after webhooks.lease_s fails its job attempt. webhooks.lease_s must now exceed pass_timeout_s + timeout_s (the defaults do). Timed-out attempts are retried with backoff and marked dead after max_attempts as before, and counted in the new metric lorebase.webhooks.timeouts (operations).
The compose db-roles step grants the application role EXECUTE on all eight security definer read functions of a database migrated before the role existed (it granted only the four of revision 0005), reading them from the catalog like the Helm chart's role setup does.
Guide no longer overshoots its token budget by up to the steps times max_result_chars: before each model call it estimates the input (prompt, messages and tool results so far) and calls only when used plus estimated input fits in guide.agent.max_total_tokens, with the completion cap cut to what remains after the estimate. The session event now names the chat model and the effective max_steps of the run (guide).
Outbox pruning keeps up with high event rates: a pass deletes batches of store.outbox.prune_batch events, each in its own transaction, until nothing is left or the new store.outbox.prune_budget_s (5 s) is spent, instead of one batch per pass. The new gauge lorebase.outbox.prune.backlog reports the events still prunable after a pass (counted up to 100,000) and the alert LorebaseOutboxPruneBacklog fires when it stays above 50,000 for 30 minutes (operations).
The per-session step cap of the agent trace holds across Keeper replicas, and listing sessions no longer scans the steps of the retention period (agent trace). Migration 0029 adds agent_trace_sessions, whose step count the trace writer reads and raises under the row lock in the transaction writing the steps (steps over keeper.trace.max_steps_per_session are dropped and counted as session_full), and agent_trace_session_domains, a summary per session and Domain (first and last step, calls, notes, tools, Proposals; indexed on principal and on Domain with the last step) that sessions are listed from under the same visibility rules as steps. Pruning lowers both and recomputes the summaries from the steps left; both tables are filled from the existing steps.

Security¶

Derived values whose sensitivity is inherited from other Kinds are masked unless a Kind-narrowed clearance covers every source Kind, on every read path: Records, lists, queries, search, history, lineage, diffs, explain and context packs, as in the Derivation preview. Filtering, ordering or grouping by such a value needs the same clearance (403 sensitive_attribute). Domain-wide clearances are unaffected; webhook clearances are Domain-wide and change feed events carry no values (design 13.5).
The structured output mode memory stores only keyed digests, never an endpoint address, and its table grants the application role no TRUNCATE, REFERENCES or TRIGGER. Closing the evidence gap for vaulted values leaves one residual inference: a reader of suggestions can tell that a quote they chose matched from the missing evidence (threat model).
Vault keys rotate (design 13.5). lorebase admin vault rotate (POST /api/v1/vault/rotate, permission vault.manage, held by installation administrators only) queues the vault.rotate job, which seals every live vault row of every Domain again under vault.encryption_key and makes its fingerprint again under the current token key, vault.rekey_batch rows per transaction, continuing where it stopped; lorebase admin vault status (GET /api/v1/vault/keys) counts rows per key and names the earlier keys no row uses. vault.token_key now rotates too: give the new key a token_key_id and keep the old one in previous_token_keys; tokens already stored keep working and a value written again finds its old token, so nothing reads as changed.
Erasure covers what it missed. Erasure by value compares matching forms (case, compatibility characters and white space ignored), so Ada@Example.com and ada@example.com are erased together. Erasure by key follows key changes: a commit that deletes a Record and writes another of its Kind holding its vault references (a key changing Operation, say) now stores those values again under the new key instead of being refused (once per reference, into a Record that does not exist yet), and records the lineage that erasure follows both ways. Erased values leave a tombstone (their fingerprint): a later write of one by a Source sync, extraction or an import (vault.tombstone_origins; writes of people and agents are not checked, so they cannot test whether a guessed value was erased) is stored erased (vault.erased_values: erase, the default), refused (refuse, issue value.vault_erased) or stored again (allow, the old behaviour). Commits that store vault values and erasures take the Domain's vault lock (shared, waiting at most vault.lock_timeout_ms, and exclusive), so a value is written either before an erasure, which erases it, or after, which finds its tombstone. The audit entry of an erasure by key holds a keyed digest of the key, no longer the key.
The parameter digest of Operation invocations is always keyed: under vault.digest_key, else the vault token key (as before), else a random secret that migration 0027 stores in installation_secrets; it was a plain SHA-256 without vault keys.
Health reports no longer list which Records share a sensitive unique or title value to callers without the clearance: they count the groups and Records (duplicates.hidden). Validation issues about values the caller may not read carry the generic value.invalid code. A vault reference nested inside another Attribute's value is refused (value.vault_reference). The NDJSON diff stream and Operation edits show vault values, [erased] or the mask, as other reads do, instead of raw references.
Operation sensitivity checks resolve the fields an expression reads against the Kind the type checker infers (comprehension variables over links() or lists of Records, conditional results, nested traversals, record($step[n])), so a same-named sensitive Attribute on another Kind no longer refuses a harmless Operation; only fields of untyped (dyn) values are still matched by name. Evaluation errors are shown in full to callers cleared for the Attributes the expression reads on the Kinds that hold them, not only to callers cleared Domain-wide.
Branches forked before the default branch tagged an Attribute can take its sensitivity tags: POST /domains/{d}/branches/{b}/rebase-tags (lorebase branch rebase-tags BRANCH) commits the tags (never removing one) and recomputes the titles and search vectors the branch stored before them. With rows_only, or automatically for every other branch when the default branch tags an Attribute (store.materialize.refresh_branches_on_tag, on by default), a store.sensitivity_refresh job rewrites those rows in resumable batches. Extraction decides what to extract by the effective tags.
Evaluation errors of Operation steps (a value that does not fit its Attribute, a created Record that exists already) are redacted when the step reads or writes values the caller may not read, and an expression the redactor cannot parse is redacted rather than shown.
python -m lorebase_store.bundle export --include-vault-values writes the vault values of the exported Records in plain text (or erased) in vault.jsonl, listed in the manifest, readable by its owner only and audited as bundle.export_plain without the values; import seals them under the target's vault keys after checking that each token is held by that Record and Attribute in the bundle. A restore from a bundle no longer has to lose vault values.
Read auditing gains keeper.auth.audit.reads.on_overflow: reject, which answers a read with 503 and Retry-After when its record cannot be queued and keeps retrying failed batches (turning readiness failed) instead of dropping them, so the read stream can serve as proof.
Operation edits resolve only vault Attribute values bound to their own Kind and Attribute, on a copy of the changes; revalidation issues on values the caller may not read are counted per Attribute instead of listed by Record; markdown context packs and explanations resolve vault references from structured data, never by matching text; writes that show sensitive values say so in their audit entry; Conflicts show vault values instead of references.
Sensitivity refresh jobs follow the effective tags (derived title and searchable Attributes included), also run for tags on Attributes new to the default branch and for copies of stale branches, are no longer swallowed by a running job, and are queued only for branches with stale Kinds; the tag rebase lists added tags only to callers who may read the default branch's schema.
Erasure takes the Domain's vault lock first (vault.erase_lock_timeout_ms, then 503) and reads the key changes under it, so a key change committing meanwhile is erased too; it follows any number of key changes, in batches, and flags an unusually long lineage in its audit entry instead of refusing. A token a key change took over is marked moved (migration 0033) and can be neither taken over again nor claimed back.
Bundle exports with --include-vault-values carry the vault's key changes and erasure tombstones, read under the Domain's vault lock and audited at start and end; import refuses tombstones made under a token key it does not have, checks values against the bundle's schema and against tombstones as vault.erased_values says, and treats a Domain with leftover vault rows as not empty.
New Domain permission pii.import (only admin among the built-in roles): values of vault Attributes can be fed through the built-in import and through Sources (defining or mapping such a Source, triggering its sync) only with it, checked before anything is written, so callers without it cannot test whether a value was erased.
Vault bundles carry the moved_to of key-changed tokens and legacy tombstones (as <token_key_id>/1, noted in the manifest), accept only key changes between Records the bundle holds, record each vault import in a bundle.import_vault audit entry with the count of oversized key-change groups, and hold the vault lock only while the export's snapshot is taken.
Erasure makes outdated fingerprints current before taking the vault lock and checks again under it; a key change replayed on another branch is allowed as the same move, a later rename carries the pointer of earlier moved tokens along, and erased references move without being marked moved.
Reviewer edits of extraction items that set a vault value need pii.import, Scribe never auto-accepts extraction items holding vault values, and python -m lorebase_keeper sources vault lists Sources that map vault Attributes so ones defined before pii.import can be confirmed or disabled.
lorebase admin erase --value takes - (standard input, asked without echo on a terminal) or @FILE; a value on the command line is refused, as it stayed in the shell history and the process list.
Keeper sends X-Content-Type-Options: nosniff, X-Frame-Options: DENY, Content-Security-Policy: frame-ancestors 'none', Referrer-Policy: no-referrer and, over https (directly or from a proxy in keeper.trusted_proxies with X-Forwarded-Proto: https), Strict-Transport-Security; keeper.security_headers configures or turns them off. The web image sends a Content Security Policy whose connect-src holds the Keeper origin of the runtime configuration (LOREBASE_WEB_CSP_CONNECT_SRC adds sources, LOREBASE_WEB_CSP replaces the policy, LOREBASE_WEB_HSTS adds HSTS), and the Helm chart has the same through web.security. The web image's Referrer-Policy changes from same-origin to no-referrer (deployment).
The application role can no longer change the audit log: migration 0015 revokes UPDATE, DELETE and TRUNCATE on audit_log from PUBLIC, from database.application_role and from every other role but the table owner, and the compose db-roles script no longer grants them. Entries are still appended and read as before; nothing in Lorebase prunes the audit log, and an operator who must do so works as the owner (deployment).
Document intake by URL no longer trusts the allowlisted host's DNS: Keeper resolves the host once, refuses it with 422 unless every address is global unicast (loopback, private, link-local, shared and other non-global ranges are refused, as for webhook targets), and connects only to the checked addresses, so a changed DNS answer cannot reach an internal service. keeper.documents.url_allow_private_targets (false) turns the check off for hosts on an internal network.
Scribe parses PDF and DOCX documents in a child process without network access, stopped after scribe.extraction.parser.timeout_s (60 s) of wall-clock time, cpu_s (60) of CPU time or memory_mb (1024 MiB, where the system enforces address-space limits), and refuses documents above max_input_bytes (50 MiB). A crafted document that hits a limit fails its extraction job with a message naming the setting instead of holding the worker; isolated: false returns to parsing in a worker thread (extraction).
Webhook signing keys can be stored encrypted: with webhooks.encryption_key (a secret:// reference to at least 32 characters, the same for Keeper and Scribe) new keys are sealed with AES-256-GCM, the delivery worker seals the keys stored before, and previous_encryption_keys lets the key change without breaking deliveries. Receivers change nothing. Migration 0016 adds webhooks.sealed_key; its downgrade refuses while sealed keys exist (agents).
Supply chain: the base images of the backend and web images (and the compose proxy image) are pinned by digest with the tag kept readable, and moved by hand (release); the build arguments PYTHON_TAG, NODE_TAG and NGINX_TAG are replaced by PYTHON_IMAGE, NODE_IMAGE and NGINX_IMAGE. A CI workflow (audit.yaml) runs pip-audit on the locked Python dependencies and pnpm audit --prod on every pull request and weekly, failing on advisories not accepted in .github/audit-allowlist.yaml; make security uses the same check.

Not done yet¶

The performance baseline on the 1M-Record fixture: the neighborhood on branches other than main is just over budget when the working set does not fit in memory (performance).
The performance targets of the design at 10M Records: the 2-hop neighborhood, full-text search (searches of very common words above all), Record history and lineage are over budget, and a commit of 1,000 changes takes 3.3 s (performance).
Full RBAC: groups from the OIDC groups claim, Kind-scoped grants on reads and sensitivity masking are done; Kind-scoped invisibility in queries is not (roadmap).
The stability guarantees for /api/v1: the policy is written and binds from 1.0; Keeper sends no Deprecation or Sunset header yet, since nothing is deprecated (API stability).
Packages on PyPI: the wheels and sdists are attached to the GitHub release.

Upgrade notes¶

In this order: configuration and secrets, then the migrations, then the restart and the one-off jobs. Keeper and Scribe of 0.1.0 must not run against the migrated database, except as noted for rolling upgrades.

Before upgrading: configuration and images

Compose: run make env once to add LOREBASE_WEBHOOK_KEY to .env; values already set there are kept. Helm: set secrets.values.webhookEncryptionKey (or the webhook-encryption-key entry of an existing Secret), or config.webhooks.allow_unsealed_keys: true for development; the chart refuses to render otherwise. Elsewhere, set the same webhooks.encryption_key for Keeper and Scribe.
Helm database roles: give the Secret db-maintenance-dsn (the owner, the old db-dsn) and point db-dsn at the application role (lorebase_app by default; the migration Job creates it when the owner has CREATEROLE). With postgresql.enabled, set secrets.values.dbAppPassword (or create the lorebase-db-app basic-auth Secret); one helm upgrade is enough, as the pg-roles hook declares the role before the migration Job. With restricted egress, allow the API server (networkPolicy.egress.kubernetesApi). database.allowSharedRole=true keeps one DSN, for development only.
Delete read.history.lineage_max_changes from configuration files.
Choose a storage profile with store.storage.profile (compose LOREBASE_STORAGE_PROFILE, Helm config.store.storage.profile with the Postgres settings of ci/storage-<profile>-values.yaml); without one, the indexes stay as they are.
Scribe: keep scribe.worker.heartbeat_interval_s at most half of store.jobs.stale_after_seconds (a deployment that sets only the heartbeat above 45 seconds must lower it or raise the threshold), and SCRIBE_STOP_GRACE (compose) or scribe.terminationGracePeriodSeconds (chart) above scribe.worker.shutdown_grace_s plus shutdown_cancel_s (35 seconds by default; the defaults are 45 and 60).
Callers that read sensitive values need pii.read, confidential.read or restricted.read through a custom role in keeper.auth.authz.custom_roles. To use the vault, set the same vault.token_key and vault.encryption_key (secret references, 32+ characters) for Keeper and Scribe; rotate the token key with token_key_id and previous_token_keys, never by replacing it.
With more than one Keeper outside Helm, set keeper.limits.rate.backend: postgres.
Images: make ghcr.io/aantenore/lorebase-ml reachable where Scribe and Guide run, or set Helm imageMl.repository: "". Give Guide a writable state directory (see Breaking changes).
Web image: a VITE_KEEPER_BASE_URL given only as a build argument does not reach the Content Security Policy; set it at run time or list the Keeper origin in LOREBASE_WEB_CSP_CONNECT_SRC.

Migrations 0013 to 0036

Run them (alembic upgrade head; the compose migrate service and the Helm hook do it) before restarting Keeper and Scribe, with database.application_role set when not on compose or Helm, so that the grants of the application role follow. On compose, db-roles keeps UPDATE and DELETE on audit_log and audit_read_log revoked on every start.

0013 web sessions; 0014 embedding jobs.
0015 makes audit_log append-only for the application role (revokes UPDATE, DELETE and TRUNCATE); revoke them yourself if the migration runs without database.application_role.
0016 adds webhooks.sealed_key; its downgrade refuses while sealed keys exist.
0017 background branch copies; 0018 the per-branch search lexicon (its functions stay for Keepers of an earlier version after 0035).
0019 vault bindings; 0020 outbox consumers; 0021 schema suggestions.
0022 read audit log and shared rate limit buckets. python -m lorebase_keeper audit verify --json gains a stream member and the audit_verify job result a reads member.
0023 agent trace; 0024 keys embedding vectors by template fingerprint; 0025 the snapshot ref kind used by reads at past Revisions.
0026 restores the plain reverse Link index, built concurrently per partition, only on a database that ran its withdrawn first version.
0027 adds vault.fingerprint_key and the tables vault_tombstones, vault_lineage and installation_secrets (with a random digest secret). Operation invocations recorded without vault keys keep their plain digest, so replaying one of their idempotency keys answers a conflict.
0028 deferred Operation events.
0029 agent trace session rows. A Keeper of an earlier version writes and prunes steps without updating them: turn keeper.trace.enabled off on old Keepers during a rolling upgrade.
0030 mat_schema.path_fanout. Branches with materialized path Derivations, or derived values reading a path, are recomputed once by the first commit on the branch or when a job worker starts.
0031 the building materialization and the build, retry and reconcile columns of branch_materializations; Scribe workers of an earlier version must not run store.materialize jobs afterwards.
0032 mat_records.lineage_revisions; until the backfill below, lineage reads the missing Revisions of earlier rows from the branch history.
0033 the moved_to of key-changed vault tokens.
0034 inference_endpoint_modes; downgrading drops it and the modes are learned again from the next rejections.
0035 lb_search_bounded and lb_search_bounded_overlay, which lexical search calls; the candidates are capped by store.reads.definer_ceilings.candidates (100,000).
0036 shared lineage maps (mat_lineage_maps, read through lb_lineage). On an existing install it shares nothing yet (see After the restart); its downgrade writes every map back into its rows.

After the restart

Run a job worker for store.lineage_backfill (Scribe runs it by default; python -m lorebase_store.jobs work too): it fills mat_records.lineage_revisions of the rows materialized earlier, branch by branch in batches of store.materialize.job_batch_rows.
Once every process runs 0.2.0, run python -m lorebase_store.storage apply as the maintenance role (compose: docker compose --profile maintenance run --rm storage apply) to share the lineage maps and apply the storage profile, then apply --compact --reindex to give the space back. Both are batched, resumable and leave reads running; the 1M-Record benchmark database took 26 and 7 minutes on a loaded machine. python -m lorebase_store.storage status shows what is left and prewarm loads the read set into memory.
Run lorebase admin vault rotate once: rows stored before keep the fingerprint of their exact value until rotated (an erasure by value rotates the rows it looks at first).
Run python -m lorebase_keeper sources vault (--check exits 1 when any is listed): Sources that map vault Attributes and were defined before pii.import existed keep syncing on their schedule until you confirm or disable each one.
Existing webhooks keep working; their signing keys are sealed by the next delivery passes.
Branches forked before the default branch tagged an Attribute take its sensitivity tags with POST /domains/{d}/branches/{b}/rebase-tags.

0.1.0 - 2026-10-21¶

First release. Lorebase keeps a typed ontology in Postgres and lets teams change it on branches, review the change in a Proposal and merge schema and data together, with Conflicts resolved per Attribute. Agents read through a REST API that never calls a model, or through the MCP server, and write only through Operations; a write that needs review becomes a task branch and a Proposal. Documents are extracted into Proposals whose items reviewers accept or reject. Pre-alpha and pre-1.0: /api/v1 is not frozen and interfaces may change in a minor version. The database schema is at migration 0012. The lines of this first section link the documentation of each feature; from the next release they link the pull request or issue.

Added¶

Schema and versioning (design, schema as code)

A Domain is a YAML schema document (format: lorebase.schema/v1) with Kinds, Attributes, Relations, Traits, Operations, Derivations and Sources, checked statically and applied as one schema change set: POST /api/v1/domains/{domain}/branches/{branch}/schema/changes, lorebase schema validate|diff|apply. Attribute ids are stable, so a rename keeps its history.
Content-addressed Revisions, branches, tags, diff and time-travel reads (?at=<revision>, lorebase diff, lorebase record get --at).
Three-way merge of schema and data with typed Conflicts (attribute value, delete or modify, add or add, missing value, type coercion, cardinality, dangling Link and more), resolved with ours, theirs, base or a custom value, one by one or from a file (lorebase proposal conflicts|resolve). Merge policies prefer, cascade and defaults. The merged data is validated against the merged schema before the commit.
Proposals with reviews, required approvals (keeper.proposals), Conflict resolution and merge over REST and CLI (lorebase proposal create|review|merge|close). The default branch of a Domain is protected (keeper.domains.protect_default_branch): changes reach it through a Proposal.
Expressions in CEL for Operations (preconditions, steps, review policies) and for expression and aggregate Derivations, type checked when a schema is applied (schema.expressions_enabled) and recomputed in the materialized head on every commit. path Derivations are not computed.

Reads for people and agents (agents, performance)

Records, Links, the JSON query DSL, traversal, neighborhoods and lexical search over REST (lorebase record, query, search), with ETags and the serving Revision in every response. Full-text search reads GIN indexes on (domain_id, branch, search) and (domain_id, branch, title gin_trgm_ops) (migration 0011, btree_gin), so a search reads only its own branch's matches, however many branches share the partition. Semantic search takes vectors supplied by the caller; nothing computes embeddings.
Context packs within a token budget (POST .../context, lorebase context), Record explanations with the origin of each value (GET .../records/{key}/explain, lorebase explain) and data health reports (GET .../health, lorebase health), all computed without a model.
A change feed as Server-Sent Events and as polling (GET .../branches/{branch}/events, .../changes) and signed webhooks (POST .../webhooks, Lorebase-Signature HMAC).
Overlay branches (store schema notes, migration 0010): a branch created with mode: overlay (POST .../branches, default store.materialize.branch_mode, always for names under agent/, extract/ and source/, and for agent task branches per keeper.agent_writes.branch_mode) stores only the rows it changes and reads its base as of the fork point. Creating one does not depend on the size of the Domain; it is promoted to a full copy above store.materialize.overlay_max_rows rows or overlay_max_base_lag_revisions generations of base lag, before a schema change and before its base is deleted. lorebase branch create --mode copy|overlay chooses the mode from the command line. Overlay reads go through _overlay variants of the security definer functions with the same argument and scope checks. An overlay gets the per-Attribute indexes of its schema on its own rows, built as store.index jobs when it is created, and its neighborhood looks Links and Records up with one index scan over both branches (migration 0012); on the 100k-Record baseline search, attribute queries, Links of a Record and the neighborhood stay within 30% of a copy at p50 (performance).
Per-Attribute indexes and security definer read functions that keep Domain isolation under row-level security. At 99,900 Records and 302,940 Links on one laptop, the measured p95 of every agent read stays within its budget (performance); the 1M-Record baseline is not measured.

Writes and ingestion (quickstart, scribe, inference)

Built-in Operations import_records, import_links and import_batch with dry run and idempotency keys, and custom Operations declared in the schema (POST .../operations/{operation}/execute).
lorebase import records|links|batch load CSV, TSV, JSON Lines or JSON (Parquet with the parquet extra): types are converted from the schema, a Record travels with its Links, files are cut into batches of one Revision each.
Schema inference from CSV, TSV, JSON, JSON Lines (.jsonl, .ndjson) and Parquet uploads, chosen by suffix (keeper.schema_inference.file_formats), and from Postgres tables (POST .../schema:infer): Kinds, typed Attributes, keys, Relations with cardinality, a confidence and the evidence for every decision. It proposes and writes nothing. Parquet uploads need the lorebase-keeper[parquet] extra; without it they are refused with a 422 naming it. A record profiler (lorebase_store.profiling.profile_records, RecordProfiler) and profile_source in Scribe profile any Source through its connector; limits are in store.profiling.records.
CSV and Postgres Sources with snapshot and incremental syncs run by Scribe, the worker and scheduler on the Postgres job queue. A sync commits on a Source branch, opens a Proposal and applies the Source's merge policy (proposal, auto_when_clean, auto_with_authority); every changed value keeps its row as Lineage (lorebase sources sync|runs).
JSON, JSON Lines, Parquet and HTTP JSON Sources (scribe): the json and jsonl types and the records_path, flatten_separator and http options (headers as secret references, cursor, page, offset or Link header pagination, a watermark parameter and a page cap). Files are opened under the allowed roots without following links, URLs are checked against private addresses, JSON Lines and Parquet are streamed, and an HTTP JSON run that passes max_pages fails instead of committing part of a snapshot. Parquet needs the lorebase-scribe[parquet] extra.
Document intake and tiered extraction into Proposals (extraction): POST .../documents takes Markdown, plain text, PDF and DOCX (PDF and DOCX with the scribe documents extra); Scribe parses and chunks them and proposes Records, Links and values with verified evidence, from rules (tier 0, no model), local GLiNER and GLiREL (tier 1) or any OpenAI-compatible model with JSON schema output (tier 2, cached, token budget per job). Reviewers accept, edit or reject items one by one or by filter (GET .../proposals/{p}/items, POST .../items:decide, lorebase documents upload, lorebase proposal items|decide); nothing reaches the default branch without a merge. Derived and sensitive Attributes are never extracted. GET .../documents/{id}/text (lorebase documents text) serves the parsed text the evidence offsets index. The web app has a Documents screen (intake, library, a viewer that highlights the evidence in the parsed text) and an Items tab on extraction Proposals to accept, edit or reject items one by one or by filter.
Inference providers behind one OpenAI-compatible interface for chat and embeddings, plus local fastembed, GLiNER and GLiREL (optional extras), and compose profiles for Ollama and a LiteLLM proxy. Extraction uses them; nothing computes embeddings yet.

Agents and access (MCP, operations)

Herald, the MCP server, over stdio and Streamable HTTP with 11 tools (lorebase_context, lorebase_query, lorebase_execute with dry run and more), two resources and one prompt. It is a thin layer over Keeper and holds no database credentials. lorebase_execute passes intent, auto_propose and auto_merge to Keeper also when the call names no task, and returns Keeper's problem text unchanged; auto_propose: true without a task is refused as validation-failed.
Agent writes that need review (agents): an Operation refused with 409 review-required (protected branch or review policy) runs, when the call names a task, on a task branch agent/{principal}/{task} and opens or updates one Proposal, merged at once only when keeper.agent_writes.allowed_roles_for_auto_merge allows the caller's role, nothing conflicts and the required approvals are in. Without a task the 409 carries next hints. Settings keeper.agent_writes, overridable per Domain.
API keys (hashed with a server-side pepper), OIDC bearer tokens, role grants per Domain, Kind, Operation and branch, and a hash-chained audit log (lorebase apikey, GET .../audit). python -m lorebase_keeper bootstrap creates the first installation administrator key from the database settings and prints it once; it refuses when an administrator already holds a usable key, unless --force.
Rate limits per principal and request class (RateLimit-* headers, 429 with Retry-After) and daily quotas per API key, shown by lorebase whoami (keeper.limits.rate, keeper.limits.quota). Failed authentications are counted per client address and per key prefix; keeper.limits.rate.auth_failures.per_key_prefix_enabled: false counts them per client address only, so that someone who knows a key prefix cannot keep that key throttled.
Domain backup and restore bundles (python -m lorebase_store.bundle export|import) and garbage collection of unreachable Revisions with a dry run (python -m lorebase_store.gc).

Clients and web app (web app, tour)

The lorebase command line and a Python client, sync and async, checked against the committed OpenAPI document (docs/openapi.json). Exit codes are documented: 0 success, 1 usage or not found, 2 validation or conflict, 3 authentication or permission, 4 server error.
A web app with Explorer, Canvas, Modeler (including a Start from data wizard over schema inference that accepts CSV, TSV, JSON, JSON Lines and Parquet files), Review with a Conflict resolver and an Items tab for extraction Proposals, Documents, Sources and Admin screens. It runs on the mock API or on Keeper, with end-to-end tests against a real Keeper for browsing, Branches, schema changes, Review and Settings. It reads /config.js (window.__LOREBASE_CONFIG__: apiMode, keeperBaseUrl, an optional apiToken or any VITE_* name) over its build values, so the lorebase-web image switches to live mode at run time, from VITE_* variables of the container or the chart's web.config. Screens load on demand: the main chunk is 193 kB, vendor libraries sit in their own chunks, and pnpm build fails when the main chunk or the first load exceeds apps/web/bundle-budget.json. The Modeler and the Sources editor keep the Source options they do not show when they save a schema.
The five-day demo: make demo brings up a dedicated compose stack and runs the proof scenario of design 3.2 with the CLI, in under a minute once the image is built (docs/demo.md).

Deployment (deployment, kubernetes)

Docker compose for Postgres 16 with pgvector, migrations, Keeper and Scribe, with profiles local-models, litellm, observability and scale-out (a second Keeper, a Postgres streaming replica and nginx; read replica routing with a freshness guard).
A Helm chart (deploy/helm/lorebase) for Keeper, a migration hook, Scribe, Herald, the web app, Ollama, Ingress, autoscaling and read replica DSNs. The Postgres parameters of the CloudNativePG cluster are explicit (postgresql.cluster.postgresql.parameters, max_connections: "100" by default); size it above Keepers x (pool_size + max_overflow) plus Scribe, migrations and administration. It passes helm lint --strict, helm template and kubeconform; it has not run on a live cluster.
An observability bundle in deploy/observability: Grafana dashboards for Keeper, Store and Scribe, Prometheus alert rules with a runbook per alert and the OpenTelemetry Collector configuration (operations). The compose observability profile loads them; the Helm chart renders them as ConfigMaps with observability.enabled=true.
A metrics catalogue (docs/metrics.md) with the Prometheus series, labels and source of every metric, among them the response cache, authentication failures, commit duration, merge attempts, security definer read calls, jobs claimed and queue depth, webhook deliveries, garbage collection runs and bytes freed, and extraction items. A test fails when a dashboard or alert rule names a series that is not emitted, or when an instrument is missing from the catalogue. Latency histograms declare explicit bucket boundaries from 5 ms to 10 s.
Release artifacts: images ghcr.io/aantenore/lorebase (Keeper, Scribe and Herald share it; the command picks the service) and ghcr.io/aantenore/lorebase-web, tagged 0.1.0, 0.1 and latest, signed with cosign keyless and carrying a CycloneDX SBOM attestation; CycloneDX SBOMs and the Helm chart package on the GitHub release. The nine Python workspace packages go to PyPI only when the maintainers configured publishing.

Documentation

A documentation site at aantenore.github.io/lorebase, built from docs/ with Material for MkDocs (make docs, make docs-serve) and published on every push to main. The command line and configuration reference pages are generated from the code (make docs-generate; scripts/docs/generate.py --check fails on a stale page) and the REST API page renders the committed docs/openapi.json.

Security¶

A security review of the trust boundaries before this release (docs/security-review.md) fixed all 23 findings, among them: a Source could resolve any secret reference, including the database DSN (critical), Source errors echoed resolved secrets, keeper.auth.mode: none could be served on a non-loopback address (now refused unless keeper.auth.allow_insecure), webhook targets in non-global address ranges were accepted, and a slow webhook target could stall delivery for every Domain.
Compose: Keeper and Scribe connect as an application role without superuser or BYPASSRLS (LOREBASE_APP_USER, LOREBASE_APP_PASSWORD, created by the one-shot db-roles), so row-level security applies to them; the owner role runs the migrations and index maintenance. The litellm and observability profiles refuse to start while LITELLM_MASTER_KEY or GRAFANA_ADMIN_PASSWORD is empty, change-me or shorter than 16 characters.
Migration 0008 adds argument, scope and search_path checks to the security definer read functions and refuses to run while the application role may create objects in their schemas; migration 0009 adds client network allowlists to API keys (allowed_networks, lorebase apikey create --allow-network, behind keeper.trusted_proxies).
Bundle import is bounded and refuses links, schema inference sources are configured per Domain, the OIDC issuer and signing keys need https, CSV Sources check the resolved address and open files without following links, failed authentications are rate limited per client address and key prefix, the Helm NetworkPolicy is on by default with narrow egress, provider error text is scrubbed, and open event streams are authorized again at an interval.

Not done yet¶

Lineage and history endpoints beyond the lineage summary on Record reads.
The 1M-Record performance baseline and the 10M-Record targets of the design.
Merge policies authority and newest_wins.
Schema suggestions from extraction; embedding jobs and hybrid search (semantic search ranks stored vectors against a query vector the caller supplies); path Derivations.
Events declared by Operations (the outbox and signed webhooks carry Revision events only), sensitivity masking, the PII vault, and Traits in time-travel reads.
Web app: OIDC login with BFF sessions, the Operation, Derivation and Source editors of the Modeler, and a Canvas that renders 5k nodes and 10k edges smoothly.
A command or endpoint that verifies the audit chain, LinkML import and export, the threat model, the proof scenario as an automated test in CI, and the stability promise for /api/v1 that comes with 1.0.

Upgrade notes¶

There is no earlier release. For a new installation, run the migrations (alembic upgrade head, migrations 0001 to 0012) before Keeper starts: the compose migrate service and the Helm hook do it. Migration 0011 creates the btree_gin extension when it is missing (it ships with Postgres contrib); on a database that already holds data it builds the search indexes concurrently per partition, without blocking writes. Keeper's shipped configuration accepts every request as an administrator (keeper.auth.mode: none); use api_key or oidc for anything shared, give api_key modes a pepper of at least 32 characters, list the administrator in keeper.auth.installation_admins and create the first key with python -m lorebase_keeper bootstrap.