Skip to content

Roadmap

The public roadmap of Lorebase, by milestone. It is derived from section 18 of the design and from what is on main today. A checked box means the item is merged on main and covered by tests. An unchecked box is not done, even when part of it exists; the note says what is missing. Dates are deliberately absent: the order is firm, the timing is not.

Lorebase is pre-alpha. v0.3.1 is the current pre-alpha release (the first was v0.1.0) (CHANGELOG); interfaces may change.

Next up

  1. The 10M-Record benchmark on the 32 GB reference hardware of design 6.14, and the 1M-Record neighborhood on branches other than main, which is just over budget on a shared laptop (M5).
  2. API stability guarantees for /api/v1, which bind from 1.0 (M5).

Everything else on this page is merged and covered by tests, except allowed merge origins per protected branch (noted under M3). The last designed block, knowledge that stays correct, is done. The ordered task list of M1 is in the M1 plan.

M1: Engine and API

Goal: define a Domain in YAML, import data, branch, change, merge with Conflicts resolved through the API, diff, and read Records in milliseconds, all through REST and CLI.

Engine and storage

  • Canonical JSON and content hashing, type system and value encoding
  • Schema element models with static validation; schema as YAML (load and dump, stable ids, renames keep identity)
  • Canonical HAMT trees, tree diff, immutable Revisions, ancestry and merge base
  • Data validation against the schema
  • Three-way merge of schema and data, full Conflict taxonomy, resolutions, merge policies prefer, cascade and defaults
  • Three-way merge of long text values by sentence or paragraph inside the data merge (design 6.8 step 3), and the Conflict resolver showing a text Conflict passage by passage (tour, governance)
  • Merge policies authority and newest_wins
  • Postgres schema and Alembic migrations, row-level security, store repositories
  • Commit pipeline, materializer and verification
  • In-memory merge demo (examples/merge_demo.py)
  • CEL expression engine with type checking and evaluation budgets (lorebase_core.expr), and expression checks in schema validation
  • Per-Attribute partial indexes on materialized heads, maintained on schema changes when the commit role owns the tables

Keeper REST API

  • Application skeleton: settings, structured logging, OpenTelemetry, error format, health and readiness
  • Authentication with API keys and OIDC bearer tokens, role grants per Domain, Kind, Operation and branch, hash-chained audit log
  • Domains, branches and tags
  • Schema read and validate, schema change sets
  • Built-in Operations: create, update, delete, link, unlink, bulk import, bulk delete, with dry run and idempotency
  • Records and Links, the query DSL, traversal, neighborhood, lexical search, time travel, ETags and conditional reads
  • Revisions, branch log and diff
  • Proposals, reviews, required approvals, merge and Conflict resolution API, scoped to what the caller may read
  • Lineage and history endpoints: per-Record history and lineage over REST, client, CLI, MCP and the web Explorer
  • Expressions (CEL) end to end: schema change sets type check them (schema.expressions_enabled), Keeper executes custom Operations with them and the materializer recomputes expression and aggregate Derivations

Clients and operations

  • Python client (sync and async), checked against the committed OpenAPI document
  • CLI (lorebase)
  • Docker image and docker compose for Postgres, migrations and Keeper
  • CI: lint, types, import contracts, unit and integration tests; CodeQL, Scorecard, SBOM and pinned actions
  • Inference interfaces and a deterministic fake provider (no model calls in M1; the read path cannot import them)
  • Read replica routing with a freshness guard and a scale-out compose profile (deployment). The design lists this under M5; it landed early. Individual read endpoints adopt it incrementally.
  • Performance baseline on the 1M-Record fixture (the harness and a 100k-Record baseline are in performance: every read is within budget for both a role not subject to row-level security and one subject to it, through the security definer read functions and the branch-scoped search index of migration 0011. The 1M-Record run is in performance: key, page, Links, attribute and full-text reads, Record history and the neighborhood on main are within budget for both roles; typo-tolerant search, about 120 ms at p95 with whole-title trigrams, is within budget (7.6 to 14 ms) since it corrects words from a per-branch word lexicon (migration 0018), the neighborhood on other branches is just over it (29 to 34 ms) when the working set does not fit in memory, which the reference-hardware item of M5 covers, and copy branches are created by a job in batches instead of one request, migration 0017)
  • M1 documentation and release checklist (release, CHANGELOG)

Acceptance (design 18): a Domain applied from YAML, filled by CLI import of 1M Records, branched, merged with Conflicts resolved through the API and diffed, with property tests for hashing, merge and diff laws, and the read targets met on the 1M fixture. Met, except the neighborhood on branches other than main on a shared laptop (see the M5 reference-hardware item).

M2: Structured ingestion and exploration

Goal: days 1 and 2 of the proof scenario run end to end.

  • Scribe job runner on the Postgres job queue, with a leader-elected scheduler (scribe)
  • CSV, Parquet and Postgres Sources, snapshot and incremental sync, plus JSON and JSON Lines file Sources (scribe); Parquet Sources need the lorebase-scribe[parquet] extra
  • MySQL/MariaDB and SQL Server Sources (mysql_table, mysql_query, mssql_table, mssql_query) with the optional lorebase-scribe[mysql] (asyncmy) and lorebase-scribe[mssql] (pymssql) extras (scribe)
  • Schema inference from CSV, JSON, JSON Lines and Parquet files and Postgres tables, with a golden test on the example supply schema (inference); any Source can be profiled through its connector, but syncs do not run inference
  • Source branches, merge policies and Lineage for syncs
  • OIDC login with BFF sessions for the web app: authorization code with PKCE, server-side sessions in Postgres (migration 0013), CSRF checks, refresh and RP-initiated logout (deployment); the web app signs in with VITE_AUTH_MODE=session
  • Web app shell, Explorer, Canvas (graph and schema graph) and Review screens on the mock API
  • Web app Modeler "Start from data" wizard over schema inference, a Sources screen (list, runs, YAML editor) and Admin basics (API keys, principals and grants, audit log) on the mock API; Source runs and Sync now use Keeper's Source routes in live mode
  • Web app verified against a live Keeper (Playwright flows against Keeper in CI)
  • Review: Proposal list and detail, reviews, data diff and merge against live Proposals
  • Canvas rendering 5k nodes and 10k edges, with a worker layout (first render of the generated graph in about 0.6 s instead of 1.7 s, see performance); frame times were measured on a software WebGL only

M3: Governance and agents

Goal: day 5 of the proof scenario, an external agent connected through MCP.

  • Herald, the MCP server: tools, resources and prompts, over stdio and Streamable HTTP (MCP server)
  • Custom Operations with CEL, review policies, outbox and webhooks: Operations declare side effects, written to the outbox in their commit transaction and delivered at least once to signed webhooks with masking of sensitive values (agents, design 10.5)
  • Full RBAC: groups from the OIDC groups claim, Kind-scoped grants on reads, sensitivity masking with pii.read, confidential.read and restricted.read (design 13.5), and Kind-scoped invisibility in queries: Kinds a Kind-narrowed caller may not read are left out of the schema it reads and answer like unknown names (design 13.3)
  • Branch protection on the default branch
  • Proposal checks (merge, approvals), required approvals and required roles (required_roles, design 10.3); allowed merge origins per protected branch are not done yet
  • Modeler with every editor: Kinds, Relations, Traits, Operations (with dry run), Derivations and Sources in one form and YAML document, with Keeper's validation and CEL type check next to the fields
  • Conflict resolver against live data
  • Traits in queries, at branch heads and in time-travel reads (listings ordered by key, with the head's cursors)
  • Reconciliation queue for contradictions between Sources, separate from merge Conflicts (design 10.6): reconciliation_candidates, optional validity periods on provenance, the valid_time Attribute flag and the reconciliation origin type (migrations 0038 and 0039); detection stages identity, values, normalization and the optional semantic stage in Scribe, with the extraction and Source sync triggers and the reconcile_sweep job; decisions written through Operations with provenance and audit, dismissal and reopening, duplicate merges as Proposals and the Proposal merge check; REST routes, Python client and CLI (lorebase reconcile); the Reconciliation screen in the web app (Scribe, tour)

Goal: day 3 of the proof scenario, on the sample contracts.

  • Document store interface and parsers (Postgres blob backend; Markdown, text, PDF, DOCX)
  • Tiered extraction: rules, local GLiNER and GLiREL, then any LLM with JSON schema output; cache, verified evidence, entity resolution, schema suggestions aggregated over documents and accepted as schema change Proposals (migration 0021, docs/extraction.md)
  • OpenAI-compatible chat client, fastembed and OpenAI-compatible embeddings, compose profiles local-models and litellm (inference providers)
  • Extraction Proposal items and evidence Review UI (an Items tab in Review accepts, edits or rejects items one by one or by filter; the Documents viewer highlights the evidence)
  • Embedding jobs and hybrid search: Kinds opt in with search.embedding_template; scheduled Scribe passes embed the Records that lack a vector of the configured model (keyed by content hash and template fingerprint, shared across branches and overlays with the same template, batched, rate-limited, resumable), lorebase-scribe embeddings backfills and reindexes (migrations 0014 and 0024, scribe); query vectors are computed on the caller's side (lorebase-inference embed, lorebase search --vector-file). The model is advertised in GET /schema and lorebase_schema; prefixes for E5/BGE style models are settings; garbage collection deletes the vectors no branch reads (orphaned content, templates no longer used)
  • Derivations: expression, aggregate and path (expression and aggregate are recomputed in the materialized head; path Derivations are virtual Relations composed at read time with a bounded fan-out, or stored and kept current incrementally when materialized, and are read by neighborhoods, Link counts, Operation links() and other Derivations; a change of the fan-out rewrites what follows it (migration 0030), see design 5.8)

M5: Enterprise hardening and 1.0

  • Overlay branch materialization (AC-16 implemented, migration 0010, store schema notes); landed early, copy stays the default mode outside the agent/, extract/ and source/ prefixes, and agent task branches are overlays (keeper.agent_writes.branch_mode). Overlays get their own per-Attribute indexes, and migrations 0011 (branch-scoped search index) and 0012 (overlay neighborhood lookups) keep overlay reads within 30% of a copy at p50 (performance)
  • PII vault storage and erasure (vault: true Attributes, migration 0019, lorebase admin erase, see design 13.5); landed early
  • Helm chart (kubernetes); landed early, validated by lint, template and kubeconform, and installed, upgraded and tested on a kind cluster with an external Postgres; the CloudNativePG mode has not been run
  • Garbage collection of unreachable objects (operations); landed early
  • Observability: a catalogue of every metric with explicit histogram buckets, Grafana dashboards for Keeper, Store and Scribe, Prometheus alert rules with runbooks and the OpenTelemetry Collector configuration, in compose and the Helm chart (metrics, operations); landed early
  • Performance targets of design 6.14 met at 10M Records on the reference hardware, with a published benchmark harness (the harness is published: make perf-10m loads 1M Records through the commit pipeline, replicates them to 10,001,100 Records and 30,327,660 Links and measures every read, the agent trace, commits and branch creation; performance. On a shared laptop with an 8 GiB Docker VM, get by key, pages, Links of a Record and attribute queries are at or near budget, branch creation stays at 42 to 147 ms; the 2-hop neighborhood (96 to 153 ms at p95, bound by pages read from disk), full-text search (59 to 111 ms, the GIN bitmap of frequent terms), history and lineage (17 to 28 ms) are over it, and a commit of 1,000 changes takes 3.3 s. Re-measured on 2026-10-23 with realistic Zipf text under heavier load: searches of very common words stay over budget whatever the ranking size, since the GIN bitmap of the word is built first, and values: false makes neighborhood responses a third of the size but not faster; performance)
  • Backup and restore guide, audit chain verification, signed images (Domain bundles and pg_dump in operations; python -m lorebase_keeper audit verify, GET .../audit/verify, lorebase audit verify and a daily Scribe job with an alert, operations; images signed with cosign by the release workflow, release)
  • HTTP JSON Source with cursor, page, offset and Link header pagination (scribe); landed early
  • LinkML schema import and export (lorebase schema export --format linkml, lorebase schema import FILE --from linkml; round trips through lorebase.* annotations, LinkML)
  • Documentation site at aantenore.github.io/lorebase, built with mkdocs build --strict (make docs) and published by the Docs workflow
  • API stability guarantees for /api/v1 (the policy is written, API stability: what is stable, compatible and breaking changes, deprecation, client compatibility, MCP tools and the schema format; it binds from 1.0, and Keeper does not emit Deprecation or Sunset headers yet since nothing is deprecated)
  • Threat model published and the security review checklist completed (threat model: STRIDE per trust boundary with mitigations and residual risks; security review: 23 findings, all fixed, none open)
  • Full proof scenario (days 1 to 5) as an automated end-to-end test: the five-day demo asserts every day against a real stack with the fake inference provider (a document extracted with the deterministic tiers, the agent over MCP writing through a Proposal), and the End-to-end workflow runs it on demand, before a release (demo)

Next: knowledge that stays correct

Designed in section 18 of the design; every item is implemented.

  • Corrections that stick (design 9.1, AC-34): per-Source last-written state (source_written); a run writes only Attributes that changed upstream; an upstream change over someone else's value is a DV1 with a preserved base, a snapshot deletion of a Record with others' values a DV2; Source setting local_edits: review | keep | source_wins (default review); seeding on first sight for existing Sources; audit source.local_edit_conflict (migration 0040, Scribe)
  • Extraction hold local_edit: value items over a value a person set are never accepted at once (design 9.1)
  • Incremental Sources: last_seen_run per Source-owned Record and stale candidates (source_unseen) after a Source's stale_after, never a deletion (migration 0043, Scribe)
  • Document versions and lifecycle (design 9.2, AC-35): optional source_key on upload, versions with replaces (one active version per key, its bytes unique per key), GET .../documents?source_key= and GET .../documents/{doc}/versions; the document_version_check job raising stale candidates (evidence_gone) when an old quote is gone; retirement (document.retire) and purge (document.purge, admin only); audit document.version, document.retire, document.purge (migration 0041, extraction)
  • Ingestion modes (design 9.2, AC-36): extraction.mode: review_all | doubts_only | direct per Domain and per source_key prefix, tightening always allowed, loosening per scribe.extraction.domain_may_loosen; direct only for all-tier-0 jobs with nothing pending, audited proposal.merge with mode: direct (extraction)
  • Stewards (design 10.7, AC-37): stewards on the Domain and per Kind, computed assignees, assigned=me|any|none on Proposal and reconciliation lists (REST, CLI --mine, web), protection rule required_approvals_from_stewards (migration 0042, operations)
  • Traits as search targets and context-pack seeds (traits), and lenses (lenses in the schema; lens on context packs, search and MCP) (design 8.8, AC-38)
  • Resolution of words to schema and values (design 8.9, AC-39): value index of selected short Attributes per branch kept by triggers (mat_values, migration 0045, Scribe value_index), aliases, value_aliases and resolvable in the schema, POST /resolve, lorebase_resolve, lorebase resolve
  • Consolidation from use (design 8.10, AC-40): resolution misses recorded where turned on (store.resolve_misses, migration 0046), consolidated by Scribe into value_alias, attribute_alias, kind_alias and optional attribute_description suggestions reviewed as schema change Proposals
  • Coverage signals (design 8.11, AC-41): coverage on POST /query and POST /resolve (missing values of the Attributes used, Source freshness, unknown phrases) from per-Kind statistics of the default branch head (mat_coverage, Scribe coverage)
  • Linking resolution misses to the next query of the same agent session (needs agent trace data that is opt-in today), coverage for context packs and search
  • Filtered vector search that keeps its limit: hnsw.iterative_scan = relaxed_order on pgvector 0.8 or later (search.semantic.iterative_scan, max_scan_tuples), over-fetch otherwise (overfetch_factor), semantic_truncated when the scan bound is reached (design 8.8)

Not on the roadmap

These are written down as non-goals or anti-goals in the design: a model in the agent read path, direct model writes to protected branches, schemaless writes, a second required datastore, an agent runtime, an ETL platform, BI dashboards, OWL reasoning, multi-region active-active writes, environment and release management (versions of systems are Records, tags or validity periods; release tooling stays the source), key aliases after duplicate merges, and live federated reads across Sources or Domains.

How to influence it

Open a discussion with the domain you would model and what your agents need to read or change. Real scenarios move items up. Design changes are proposed as issues that cite the section of the design they challenge; see CONTRIBUTING.md.