Skip to content

Scribe

Scribe is the Lorebase worker. It runs the jobs queued in Postgres and schedules the periodic ones: Source synchronization, document extraction (extraction.md), webhook delivery, queued index builds, garbage collection, embeddings (below) and the detection of contradictions between Sources (below). Design references: 5.9 (Source), 9.1 (structured Sources), 6.6 (commit pipeline), AC-18 (no external broker).

Scribe writes through the same commit pipeline as Keeper, under a service principal (scribe.worker.principal, default svc_scribe): every change it makes has a Revision, Lineage and an audit entry.

Running it

lorebase-scribe                                   # work and schedule until SIGTERM or SIGINT
lorebase-scribe once [--limit N]                  # run the due jobs, then exit
lorebase-scribe sync supply approved.suppliers    # synchronize one Source now, print the report
lorebase-scribe sync supply approved.suppliers --dry-run
lorebase-scribe embeddings supply                 # embed what lacks a vector now (backfill)
lorebase-scribe embeddings supply --reindex --prune
lorebase-scribe modes clear                       # forget the remembered structured output modes

Configuration comes from the same layered settings as Keeper (LOREBASE_CONFIG, then LOREBASE__SCRIBE__... variables); every key with its default is in config/lorebase.example.yaml under scribe. Retries and stale locks use store.jobs, webhook delivery uses webhooks, collections use store.gc.

In docker compose the scribe service runs the backend image with lorebase-scribe. It mounts examples/supply/data read-only at /data/sources/supply and points the example CSV Source at it, so POST /api/v1/domains/supply/sources/approved.suppliers/sync (or lorebase sources sync approved.suppliers -d supply, then lorebase sources runs approved.suppliers -d supply) works once the supply Domain exists (see the quickstart). The runs listing needs a Domain-wide data.read grant.

Job types

Type Queued by Does
source_sync Keeper (POST .../sources/{s}/sync), the scheduler (Source schedule) Synchronizes one Source (below). Payload {"source": id, "dry_run": bool}
webhook_delivery the scheduler, every webhook_interval_s One pass of lorebase_store.webhooks.deliver_pending: fan-out of outbox events, signed POSTs, retries with backoff
store.index commits whose per-Attribute index plan changed (large Kinds, or roles that do not own mat_records) Builds or drops one per-Attribute index (CONCURRENTLY), as database.maintenance_dsn when it is set (store schema notes)
store.materialize creation of a copy branch of a large source (store.materialize.async_copy_min_rows); commits that move a base branch past an overlay's overlay_max_base_lag_revisions; the deletion of a large base Copies the base rows of the overlay branch in batches, then makes it a full copy; resumes from its cursor (store schema notes). Payload {"branch": name}
store.branch_delete deletion of a large branch (store.materialize.async_delete_min_rows) Deletes the rows of the hidden branch in batches, then its ref; waits for the copies of overlays based on it (store schema notes). Payload {"branch": name}
gc the scheduler, at gc_schedule per Domain; python -m lorebase_store.gc --enqueue Garbage collection of a Domain (operations)
extraction {"documents": [ids], "tiers": [..] or null, "requested_by"} Documents into a Proposal of items (extraction.md)
embeddings the scheduler, every embeddings_interval_s per Domain; a job that used its row budget (continuation) One embedding pass over the Domain's materialized branches (below). Payload {"branches"?, "reindex"?, "resume"?}; the scheduled one is {}
reconcile_sweep the scheduler, every scribe.reconciliation.sweep_interval_s per Domain (0, the default, disables it); a job that used its work budget (continuation) One reconciliation pass over a branch (below). Payload {"branch"?, "resume"?}; the scheduled one is {} (the default branch)
document_version_check an extraction job that read a new version of a keyed document, once it finished; Keeper at a keyed upload that does not extract, and at a retirement; a job that used its value budget (continuation) The evidence check of a new document version or a retired document (below, extraction). Payload {"document", "mode": "version" or "retire", "resume"?}
audit_verify the scheduler, every audit_verify_interval_s Recomputes every audit hash chain; the result names the first broken entry, and lorebase.audit.verifications counts the outcome (operations)

Worker loop

Each process runs scribe.worker.concurrency slots. A slot claims one due job with SELECT ... FOR UPDATE SKIP LOCKED, so replicas never run the same job, runs it outside any transaction and records the outcome. A failure is retried after store.jobs.retry_delay_seconds until store.jobs.max_attempts; failures that a retry cannot fix (an unknown Source, an invalid mapping, rows the branch rejects, more failed rows than allowed) fail the job at once.

Jobs run at least once, so every handler tolerates running the same job again. Running jobs refresh their lock every heartbeat_interval_s (30 s). A job without heartbeat for store.jobs.stale_after_seconds (90 s, three missed heartbeats) belongs to a worker that is gone (killed, out of memory, cut off from the database): the scheduler leader reclaims it at its next tick, queued again, or failed when it has used store.jobs.max_attempts (a job that keeps killing its worker is not run forever). A worker that lost a job this way cannot record its outcome over the next run. lorebase.jobs.reclaimed counts reclaimed jobs by type and outcome.

An outcome the store refuses to take (a dropped connection, a statement timeout) is written once more after outcome_retry_delay_s (1 s). When that fails too the run counts unrecorded in lorebase.scribe.jobs, the slot goes on claiming, and the job, no longer heartbeated, is reclaimed like a lost one. A slot whose task ends with an error is logged and started again after poll_interval_s, so a process never runs with fewer slots than configured for long; lorebase.scribe.worker.slots reports the slots live and lost.

Scribe serves no HTTP, so its liveness is a file. With liveness_file set, the loop rewrites it every liveness_interval_s (10 s) with its live slots, the jobs running and the time it last reached the store, and removes it when it stops. python -m lorebase_scribe.liveness FILE [--max-age SECONDS] [--min-slots N] [--max-store-age SECONDS] reads it (standard library only: one interpreter start) and exits 0, or 1 with the reason: no file, a file older than three intervals (the process hangs or is gone), fewer live slots than --min-slots (1) or, with --max-store-age, a store not reached within that long. The compose scribe service sets LOREBASE__SCRIBE__WORKER__LIVENESS_FILE=/tmp/scribe-liveness.json and runs that command as its healthcheck; the scribe values of the chart show the same as an exec livenessProbe.

On SIGTERM (Scribe runs as PID 1 of its container and handles the signal itself) the slots stop claiming and running jobs get shutdown_grace_s to finish. The rest are cancelled and go back to the queue without counting the attempt; a job whose handler has not stopped shutdown_cancel_s later is released anyway, so a worker that stops cleanly leaves no job running (lorebase.jobs.released). Keep shutdown_grace_s plus shutdown_cancel_s below the time the platform waits before it kills the process (SCRIBE_STOP_GRACE in compose, scribe.terminationGracePeriodSeconds in the chart).

Spans (scribe.job <type>, scribe.source_sync, scribe.scheduler.tick) and metrics (lorebase.scribe.jobs, lorebase.scribe.job.duration, lorebase.scribe.worker.slots, lorebase.scribe.scheduled, lorebase.scribe.source.rows, lorebase.scribe.source.changes) are exported when telemetry.otlp_endpoint is set. Logs are structured JSON with the job, type, Domain, status and duration.

Scheduler and leader election

Every process may run the scheduler; the one holding a Postgres advisory lock (key derived from scheduler.lock_name, held on a dedicated connection) is the leader, so several replicas never schedule twice. If the leader dies, its connection and lock go with it and another process takes over at its next tick (tick_s). The leader requeues stale jobs and queues:

  • a webhook_delivery job every webhook_interval_s (an identical queued job is reused, so a backlog never piles up);
  • a gc job per Domain at each firing of gc_schedule (cron, UTC);
  • a source_sync job for each Source whose schedule fired, read from the schema of the Domain's default branch;
  • an embeddings job per Domain every embeddings_interval_s (0 disables; an identical queued job is reused);
  • a reconcile_sweep job per Domain every scribe.reconciliation.sweep_interval_s (0, the default, disables it; an identical queued job is reused);
  • an audit_verify job every audit_verify_interval_s (one day; 0 disables; an identical queued job is reused). The interval is kept in memory, so a process that becomes leader queues one at its first tick.

Cron jobs are queued with run_after set to the firing time; a firing is due when it is later than the newest such job or, without one, than the moment the process became leader. Missed firings are not replayed one by one: a single job covers them.

Sources

A Source (design 5.9) maps an external table into one Kind. It lives in the schema:

sources:
  approved.suppliers:
    id: src_01KEZF7C00EDDEFRGSAXSHCWJ2
    type: csv
    connection: secret://env/SUPPLY_APPROVED_SUPPLIERS_CSV   # a path or an https URL
    target_kind: Supplier
    key_mapping: [supplier_code]
    attribute_mapping:
      legal_name: {column: legal_name, transforms: [{name: trim}]}
      country: {column: country, transforms: [{name: trim}, {name: upper}]}
      risk_score: {column: risk_score, transforms: [{name: cast, args: {to: integer}}]}
      onboarded_on: {column: onboarded_on, transforms: [{name: parse_date, args: {format: '%Y-%m-%d'}}]}
    sync_mode: snapshot
    target_branch_policy: {branch_template: 'source/{source}', merge_policy: auto_when_clean}
    local_edits: review                   # the default; keep or source_wins (below)

  erp.purchase_orders:
    id: src_01KEZF7C00PMKRJJ98PAYYB4SX
    type: postgres_table                  # or postgres_query with `query: SELECT ...`
    connection: secret://env/ERP_DSN      # DSN of a read-only role
    table: erp.purchase_orders            # optional; default: last segment of the api name
    target_kind: PurchaseOrder
    key_mapping: [po_no]
    attribute_mapping:
      supplier: {column: vendor_id}
      status: {column: state, transforms: [{name: lower}]}
    relation_mapping:
      - {relation: for_part, columns: [part_no]}
      - {relation: delivered_to, columns: [plant_code]}
    scope_filter: "state <> 'archived'"
    sync_mode: incremental
    watermark_column: updated_at
    tombstone_column: deleted_at
    schedule: '0 2 * * *'
    target_branch_policy: {merge_policy: auto_with_authority}

MySQL, MariaDB and SQL Server Sources take the same fields as Postgres ones (table, query, scope_filter, watermark_column, tombstone_column):

sources:
  erp.shipments:
    id: src_01KEZF7C00SHIPMENTSMYSQL01
    type: mysql_table                     # mysql_query with `query:`; MariaDB works too
    connection: secret://env/ERP_MYSQL_DSN     # mysql://reader:...@db:3306/erp
    table: shipments
    target_kind: Shipment
    key_mapping: [shipment_no]
    sync_mode: incremental
    watermark_column: updated_at
    target_branch_policy: {merge_policy: auto_when_clean}

  finance.invoices:
    id: src_01KEZF7C00INVOICESMSSQL001
    type: mssql_query                     # mssql_table reads `table: dbo.invoices`
    connection: secret://env/FINANCE_MSSQL_DSN # mssql://reader:...@db:1433/finance
    query: SELECT invoice_no, supplier_code, total, modified_at FROM dbo.invoices
    target_kind: Invoice
    key_mapping: [invoice_no]
    sync_mode: incremental
    watermark_column: modified_at
    target_branch_policy: {merge_policy: proposal}

JSON documents, JSON Lines and Parquet files are local files, like local CSV files:

sources:
  plm.parts:
    type: json
    connection: secret://env/PLM_PARTS_JSON     # a path, e.g. /data/sources/plm/parts.json
    records_path: data.parts                    # the array of records; omit when the document is one
    flatten_separator: '.'                      # optional: {"spec": {"weight": 2}} -> spec.weight
    target_kind: Part
    key_mapping: [part_no]
    attribute_mapping:
      name: {column: name}
      weight_kg: {column: spec.weight}
      dimensions: {column: dimensions}          # a nested object, kept as a json value

  mes.events:
    type: jsonl                                 # one object per line, streamed
    connection: secret://env/MES_EVENTS_JSONL
    target_kind: ProductionEvent
    key_mapping: [event_id]
    attribute_mapping:
      status: {column: status}
    sync_mode: incremental
    watermark_column: recorded_at

  finance.invoices:
    type: parquet                               # needs lorebase-scribe[parquet]
    connection: secret://env/FINANCE_INVOICES_PARQUET
    flatten_separator: '_'                      # struct columns: billing_country, billing_city
    target_kind: Invoice
    key_mapping: [invoice_no]
    attribute_mapping:
      amount: {column: amount}
      country: {column: billing_country}

An http_json Source reads a JSON API with GET requests, page by page:

sources:
  crm.accounts:
    type: http_json
    connection: secret://env/CRM_ACCOUNTS_URL   # https://crm.example.com/api/v2/accounts
    records_path: data                          # omit when each response is the array
    target_kind: Account
    key_mapping: [id]
    attribute_mapping:
      name: {column: name}
      owner: {column: owner.email}
    flatten_separator: '.'
    http:
      headers:
        Authorization: secret://env/CRM_TOKEN   # credential headers take secret references only
        X-Tenant: acme                          # other headers may be literal
      pagination:
        type: cursor                            # none | cursor | page | offset | link
        cursor_path: meta.next_cursor           # cursor: where the next cursor is in a response
        cursor_param: cursor                    # cursor: query parameter that carries it
        # page:   page_param (default page), start_page (default 1)
        # offset: offset_param (default offset), page_size (required)
        # link:   follows the rel="next" URL of the Link header
        size_param: limit                       # optional, with page_size: asks for the page size
        page_size: 200
      since_param: updated_since                # incremental runs send the previous watermark
      max_pages: 500                            # optional, below scribe.sources.http_json.max_pages
    sync_mode: incremental
    watermark_column: updated_at

Connections are secret references, never literals. Scribe resolves them in its own environment: for CSV the value is a local path or an http(s) URL, for JSON, JSON Lines and Parquet a local path, for http_json the URL of the first page, for Postgres a DSN, for MySQL and SQL Server a URL (mysql://user:password@host:3306/database, mssql://user:password@host:1433/database). Only the references listed in scribe.sources.allowed_connections are resolved (empty by default), and so are secret header values of http_json Sources: Sources are written by schema authors, and without the list any of them could name a reference meant for something else, such as the DSN of the Lorebase database. Errors name the reference, never the value behind it.

  • CSV files must lie under scribe.sources.csv.allowed_roots; URLs need a scheme in allowed_schemes and a host in allowed_hosts, are fetched without following redirects and stop at max_bytes. Both lists are empty by default, which refuses every CSV location. The delimiter is sniffed, the first encoding of encodings that decodes the file wins and the first row is the header.
  • A local path is resolved, links included, before the roots check, so links inside a root that point inside a root still work. The file is then opened from the root one component at a time without following links: a component replaced by a link after the check is refused. It must be a regular file (not a directory, FIFO or device) and is read from that descriptor, at most max_bytes.
  • A URL host must resolve to global unicast addresses only, checked as for webhook targets; the download connects to those addresses, never to a second lookup, and timeout_s bounds it from lookup to last byte. Hosts on private networks need allow_private_targets: true (off by default; earlier versions did not check addresses).
  • JSON and JSON Lines files must lie under scribe.sources.json_files.allowed_roots, Parquet files under scribe.sources.parquet.allowed_roots (both empty by default), and are opened with the same link and file type checks as CSV files. A json document is parsed whole, at most json_files.max_bytes; its records are the array at records_path (dotted keys) or the document itself. A jsonl file is streamed line by line (blank lines skipped, lines at most max_line_bytes). A parquet file is streamed in batches of parquet.batch_rows rows and needs pyarrow: install lorebase-scribe[parquet] (the run fails with a message naming the extra otherwise). Locators are file.json:<record>, file.jsonl:<line> and file.parquet:<row>.
  • http_json URLs, next pages included, follow the CSV URL rules with their own settings (scribe.sources.http_json): a scheme in allowed_schemes, a host in allowed_hosts, global unicast addresses only unless allow_private_targets, looked up once per host and run and pinned, no redirects, at most max_bytes per response and timeout_s per request. 429 and 5xx answers (retry_statuses), network errors and timeouts are retried up to max_attempts times with exponential backoff (backoff_base_s, backoff_factor, at most backoff_max_s, a longer Retry-After honored up to that cap); other statuses fail the run. Reading stops at a page without records or without a next page; a Source that needs more than max_pages pages (or its own lower http.max_pages) fails instead of committing part of itself, since a partial snapshot would remove the Records it did not see. Headers whose name suggests a credential (Authorization, Cookie, names containing key, token, secret, auth...) must be secret references, checked when the schema is written, so a credential never lands in the schema or its history; Host and framing headers cannot be set. Locators are <last URL path segment>:<record number>.
  • Records keep their JSON (or Arrow) types: nested objects and arrays are JSON values, which a json Attribute stores as they are and textual Attributes receive as canonical JSON text. flatten_separator turns nested objects into columns named parent<separator>child instead (arrays stay values). Records have no header: a key missing from a record is null there, and a mapped column that no record of the run has fails the run.
  • Postgres Sources are read in one READ ONLY transaction with scribe.sources.postgres.statement_timeout_ms, fetch_rows rows per round trip. The table name is checked as an identifier and quoted; scope_filter and query are SQL written by the schema's modelers and reviewed with the schema, so give the DSN a role that can only read what the Source needs.
  • MySQL, MariaDB and SQL Server Sources (mysql_table, mysql_query, mssql_table, mssql_query) are read the same way, fetched fetch_rows rows at a time, with their own settings (scribe.sources.mysql, scribe.sources.mssql: statement_timeout_ms, connect_timeout_s, fetch_rows). Their drivers are optional extras and the run fails with a message naming the extra when it is missing: lorebase-scribe[mysql] (asyncmy, async, wheels for the usual platforms) and lorebase-scribe[mssql] (pymssql, run in worker threads; FreeTDS is bundled, so no ODBC driver is needed). MySQL and MariaDB sessions are READ ONLY and bounded by max_execution_time (MariaDB: max_statement_time). SQL Server has no read-only transaction: the login must be able to read only. query is wrapped as SELECT * FROM (<query>) AS lorebase_source; SQL Server refuses an ORDER BY in a derived table unless it has TOP or OFFSET, so a query whose top level ends in ORDER BY gets OFFSET 0 ROWS appended (the rows and their order are unchanged). datetime2 watermarks keep their microseconds and a rowversion watermark is binary: it is stored in the run as hexadecimal text and bound back as bytes. Locators are <table>:<row> or query:<row>.

Mapping. Each mapped Attribute reads one column through a list of transforms: trim, lower, upper, null_if (value or values), constant (value), concat (columns, separator), map (values, optional default), parse_date (format, strptime syntax) and cast (to: string, integer, decimal, float, boolean, date, timestamp). The result is coerced to the Attribute type through the core canonical encoding: text is cast with the core cast rules ('true' to a boolean, '12' to an integer), native Postgres values are encoded directly, empty text is an absent value and timestamps without a zone are read as UTC. The key comes from key_mapping, one column per primary key Attribute, and sets those Attributes. Each Relation mapping names the columns holding the primary key of the to Record; empty columns mean no Link. A row whose values do not fit is skipped and reported; more than max_row_errors such rows fail the run without committing anything. An Attribute mapped by expression instead of column starts from a CEL expression over the row: each column whose name is an identifier is a variable of unknown type (dyn, so CSV text needs int(price)), and row["Legal name"] reads any column. Variables are the columns of the Source (the header, or every key of the JSON records read, whose rows are then mapped once all are read): a column a row lacks is null. The expression is bounded by schema.expression_limits; its result then goes through the transforms and the coercion like a column's, and the Modeler previews it (POST /domains/{d}/sources:preview).

The Source owns its mapped Attributes and its mapped Relations. Other Attributes of an existing Record are kept, and Links of other Relations are left alone.

A synchronization run

  1. Scribe resolves the Source (by id or api name) in the schema of the Domain's default branch, the target branch, and records the run in source_runs.
  2. It reads the rows (incremental Sources only those whose watermark column is greater than the highest watermark of the last successful run) and maps them. Postgres, MySQL and SQL Server compare watermarks in SQL with the column's type and CSV as text (use ISO timestamps or zero-padded numbers); JSON, JSON Lines, Parquet and http_json Sources compare natively (numbers, Parquet timestamps) or as text when the types differ, so string watermarks should be ISO timestamps too.
  3. It prepares the Source branch, source/<api name> by default: created from the target branch the first time, and recreated at the target head when no Proposal from it is open and the heads differ, so schema and unrelated target changes are always current.
  4. It computes the delta against the branch head, writing only what changed upstream since the Source last wrote it (below): added keys, changed keys (the content hash of the new Record differs from the stored one; Lineage is not content, so rewriting the same values is no change) and removed keys. Snapshot Sources remove the Records they last wrote (before any last-written state: whose Lineage names this Source) that the run did not see, unless a row failed; incremental Sources remove the Records of rows whose tombstone column is set.
  5. It commits the delta on the Source branch in Revisions of at most commit_batch changes (the changes of one row stay together), with origin source_sync and one provenance entry per changed row: Source id, run id and row locator (file.csv:17, orders:42).
  6. It raises the reconciliation candidates of the run's changes against the target head (below), linked to the Proposal, then applies the merge policy of target_branch_policy to the Source's Proposal into the target branch, opened on the first changes and updated by later runs (its title carries the counts of the last run):
Policy Effect
proposal The Proposal stays open with its merge preview, for review in Keeper or the web app
auto_when_clean Merged at once when no Conflict remains; otherwise left open with its Conflicts
auto_with_authority As auto_when_clean, and value Conflicts on Attributes whose authority is this Source are resolved in the Source's favor

Every Source merge also applies the Domain's merge_policy setting, so its authority ranking (for example ["source:<erp source id>", "principal:usr_ops"]) and newest_wins decide the remaining value Conflicts from the Lineage of both branches (design 6.10).

Automatic merges into a protected target happen only with auto_merge_protected (default true): the policy is part of the reviewed schema, and the setting is the operator's trust in automation, so it waives the plain required_approvals count. It does not waive the rules that name who must approve: when steward approvals (required_approvals_from_stewards above 0) or required roles (required_roles not empty) apply to the target, from keeper.proposals or the Domain setting of the same name, the merge is not performed, the Proposal stays open for review, and the run's stats say why (merge_blocked: approvals, with the rules in blocking_rules). With auto_merge_protected off, those Proposals stay open whatever the rules. Unprotected targets are not affected. A run with no changes leaves an open Proposal untouched; with an automatic policy it retries its merge.

A dry run (dry_run: true) reads, maps and computes the delta against the state step 3 would produce, records the counts in the run (local_edits and seeded included) and writes nothing else.

Corrections that stick

A Source changes only what changed upstream (design 9.1, AC-34). For each Source and target branch Lorebase keeps the last-written state (source_written): per Record (and per Record's Links of each mapped Relation) the hash of each value the Source last wrote, never the value, and the Source branch Revision that holds it. Each run compares the fresh value F with the last-written one W, not with the current one C:

Case Action
F = W (no upstream change) nothing, whatever C is: a person's correction, an Operation or another Source's value stays
F != W, C = F nothing; the state advances
F != W, C set by this Source F is written
F != W, C set by anyone else a local edit: decided by local_edits

local_edits on the Source:

  • review (default): F is written on the Source branch and the Proposal's merge attempt holds a DV1 Conflict whose base is W (ours: the correction, theirs: the new upstream value), resolved like any Conflict, by a person or by the Domain's merge policies. An auto_when_clean Source therefore leaves such a run open.
  • keep: the same Conflict, resolved at once with resolver policy:local_edits by the ranking [principal:*, operation, merge_resolution, reconciliation, source:{id}]: a value set by an Operation or a decision stays; one set by another Source, an extraction or an import is overwritten; an unknown origin stays a Conflict.
  • source_wins: the behavior before this change, F overwrites C.

Under auto_with_authority the Attributes whose authority is the Source are always overwritten, as before. A snapshot deletion of a Record holding values someone else set is a DV2 delete/modify Conflict (keep: the Record stays, with its Links; source_wins: deleted, with its Links). A Record someone else deleted and upstream changed comes back, when the DV2 takes upstream's side, with every Link upstream holds. While a keep Source's Proposal stays open, each run decides again the Conflicts a person reopened by editing a kept value again. Links follow the same rule without Conflicts: a Link someone removed is not added again until upstream changes it, and a Link someone added is not removed. Each local edit writes one source.local_edit_conflict audit entry (Source, run, Proposal, Kind, key, Attribute, origin type and actor of the current value, mode and outcome conflict, kept, overwritten or deleted, never values), and the run's stats count them in local_edits.

The state advances only when the Source's Proposal merges, in the merge transaction, whoever merges it: a Proposal closed unmerged advances nothing, and its upstream changes come back with the next run. A run with nothing to merge and no open Proposal records it at once. Sources synchronized before this change have no state: the first run seeds it (seeded in the stats). A value whose Lineage is this Source counts as last written; any other value is compare-only for that run, so an equal one seeds the state and a different one is a local edit.

Incremental Sources and upstream deletions

An incremental Source reads only the rows past its watermark, so it cannot tell a row removed upstream (without a tombstone) from one that did not change. Each run of an incremental Source records, per Record it read, the run and the time (source_written.last_seen_run and last_seen_at, migration 0043), changed or not; a Record whose state advances through a merge counts as seen by the run that staged it. Set stale_after on the Source (ISO 8601 duration, days to seconds; default null: off; incremental Sources only) to have the reconciliation sweep raise a stale candidate with signal source_unseen for each Record the Source last wrote to the branch, not deleted and still there, that no run has read for longer than that (before the first run after the upgrade, the time of the Record's last state advance stands in):

sources:
  erp.purchase_orders:
    sync_mode: incremental
    watermark_column: updated_at
    stale_after: P30D             # a Record no run read for 30 days is raised for review

stale_after suits feeds that deliver every live row again within that window (a query that re-reads the rows still current, a periodic full extract past the watermark). A feed that sends only the rows that changed never re-delivers a Record that stays unchanged upstream, so every such Record is raised once stale_after has passed although it still exists: false positives. Leave stale_after null for those feeds.

Nothing is deleted on that signal. The candidate names the Source, the last run that saw the Record and when, and no Attribute: a reviewer keeps it (keep_current), dismisses it, or deletes the Record through an Operation. Its fingerprint holds the Source and that run, so a dismissal holds while the Record stays unseen, and a Record seen again and later lost again is raised anew. The sweep marks it obsolete when a run sees the Record again (source_seen), the Record is deleted (record_deleted) or the check no longer applies to the Source (source_unseen_off: stale_after removed, the Source no longer incremental or gone, or stale.source_unseen: false). The operator or a Domain turns the check off with scribe.reconciliation.stale.source_unseen: false. The sweep pages these Records in key order within its work budget, like the others.

Reconciliation

Scribe detects contradictions between Sources (design 10.6) and stores them as reconciliation candidates for review; it never changes a value because of one. Settings are scribe.reconciliation (configuration). A Domain overrides some of them in its reconciliation setting: enabled, triggers, hold_auto_accept, stale.max_age, stale.source_unseen, normalize.numeric_tolerance and negation_words, duplicates.trigram_threshold, stages (to leave stages out, never to add one) and, when the operator turned the semantic stage on, the members of semantic except judge. Files, work limits and model spend (normalize.units_file, sweep_*, max_claims_per_candidate, duplicates.max_pairs_per_kind, semantic.judge) are the operator's only; a member that is not allowed or not valid is logged and ignored, the rest of the setting still applies.

The merges Scribe runs (Source syncs, extraction) apply the Proposal check reconciliation with the stricter of scribe.reconciliation.merge_check (off, warn by default, block) and the Domain setting reconciliation_check: in block, open contradictions hold the merge back and the Proposal stays open for review (design 10.6).

  • Extraction. Value items whose value differs from the target's are compared with the claim of the current value (its provenance) while the items are assembled. With hold_auto_accept an item that raises a candidate (contradiction or superseded) stays pending whatever its confidence (held_by_reconciliation in the job stats). Mentions left ambiguous between two Records raise a duplicate. Tier 2 asks for an optional period (valid) per value of a valid_time Attribute, kept only when its dates are written in the value's quote.
  • Source syncs. After the commit, each value the run wrote is compared with the target's: different source identities raise a contradiction (or superseded, periods apart); the same Source with a newer value waiting in a proposal Proposal raises stale. Added Records are checked for duplicates. The run's counts carry reconciliation_raised and reconciliation_claim_added.
  • Document versions (document_version_check). For a new version of a keyed document, every value whose current provenance is extraction from an earlier version of the key (found through the accepted items of those versions, on the branch their Proposals targeted) and that the new version's own items do not propose again: its old evidence quote is looked for in the new version's parsed text with the white-space-insensitive match of evidence verification. Gone, or unreadable (purged), raises stale with signal evidence_gone ({document, from_version, to_version, quote}); found changes nothing. A retirement runs the same check with empty text. No model is called and nothing is retracted. It looks at scribe.extraction.versions.max_values_per_job values, then queues its continuation; a Domain turns it off with triggers.document_versions: false.
  • Sweep (reconcile_sweep). Marks obsolete, with the reason, the open candidates whose disagreement left the data (Record deleted, Proposal closed or item rejected, claims equal under the current options, stale value replaced); then walks the Records of each Kind for values in the branch history that replaced a value of another source identity, values older than stale.max_age or whose period ended, and duplicates (equal folded unique or title values, similar titles, and stored Record vectors with the semantic stage); last, the Records incremental Sources have not seen (below). It stops after sweep_max_changes_per_job units of work and queues its continuation.

Values of derived and vault Attributes are not compared, and claims whose source identity cannot be told (a merge resolution, a schema migration) are left out. The values written by a plain commit count as their import batch's (import:{ref}) or their author's (principal:{id}).

Semantic stage (optional, off). It runs when semantic.enabled is set, semantic is in stages and an embedding model is configured (inference.embeddings); otherwise it is off, logged once. It only adds signals and filters, and never decides a candidate. Values of vault Attributes and of Attributes with any effective sensitivity tag (their own, the default branch's, or carried through Derivations, as extraction reads them) are never sent to a model.

  • competing values of text Attributes are embedded; the least similarity of two of them is the similarity signal; at semantic.equivalent_similarity (0.9) or more the candidate is marked equivalent (near-paraphrases, ranked last); at semantic.equivalence_threshold or more (null by default: always raised) it is not raised and counted filtered;
  • the sweep pairs Records of Kinds with an embedding_template whose stored vectors (from the embeddings job) are at least semantic.duplicate_threshold similar, without any model call: a duplicate with the semantic signal, counted in duplicates.max_pairs_per_kind;
  • with semantic.judge the chat model of task reconciliation is asked whether the two least similar values contradict: a judge signal (label, score, model). Answers go through the extraction cache (scribe.extraction.cache) and a job stops asking past scribe.extraction.max_tokens_per_job tokens.

A model that fails or is missing gives no signal; detection goes on.

Validity periods. A Source mapping may give a valid_time Attribute the period of its value, from a column or an expression for either end; dates are their first instant in UTC:

attribute_mapping:
  city:
    column: city
    valid_from: {column: moved_on}
    valid_to: {expression: 'row.left_on'}

The period goes on the provenance entry of the value (one entry per value with a period), never into content: a run repeating the current value with another period changes nothing.

Embeddings

Semantic and hybrid search rank stored vectors against a query vector the caller computes (design 8.4); Scribe computes the stored ones (design 9.3) with the model of inference.embeddings (inference providers).

A Kind opts in with an embedding template in the schema; nothing is embedded otherwise, and a Domain without templates never creates the model:

kinds:
  Supplier:
    search:
      embedding_template: "{legal_name}, {city} ({country}). Tier {tier}."
      # embedding_sensitive: true   # needed to name a pii/confidential/restricted Attribute

Placeholders are Attribute api names, plus {_kind} (the Kind's display name) and {_key}; {{ and }} are literal braces. Derived Attributes cannot be named. Schema validation reports schema.embedding_template_invalid for a template it cannot use.

Each embeddings job runs one pass:

  • branches with no template, and branches whose head has not moved since the last complete pass for the model, are skipped;
  • otherwise the branch's own rows that lack a vector of the model under their Kind's template are rendered and embedded, plus every row of the Kinds whose template changed (the template fingerprint is recorded per branch in embedding_cursors);
  • a text whose hash equals the stored one is left alone; a text already embedded for another Record content or another template is copied, not sent to the model;
  • vectors are keyed by content hash and template fingerprint, so overlay and copied branches share the vectors of their unchanged Records and only add work for the Records they change, while a branch whose template differs keeps vectors of its own and Keeper searches each branch with its own template's vectors;
  • the templates of a schema are resolved once per schema hash and kept for the last template_cache_size schemas, so branches sharing a schema do not read and parse it again.

Garbage collection (operations) deletes the vectors no branch reads any more: those whose content no branch holds, and those of templates no branch uses for that content.

Setting (scribe.embeddings) Default Meaning
page_rows 500 rows read and rendered per query
batch_texts 64 texts per model call
max_records_per_job 20000 rows one job reads; the rest goes to a queued continuation
max_calls_per_second 0 pacing of model calls (a paid endpoint's rate limit); 0: none
max_text_chars 8000 rendered texts are cut to this length
template_cache_size 64 schemas whose resolved templates a process keeps, by schema hash
retry.attempts, backoff_s, backoff_max_s 2, 2.0, 60.0 retries of a failed call inside the job

store.embeddings.dims must equal inference.embeddings.dims: the job checks the model's dimension and embeds a probe text before its first call, and fails without retries on a mismatch. One pass per Domain runs at a time.

Switching model. Set the new inference.embeddings and run lorebase-scribe embeddings DOMAIN (or let the schedule do it): vectors are stored under the new model name next to the old ones. Then point Keeper's search.semantic.model at the new name (fake:<model> for the fake provider) and run lorebase-scribe embeddings DOMAIN --prune to delete the old vectors. --reindex renders every text again and re-embeds those that changed.

Query vectors. Keeper and Herald never call a model. Compute the query vector with the same settings: lorebase-inference embed --query "late deliveries" | lorebase search "late deliveries" --vector-file - (hybrid), or create_embedding_model(settings).embed([text]) in Python.

Operations

  • Trigger a run: POST /api/v1/domains/{d}/sources/{s}/sync with an optional {"dry_run": true} (permission source.sync: editor, maintainer, admin; audited as source.sync; a Source that maps vault Attributes also needs pii.import Domain-wide, as do schema changes and merges that add or change such a Source, design 13.5). The answer is 202 with the job id; a second request while the job is pending returns the same job.
  • Follow runs: GET /api/v1/domains/{d}/sources/{s}/runs (Domain-wide data.read), newest first, with status, counts, rows read and failed, row error samples, Revision, Proposal, merge status and duration.
  • Scale: run more Scribe processes (or raise concurrency); claiming and scheduling are safe with replicas. Restrict a process to some job types with scribe.worker.types.
  • Inspect the queue: SELECT type, status, attempts, run_after, result FROM jobs ORDER BY id DESC. Failed jobs keep their last error in result.