Storage protocols

Storage is defined by protocols — interfaces every backend implements. The memory, elastic, and neo4j adapters all satisfy these, which is why the entire pipeline runs unchanged regardless of where data lives. See Architecture for the two-stores design and Configuration for selecting backends.

.. py:module:: foodscholar.storage.protocols

.. py:class:: ChunkStore(*args, **kwargs)

module:

foodscholar.storage.protocols

Bases: :py:class:~typing.Protocol

.. py:method:: ChunkStore.init()

module:

foodscholar.storage.protocols

Provision the underlying store (index, schema, etc.). Idempotent.

Local stores (e.g. InMemoryChunkStore) implement this as a no-op so that fs.init() works the same regardless of backend.

.. py:method:: ChunkStore.upsert(chunks)

module:

foodscholar.storage.protocols

.. py:method:: ChunkStore.get(chunk_id)

module:

foodscholar.storage.protocols

.. py:method:: ChunkStore.get_many(chunk_ids)

module:

foodscholar.storage.protocols

.. py:method:: ChunkStore.search(query, theme_ids=None, shelf_ids=None, k=10, use_vector=True, use_bm25=True)

module:

foodscholar.storage.protocols

.. py:method:: ChunkStore.update_attachments(chunk_id, shelf_ids, theme_ids)

module:

foodscholar.storage.protocols

.. py:method:: ChunkStore.bulk_update_attachments(items, *, wait_for_refresh=False)

module:

foodscholar.storage.protocols

Patch shelf_ids + theme_ids on many chunks in one round-trip.

Used by fs.attach() so the chunk-side denormalization doesn’t pay one ES _update per chunk. wait_for_refresh=True blocks until the new values are searchable — set on the last flush only so subsequent queries in the same cell see the result. Remote backends collapse to a single bulk call per invocation; in-memory loops.

.. py:method:: ChunkStore.clear_attachments()

module:

foodscholar.storage.protocols

Reset all chunk-side shelf/theme denormalization.

fs.attach() calls this at the start so a re-run produces honest shelf_ids even when the projection changed (a chunk that previously attached to a now-pruned shelf no longer carries its id). Theme attachments are wiped too — they’re written by Layer B, which always rebuilds them. Per-chunk content (text, mentions, entity_links, embedding) is untouched.

.. py:method:: ChunkStore.update_annotations(chunk_id, mentions, entity_links, foodon_ids, enrichment_version)

module:

foodscholar.storage.protocols

.. py:method:: ChunkStore.update_embedding(chunk_id, embedding, embedding_model)

module:

foodscholar.storage.protocols

Patch the chunk’s embedding + embedding_model only.

Used by fs.embed() so re-embedding doesn’t rewrite the mentions / entity_links / foodon_ids payload — a single field-scoped update on the remote backends, a model_copy on the in-memory ones.

.. py:method:: ChunkStore.update_embeddings_bulk(items)

module:

foodscholar.storage.protocols

Bulk variant of update_embedding. Each item is (chunk_id, embedding, embedding_model). Implementations should coalesce into a single network round-trip on remote backends — this is the hot path for fs.embed() over a tunneled cluster, where per-doc updates are network-bound.

.. py:method:: ChunkStore.bulk_set_theme_ids(items)

module:

foodscholar.storage.protocols

Set theme_ids on many chunks without touching shelf_ids.

Layer B’s persist path uses this instead of bulk_update_attachments to avoid a read-then-overwrite race: if a separate writer (e.g., a concurrent fs.attach()) updates shelf_ids between the persist read and the persist write, the bulk-update would clobber the new shelf attachments. bulk_set_theme_ids only touches theme_ids, so it’s safe under any shelf_ids writer.

Passing theme_ids=[] is the explicit ‘remove all themes from this chunk’ signal — used by clear-and-rebuild Layer B runs. Items targeting a missing chunk_id are silently skipped.

.. py:method:: ChunkStore.knn_search_chunks(query_vector, *, k, exclude_ids=None, candidate_ids=None)

module:

foodscholar.storage.protocols

Return the top-k cosine-nearest chunks to query_vector.

• exclude_ids: chunks to omit from the result (typically [query_id]).

• candidate_ids: if provided, restrict the search to these ids (used to constrain the global similarity pass to attached chunks).

Returns [(chunk_id, cosine_score), ...] sorted by score descending. Implementations may skip the score sort if their backend returns it unordered (callers re-sort).

.. py:method:: ChunkStore.scan()

module:

foodscholar.storage.protocols

.. py:method:: ChunkStore.iter_chunks(batch_size=1000)

module:

foodscholar.storage.protocols

.. py:class:: GraphStore(*args, **kwargs)

module:

foodscholar.storage.protocols

Bases: :py:class:~typing.Protocol

.. py:method:: GraphStore.init()

module:

foodscholar.storage.protocols

Provision the underlying store (constraints, indexes). Idempotent.

.. py:method:: GraphStore.upsert_shelves(shelves)

module:

foodscholar.storage.protocols

.. py:method:: GraphStore.clear_layer_a()

module:

foodscholar.storage.protocols

Delete every (:Shelf) node and any edges attached to it.

Called by build_layer_a before re-upsert so stale shelves from a previous projection (with a different blacklist / threshold) don’t survive as ghosts. Local stores clear their shelf dict; Neo4j runs MATCH (s:Shelf) DETACH DELETE s, which kills PARENT_OF, HAS_THEME, HAS_CHUNK, DESCRIBES edges in one shot.

Idempotent — a no-op when no shelves exist.

.. py:method:: GraphStore.upsert_themes(themes)

module:

foodscholar.storage.protocols

.. py:method:: GraphStore.upsert_cards(cards)

module:

foodscholar.storage.protocols

.. py:method:: GraphStore.clear_attachments()

module:

foodscholar.storage.protocols

Delete every (:Chunk)-[:ATTACHED_TO]->(:Shelf) edge.

Called at the start of fs.attach() so a re-run doesn’t leave ghost edges from a previous projection. (:Shelf) nodes themselves survive (they’re the output of build_layer_a, not of attach), as do (:Chunk) stub nodes and any :ATTACHED_TO edges pointing at (:Theme) (those belong to Layer B). Idempotent.

.. py:method:: GraphStore.attach_chunks_to_shelf(shelf_id, attachments)

module:

foodscholar.storage.protocols

Wire (:Chunk)-[:ATTACHED_TO {lifted_from}]->(:Shelf) edges.

Each tuple is (chunk_id, lifted_from). lifted_from lists the FOODON ids on the chunk whose ancestry/collapse resolved to this shelf — empty when the chunk linked the shelf’s own foodon_id directly. Used by the attach phase to record projection provenance on every edge so audits can answer “why is this chunk on this shelf?” without re-running the resolver. Idempotent — re-running with the same shelf+chunk pair overwrites lifted_from.

.. py:method:: GraphStore.attach_chunks_to_theme(theme_id, chunk_ids)

module:

foodscholar.storage.protocols

.. py:method:: GraphStore.attach_chunks_to_themes_bulk(items)

module:

foodscholar.storage.protocols

Wire (:Chunk)-[:THEME_OF {primary, weight}]->(:Theme) edges in bulk.

Each tuple is (chunk_id, theme_id, primary, weight). primary marks the per-shelf primary theme for a chunk (used by retrieval ranking); weight is a continuous score (centroid-distance for similarity themes, edge-degree for relatedness, max-of-both for merged). One network round-trip per call on remote backends — the hot path for Layer B persistence on a tunneled Neo4j.

Idempotent — re-running with the same (chunk_id, theme_id) pair overwrites primary + weight.

.. py:method:: GraphStore.clear_themes(facet=None)

module:

foodscholar.storage.protocols

Delete (:Theme) nodes along with their HAS_THEME and THEME_OF edges. When facet is given, delete only that facet’s themes; when None, delete every theme.

fs.build_layer_b() calls this scoped to the facet it is rebuilding so a re-run with a different config doesn’t leave ghost themes — and, critically, building one facet never wipes another facet’s themes (the notebook loops over facets). Shelves and chunks survive (they’re Layer A artifacts); chunk-side theme_ids denorm is the caller’s responsibility via chunk_store.bulk_set_theme_ids. Idempotent — a no-op when no matching themes exist.

.. py:method:: GraphStore.get_shelf(shelf_id)

module:

foodscholar.storage.protocols

.. py:method:: GraphStore.get_themes_for_shelf(shelf_id)

module:

foodscholar.storage.protocols

.. py:method:: GraphStore.get_chunks_for_theme(theme_id)

module:

foodscholar.storage.protocols

.. py:method:: GraphStore.get_neighbors(shelf_id, hops=1)

module:

foodscholar.storage.protocols

.. py:method:: GraphStore.get_card(target_id, target_type)

module:

foodscholar.storage.protocols

.. py:method:: GraphStore.list_shelves()

module:

foodscholar.storage.protocols

.. py:method:: GraphStore.list_themes()

module:

foodscholar.storage.protocols

.. py:method:: GraphStore.list_chunk_shelf_attachments()

module:

foodscholar.storage.protocols

Return every (:Chunk)-[:ATTACHED_TO]->(:Shelf) edge as a map chunk_id -> {shelf_id, ...}. Used by audit to cross-check the Elastic shelf_ids denorm against the actual edge graph. One round trip; output size proportional to total attach edges.

.. py:method:: GraphStore.list_chunk_foodon_mentions()

module:

foodscholar.storage.protocols

Return every (:Chunk)-[:MENTIONS]->(:Entity) edge whose entity is FOODON-prefixed, as a map chunk_id -> {ontology_id, ...}. Used by audit to verify the Elastic foodon_ids denorm matches what build_entities wrote. One round trip.

.. py:method:: GraphStore.upsert_entities(entities)

module:

foodscholar.storage.protocols

.. py:method:: GraphStore.attach_chunks_to_entity(ontology_id, chunk_links)

module:

foodscholar.storage.protocols

Wire up (:Chunk)-[:MENTIONS {confidence, method}]->(:Entity) edges.

chunk_links is a list of (chunk_id, confidence, method) tuples carrying the per-mention metadata. Implementations must be idempotent — re-running with the same links must not duplicate the edges.

.. py:class:: EntityStore(*args, **kwargs)

module:

foodscholar.storage.protocols

Bases: :py:class:~typing.Protocol

Dedicated, queryable store for first-class linked entities.

Local stores implement init() as a no-op; the Elastic adapter creates a foodscholar_entities index alongside the chunk index.

.. py:method:: EntityStore.init()

module:

foodscholar.storage.protocols

.. py:method:: EntityStore.upsert(entities)

module:

foodscholar.storage.protocols

.. py:method:: EntityStore.get(ontology_id)

module:

foodscholar.storage.protocols

.. py:method:: EntityStore.get_many(ontology_ids)

module:

foodscholar.storage.protocols

.. py:method:: EntityStore.list_by_prefix(prefix, *, k=100)

module:

foodscholar.storage.protocols

.. py:method:: EntityStore.search(query, *, prefix=None, k=10)

module:

foodscholar.storage.protocols

.. py:method:: EntityStore.scan()

module:

foodscholar.storage.protocols

.. py:class:: Embedder(*args, **kwargs)

module:

foodscholar.storage.protocols

Bases: :py:class:~typing.Protocol

.. py:attribute:: Embedder.model_id

module:

foodscholar.storage.protocols

type:

str

.. py:property:: Embedder.dim

module:

foodscholar.storage.protocols

type:

int

.. py:method:: Embedder.embed(texts)

module:

foodscholar.storage.protocols

.. py:class:: CardStore(*args, **kwargs)

module:

foodscholar.storage.protocols

Bases: :py:class:~typing.Protocol

Vector-searchable store for Layer C cards. Cards also live in the graph store (Neo4j); this store holds the embedding for kNN retrieval.

.. py:method:: CardStore.init()

module:

foodscholar.storage.protocols

Idempotently provision backing storage (e.g. the ES cards index).

.. py:method:: CardStore.upsert(cards)

module:

foodscholar.storage.protocols

.. py:method:: CardStore.get_many(card_ids)

module:

foodscholar.storage.protocols

.. py:method:: CardStore.knn_search_cards(query_vector, *, k, exclude_ids=None)

module:

foodscholar.storage.protocols

Return up to k (card_id, cosine_score) nearest the query, best first. Cards without an embedding are not returned.

.. py:class:: LLMClient(*args, **kwargs)

module:

foodscholar.storage.protocols

Bases: :py:class:~typing.Protocol

.. py:attribute:: LLMClient.model_id

module:

foodscholar.storage.protocols

type:

str

.. py:method:: LLMClient.generate(prompt, max_tokens=1024)

module:

foodscholar.storage.protocols

.. py:method:: LLMClient.generate_json(prompt, schema, max_tokens=1024)

module:

foodscholar.storage.protocols

Return a JSON object conforming to schema (a JSON-schema dict).

Uses the provider’s native structured-output mode where available. Guarantees the result parses and matches the schema’s shape — it does NOT guarantee the values are semantically correct (e.g. an LLM-reported character offset may be a valid integer yet wrong). Callers that need correct positions must verify them against the source themselves.

.. py:class:: NER(*args, **kwargs)

module:

foodscholar.storage.protocols

Bases: :py:class:~typing.Protocol

Span-level food entity recognizer.

Implementations should be deterministic given fixed model weights so pipeline reruns produce stable results.

.. py:attribute:: NER.model_id

module:

foodscholar.storage.protocols

type:

str

.. py:method:: NER.extract(text)

module:

foodscholar.storage.protocols

.. py:class:: Linker(*args, **kwargs)

module:

foodscholar.storage.protocols

Bases: :py:class:~typing.Protocol

Maps a Mention to a single ontology id, or None if no candidate clears the threshold.

.. py:attribute:: Linker.linker_id

module:

foodscholar.storage.protocols

type:

str

.. py:method:: Linker.link(mention)

module:

foodscholar.storage.protocols