Layer C — Cards¶

Layer C writes the prose a reader actually sees: a short summary card for each Layer B theme. Where Layers A and B are structure, Layer C is the human-facing explanation — produced by a deliberately cheap two-stage pipeline and stored as a vector-searchable artifact.

The cost problem, and the two-stage answer¶

A single theme can hold hundreds of chunks (tens of thousands of words). Sending all of that to an LLM, per theme, across a whole facet, is expensive. Layer C’s core idea is to do the bulk compression with a near-zero-cost extractive method first, and only hand the LLM a small, already-distilled extract.

        flowchart LR
    T["theme<br/>(N chunks, ~150 KB)"] --> S1["Stage 1 — extractive<br/>(LexRank, free)"]
    S1 --> X["extract<br/>(~1–2 KB)"]
    X --> S2["Stage 2 — LLM refine<br/>(Llama 8B)"]
    S2 --> CARD["Card<br/>(title, summary, …)"]
    CARD --> EMB["embed title+summary"]
    EMB --> ES[("cards index<br/>dense_vector")]
    CARD --> NEO[("Neo4j<br/>(:Card) node")]

The LLM sees only the extract, never the raw chunks — that is the whole cost saving.

Stage 1 — cheap extractive compression¶

Stage 1 reduces a theme’s chunks to a compact extract using a classical, zero-API-cost summarizer chosen by config.layer_c.stage1_method. Five are available behind one BaseSummarizer interface:

Method	Backend	Notes
`lexrank` (default)	sumy	Graph-centrality sentence ranking. Won the benchmark — cleanest prose.
`lsa`	sumy	Latent-semantic ranking. The other front-runner.
`luhn`	sumy	Classic word-frequency significance.
`textrank`	sumy	PageRank over a sentence-similarity graph.
`nltk_freq`	nltk	Hand-rolled normalized word-frequency scorer.

Map-reduce for big themes¶

Extractive methods like LexRank build an S × S sentence-similarity matrix — O(S²). A theme with hundreds of chunks → thousands of sentences → an expensive matrix. So Stage 1 escalates to map-reduce once the input exceeds map_reduce_threshold sentences (default 400): partition the chunks into character-budgeted groups, summarize each, concatenate the group summaries, and summarize once more. Nothing is dropped.

        flowchart TD
    C["theme chunks"] --> Q{"> map_reduce_threshold<br/>sentences?"}
    Q -->|no| ONE["single pass → extract"]
    Q -->|yes| MAP["MAP: group by char budget<br/>→ summarize each group"]
    MAP --> GS["group summaries"]
    GS --> RED["REDUCE: summarize the<br/>concatenated summaries"]
    RED --> EX["extract"]
    ONE --> EX

Tables defeat naïve sentence splitting

On a real corpus full of markdown nutrition tables (| whole | 150 | 275 |), a naïve “split on .!?” treats an entire table as one giant sentence — so a “top-8 sentences” extract becomes a 17 KB table wall. Layer C’s splitter strips pipe-delimited table runs, drops non-prose fragments, and clamps over-long pseudo-sentences before any method runs. This was found by the benchmark (below) and is why LexRank/LSA — which rank prose above tables — looked clean while the others didn’t until the fix.

Honest about evidence¶

A card carries metadata that keeps it honest rather than authoritative:

evidence_quality grades the overall strength of the material: high → medium → low, plus debated (sources genuinely conflict) and unclear (too thin to judge).
controversy_note is the specifics of any disagreement; confidence_note flags thin coverage.
safety_flagged is set when the theme’s facet is in config.layer_c.safety_sensitive_facets (e.g. allergies), so such cards can be reviewed or withheld.

Stage 3 — embed + vector-search¶

Every card is embedded (title + summary, via the same BGE embedder used for chunks) and written to two stores: the Neo4j (:Card) node (as before) and a dedicated Elastic foodscholar_cards index (dense_vector, HNSW, cosine). That makes cards retrievable by meaning, not just by graph lookup.

        flowchart LR
    Q["query text"] --> E["embed (BGE)"]
    E --> KNN["knn over foodscholar_cards"]
    KNN --> H["nearest card ids + scores"]
    H --> R["fetch Card records"]

fs.build_layer_c(facet="foods")            # build, embed, persist (Neo4j + cards index)
fs.search_cards("calcium and vitamin D for bones", k=5)   # vector search → nearest Cards
fs.graph.theme("...").card().summary       # graph lookup of a theme's card

Note

fs.search_cards is a thin retrieval helper (embed → kNN → fetch). The full fs.query() with answer synthesis remains deferred; Stage 3 provides the searchable card store it will build on.

Where Layer C sits¶

        flowchart LR
    LA["Layer A<br/>shelves"] --> LB["Layer B<br/>themes"]
    LB --> LC["Layer C<br/>cards"]
    LC --> NEO[("Neo4j<br/>(:Card)")]
    LC --> CARDS[("cards index<br/>(vectors)")]

fs.build() runs the whole arc — embed → build_entities → build_layer_a → attach → build_layer_b → build_layer_c. Because every card traces back through its theme’s chunks to their source documents, the trail card → theme → chunk → source stays auditable end to end.