The plan

Working name of the library: hmem (hybrid memory). One Python package, a handful of small modules, SQLite underneath, any OpenAI-compatible LLM on top.

1. Goal

Build the memory layer from the class notes (memory/notes.md) as real, runnable, modular code, borrowing the best ideas from Mem0 (the code in mem0/) and HydraDB (append-only temporal graph), and measure it on the three public benchmarks: LoCoMo, LongMemEval, BEAM.

Success, in order:

1. We understand memory systems well enough to explain every design choice in one sentence (see knowledge/).
2. hmem runs end to end on a laptop with no Docker and no Ollama, only an API key (OpenAI-compatible) and SQLite.
3. Benchmark scores that beat the open-source numbers people actually reproduce, with a smaller token budget than Mem0's 7K per question.
4. A scale story (sharding, quantization, async writes) that we can defend with the numbers in the notes (1M users, 200M memories, ~664 GB raw).

Non-goals for v1: document RAG, multi-agent shared memory, a hosted API, a UI. Graph databases (Neo4j etc.) are out: the "graph" lives as two SQLite tables.

2. What the research says (short)

Two lessons dominate everything else:

• Recall is the whole game. Every leaderboard system retrieves a lot (Mem0 uses top-200) and lets a strong answer model sort it out. Precision matters for cost, recall matters for score.
• Never destroy information. Mem0 v3 dropped UPDATE/DELETE and gained 20 points. HydraDB and Zep keep every version with timestamps. Overwriting is how "where did I use to live?" becomes unanswerable.

3. The hybrid: what goes where

                     class notes            Mem0 v3               HydraDB / Zep
write path   ------  categories,     +  ADD-only rich      +  supersede-with-time,
                     importance,        memories, dedup,      provenance, resolve
                     privacy policy     entity links          references at write time
read path    ------  recency +       +  dense + BM25 +     +  time filter on
                     importance         entity boost          valid_from / valid_to,
                     scoring                                  graph hop over links
store        ------  one collection, +  SQLite history    +  append-only, versions
                     user_id filter     table                 never overwritten
forgetting   ------  episodic decay, +  (none)            +  (roadmap "decay engine")
                     dedup at write

Concretely, our memory row is a bitemporal fact:

id            uuid
user_id       maya                        privacy wall: every query filters on it
text          "Maya moved from London to Paris in March 2026 for a payments job"
category      semantic | episodic | procedural
importance    1..10  (LLM-rated at write time)
event_date    2026-03-01        when it happened / started (valid time)
valid_from    2026-03-02T09:14  when the system started believing it
valid_to      NULL              set when a newer fact supersedes it; never deleted
superseded_by uuid or NULL      the newer fact
source_ids    ["s18241:t3"]     provenance: session + turn ids the fact came from
hash          md5(text)         exact dedup
embedding     float32[d]        blob; int8 / turbovec later
last_accessed 2026-06-09T18:02  recency term

Plus three small side tables: episodes (the raw turns, round-level, with timestamps, so retrieval can hand the answer model the original words), entities (name -> ids of memories that mention it), and SQLite FTS5 over text for BM25. That is the whole "graph": memories linked by shared entities and by superseded_by.

4. Architecture

flowchart TD
    subgraph read["READ PATH (sync, must be fast)"]
        Q["user question + user_id"] --> X["query prep<br/>time expressions -> date range<br/>entities from query"]
        X --> D["dense search<br/>cosine over user's vectors"]
        X --> K["keyword search<br/>SQLite FTS5 (BM25)"]
        X --> E["entity match<br/>entities table"]
        D --> F["fuse<br/>weighted sum, then<br/>recency + importance + time filter"]
        K --> F
        E --> F
        F --> H["hop: pull superseded_by chain<br/>and source episodes for top hits"]
        H --> P["prompt block<br/>dated facts, newest first"]
    end
    subgraph write["WRITE PATH (async, seconds are fine)"]
        M["session messages"] --> S["similar existing memories (top 10)<br/>+ profile summary + last k turns"]
        S --> L[["extractor LLM, one call<br/>strict JSON"]]
        L --> C["candidates: text, category, importance,<br/>event_date, links, supersedes"]
        C --> DD["dedup: hash, then cosine > 0.95"]
        DD --> W["insert memory + embedding + entities<br/>stamp valid_to on superseded rows<br/>save raw episodes"]
    end
    W --> DB[("SQLite: memories, episodes,<br/>entities, memories_fts")]
    DB --> D
    DB --> K
    DB --> E

Write path, in pseudocode

add(messages, user_id, observed_at):
    save_episodes(messages, user_id, observed_at)          # raw turns, round-level
    similar = dense_search(embed(join(messages)), user_id, k=10)
    profile = latest_profile(user_id)                      # optional running summary
    cands   = llm.json(EXTRACT_PROMPT, {similar, profile, last_k, messages, observed_at})
    for c in cands:
        if md5(c.text) in known_hashes: continue           # exact dup
        v = embed(c.text)
        if max_cos(v, similar) > 0.95: continue            # near dup
        mid = insert_memory(c, v, user_id)
        link_entities(mid, c.entities or ner(c.text))
        for old in c.supersedes: mark_superseded(old, by=mid, at=observed_at)

One LLM call per add, one batch embed call. No per-fact update calls. That is the whole reason Mem0 v3 is 2x faster than Mem0 v1.

Read path, in pseudocode

search(query, user_id, k=10, as_of=None):
    q     = embed(query)
    dates = parse_time(query)                              # "last summer" -> range
    dense = top(cos(q, vectors[user_id]), 4k)              # over-fetch
    bm25  = fts5(query, user_id, 4k)
    ents  = entity_hits(query, user_id)
    score = 0.6*dense + 0.3*bm25 + 0.1*ents                # each normalized to [0,1]
          + w_rec * recency(last_accessed)
          + w_imp * importance/10
    if dates: keep rows whose event_date in range (soft boost if none match)
    if as_of: keep rows with valid_from <= as_of < valid_to
    hits = top(score, k); touch(last_accessed)
    return hits + superseded chain + source episodes (opt-in)

The prompt block is the class notes' injection format, plus dates:

Known facts about the user (newest first, with dates):
- Maya lives in Paris. (since 2026-03; replaced "lives in London", 2025-01 to 2026-03)
- Maya is vegetarian. (since 2026-01)

5. Code layout (KISS)

hmem/
  __init__.py      exports Memory
  config.py        one dataclass: model names, base_url, db path, weights
  llm.py           LLM protocol + OpenAI-compatible impl (+ MockLLM for tests)
  embed.py         Embedder protocol + OpenAI-compatible impl + local (fastembed) + Mock
  store.py         SQLite schema, insert/search/list/delete, FTS5, vector blobs
  extract.py       write path: prompt -> candidates -> dedup -> store
  retrieve.py      read path: dense + bm25 + entity -> fuse -> rerank
  prompts.py       EXTRACT, ANSWER (chain-of-note JSON), JUDGE
  timeparse.py     "last week", "in 2023", "yesterday" -> (start, end) given a reference date
  memory.py        the facade: add / search / get_all / forget / delete_user
bench/
  common.py        answer + judge loop, caching, per-category report
  locomo.py        10 conversations, ~1,540 Q (drop adversarial), categories 1-4
  longmemeval.py   500 Q, S variant first
  beam.py          128K conversations first, then 1M
tests/
  test_store.py, test_extract.py, test_retrieve.py, test_memory.py (mock LLM/embedder, no network)

Rules: every module under ~250 lines, no module imports from a sibling except through the protocols in llm.py / embed.py / store.py, no framework. numpy for the dot products, stdlib sqlite3 for everything else, openai for the API. Optional extras: fastembed (local embeddings), spacy (better entities; regex proper-noun fallback without it).

6. Benchmark plan

Order: LoCoMo (smallest, 10 conversations) -> LongMemEval-S -> BEAM-128K -> BEAM-1M. Same loop every time: ingest sessions with their timestamps -> search per question -> answer with a fixed prompt -> judge with a fixed prompt and model -> report per category.

Rules we hold ourselves to: same answer model and judge model across all systems we compare; publish token count per question next to every score; never tune on the test questions (hold out 2 LoCoMo conversations for development).

Where the points come from, by question type:

• single-hop / information extraction: recall, so hybrid search + over-fetch
• multi-hop / multi-session: entity links + one hop over linked memories
• temporal: dates on every memory, time-aware query expansion, event_date filter
• knowledge update / contradiction: superseded_by chain shown in the prompt
• abstention: answer prompt that must cite a memory id or say "unknown"
• preference / instruction following: procedural category always injected

7. Scale plan (after the benchmarks work)

1. Async write: add returns immediately; a worker pulls from a queue (stdlib queue first, Redis later). Notes: 4.6 writes/s average, 14/s peak.
2. Sharding: one SQLite file per user-hash bucket (or Postgres + pgvector behind the same Store protocol). The user filter is built into the shard key, so a cross-user leak is structurally impossible.
3. Vector budget: 200M memories x 3,072 B = 614 GB of float32. int8 gets 4x (154 GB). turbovec with TurboQuant at 3.5 bits/coordinate gets ~9x (~67 GB) with near-zero recall loss and no training step, which is why it beats product quantization for a store that changes every second.
4. Forgetting: nightly job: expire episodic memories 90 days after last_accessed, never semantic or procedural; superseded rows stay but drop out of default search.
5. Read budget: the injected block is capped in tokens, not rows, so cost is flat no matter how much is stored.

8. Milestones

• M0 uv init, deps (numpy, openai, pytest), hmem/config.py, mocks
• M1 store.py with tests: insert, user-filtered dense search, FTS5, supersede, delete_user
• M2 extract.py + prompts.py: ADD-only extraction with links/supersedes, dedup; tests with MockLLM
• M3 retrieve.py + timeparse.py: fusion, recency/importance, time filter; tests
• M4 memory.py facade + a 30-line demo script reproducing the class experiments
• M5 bench/locomo.py end to end on 2 dev conversations, then all 10; per-category report
• M6 error triage by category, prompt/weights iteration (never on held-out)
• M7 bench/longmemeval.py (S), abstention handling
• M8 bench/beam.py at 128K, then 1M; profile the ingest cost
• M9 async worker, int8 vectors, turbovec spike with TurboQuant
• M10 write-up: scores, tokens, latency, what moved the needle

9. Decisions made

• ADD-only extraction with explicit supersedes. The LLM never deletes. It may say "this replaces memory 3", and we stamp valid_to. Old truth stays queryable. (Mem0 v3 + HydraDB.)
• Facts and raw rounds both indexed. LongMemEval showed fact-augmented keys and round granularity each add several points. Facts are the keys, rounds are the values the answer model may read.
• SQLite, not a vector DB, for v1. 200 memories per user is a numpy dot product. FTS5 gives BM25 for free. The Store protocol is the seam for Postgres/Qdrant later.
• Weighted-sum fusion, not RRF, by default. Mem0's additive score is easy to explain and to explain=True. RRF stays as a one-line option.
• No spaCy dependency in the core. Entities come from the extractor LLM (it already knows the names) with a regex proper-noun fallback.

10. Open questions (answer with data, not opinion)

• Does a running per-user profile summary (LIGHT's scratchpad) help the extractor enough to pay for itself?
• Weighted sum vs RRF vs cross-encoder rerank: measure on LoCoMo dev split.
• How large should the default read budget be in tokens: 500, 2K, 7K?
• Is event_date extraction reliable enough from a small model, or do we need a second pass only for temporal questions?

11. Environment notes

• Machine has Python 3.12 and uv; no Ollama, no Docker, no API keys in the shell. Set HMEM_BASE_URL / HMEM_API_KEY / HMEM_LLM / HMEM_EMBED (any OpenAI-compatible endpoint: OpenAI, OpenRouter, Groq, a local vLLM).
• Tests run fully offline with MockLLM and MockEmbedder.
• Benchmarks need a real key. Budget estimate for LoCoMo end to end with a small extractor and gpt-4o-class answerer/judge: a few dollars per run.