Article 09 of 11

How is agent memory measured?

3 min read · LoCoMo, LongMemEval, BEAM, and why a score needs its token cost.

In this blog, we will learn about the three public benchmarks for long-term memory: LoCoMo, LongMemEval, and BEAM. We will also see why two systems reporting "92" and "94" cannot be compared, and what a fair comparison needs.

The exam analogy

A memory benchmark is an exam with an open book. The book is a very long conversation. The questions ask about things said in it. The grader is another LLM that compares your answer to the gold answer. Three exams exist, and each one tests a different weakness.

LoCoMo: does memory survive a long conversation?

Paper: "Evaluating Very Long-Term Conversational Memory of LLM Agents" (2024). Ten conversations between two people, each about 300 turns across up to 35 sessions, roughly 9,000 to 16,000 tokens, with dates on every session and some shared images. Built with LLM agents grounded in personas and event timelines, then edited by humans.

Question categories:

1  single-hop     the answer is in one turn
2  multi-hop      combine two or more turns
3  temporal       "how long after X did Y happen?"
4  open-domain    needs the conversation plus common sense
5  adversarial    the premise is false; most systems drop this one

Most leaderboards use the ~1,540 questions from categories 1 to 4 and an LLM judge for correctness. Human F1 in the paper was about 88; GPT-4 Turbo with the whole conversation scored about 52. Temporal questions were the biggest gap.

LongMemEval: does memory stay correct when facts change?

Paper: "Benchmarking Chat Assistants on Long-Term Interactive Memory" (2024). Five hundred questions, each planted inside a haystack of user and assistant sessions. The S variant is about 115K tokens over about 40 sessions; the M variant is about 500 sessions, 1.5M tokens.

Question types:

single-session-user         recall something the user said
single-session-assistant    recall something the assistant said
single-session-preference   apply a stated preference
multi-session               combine facts across sessions
knowledge-update            the fact changed; use the newest
temporal-reasoning          dates and durations
abstention                  the answer is not in the history; say so

Findings: commercial assistants dropped about 30 percent versus an oracle that only sees the evidence sessions. Three fixes helped: store at round granularity rather than whole sessions, add extracted facts as extra keys for each round (+9.4 recall, +5.4 accuracy), and expand time expressions in queries (+7 to +11 on temporal). The judge is GPT-4o, agreeing with humans 97 percent of the time.

BEAM: does memory hold at extreme length?

Paper: "Beyond a Million Tokens" (ICLR 2026). One hundred conversations at 128K, 500K, 1M and 10M tokens, with 2,000 validated questions across ten abilities:

information extraction    multi-hop reasoning      knowledge update
temporal reasoning        summarization            preference following
abstention                contradiction resolution event ordering
instruction following

Plain long-context models score around 0.30 to 0.36 and fall further at 10M. The paper's own system, LIGHT, adds three memories, episodic (vector retrieval over segments), working (recent turns), and a scratchpad of salient facts, for +3.5 to +12.7 points. Mem0 reports 64.1 at 1M and 48.6 at 10M.

Why the leaderboard is a mess

Mem0's own 2026 leaderboard post lists LoCoMo scores of 96.1, 94.7, 92.5, 92.2 and 88.2 from five vendors, all self-reported, one of them disputed by a third party at 75.1. The post's own caveat: small protocol differences compound into large score differences.

What moves a score without touching the memory system:

the answer model (GPT-4o-mini versus Gemini 3 Pro is tens of points)
the judge model and its prompt
the retrieval budget (Mem0 uses top-200, about 7K tokens)
whether a reranker was used
which categories were dropped

So a score is meaningful only as a row in a table where every other row used the same models, the same budget, and the same categories.

What a fair run looks like

ingest     every session, with its date, through the write path
search     one query per question, fixed k, fixed token cap
answer     one fixed prompt, one fixed model, JSON with cited memory ids
judge      one fixed prompt, one fixed model
report     accuracy per category + mean tokens + p50/p95 latency

Mem0 open-sourced this loop (memory-benchmarks on GitHub) with GPT-4o-mini for extraction, text-embedding-3-small, and GPT-4o as answerer and judge, at cutoffs of 10, 20, 50, and 200. We reuse the shape and hold out two LoCoMo conversations so we never tune on the test.

Reading a score

The number that matters is not the single accuracy. It is the pair (accuracy, tokens per question) and the pair (single-session score, multi-session score). Mem0's LongMemEval shows 98 on single-session assistant recall and 88 on multi-session. Every system's frontier is the same: connecting facts across sessions.

Closing

Three exams, three weaknesses: LoCoMo for long conversations, LongMemEval for facts that change, BEAM for extreme length. A score without its token cost and its model stack is a rumour. We will publish all three together.

← All articles