Article 11 of 11

How does memory scale to a million users?

3 min read · Async writes, shard by user, shrink the vectors with TurboQuant.

In this blog, we will learn what breaks when the card file grows from one user to a million, and the three moves that fix it: async writes, sharding by user, and compressing vectors. We will also see what TurboQuant does and why it fits a store that changes every second.

The numbers

From the class notes:

registered users             1,000,000
daily active                 200,000
read path                    ~28 questions/s, peak ~83/s
write path                   ~4.6 sessions/s, peak ~14/s
memories per user            ~200
total memories               200,000,000
one memory                   ~100 B text + 3,072 B vector + ~150 B payload = ~3.3 KB
raw store                    ~664 GB, and 92 percent of it is vectors

Read the last line twice. The text is 50 GB. The vectors are 614 GB. If you want a smaller store, shrink the vectors.

Move 1: never make the user wait for a write

The write path is an LLM call plus an embed call plus a few inserts, one to three seconds. The user must never sit through it. The answer streams; the transcript goes on a queue; a worker extracts later. Freshness target is "by the next session", which is seconds, not milliseconds.

request --> answer LLM --> stream to user
                 |
                 +--> queue --> worker: extract, embed, insert

At 14 sessions a second at peak and about 3 seconds per job, a handful of workers keep up. The stdlib queue works on one box; Redis works on many.

Move 2: shard by user

Every memory query filters on user_id. That filter is the privacy wall, and it is also the sharding key. Hash the user id, pick a bucket, and each bucket is a self-contained store.

user_id --> hash --> bucket 0..N-1 --> its own SQLite file / Postgres partition

A query never crosses buckets, so a cross-user leak becomes structurally impossible instead of a filter you might forget. Replicate each bucket three times for durability. The notes land at about 204 GB per full copy after quantization, 612 GB on disk with three replicas.

Move 3: shrink the vectors

A 768-dimensional float32 vector is 3,072 bytes. Almost all of that precision is wasted on a nearest-neighbour search.

float32            3,072 B    baseline
int8               768 B      4x smaller, recall loss about nil
TurboQuant 3.5 bit ~336 B     9x smaller, quality neutral in the paper
TurboQuant 2.5 bit ~240 B     13x smaller, marginal loss

int8 is the easy first step: scale each coordinate into 256 levels. It takes the vector store from 614 GB to 154 GB with no visible recall change. That is what the notes assume.

What TurboQuant does

TurboQuant (Google Research, 2025) is a way to squeeze each coordinate to a few bits without training anything on your data. Two stages:

1. rotate    multiply by a random rotation matrix. In high dimensions this
             makes every coordinate look like the same bell-shaped
             distribution, no matter what the original vector was.
2. quantize  since every coordinate now has the same known shape, use the
             optimal scalar quantizer for that shape on each one,
             independently. A few bits per coordinate.
3. residual  keep a 1-bit sketch of what was lost (a "quantized
             Johnson-Lindenstrauss" transform) so that inner products
             are unbiased, not just close on average.

The theory says this is within a small constant of the best any quantizer can do at that bit rate. The practical claims: 3.5 bits per coordinate is quality-neutral, better recall than product quantization on nearest neighbour search, and indexing time close to zero.

That last one is why it fits a memory store. Product quantization has to learn a codebook from your data, and the codebook goes stale as the data changes. TurboQuant needs one random matrix, fixed forever. New memories arrive every second; nothing has to be retrained.

Analogy: product quantization is a tailor who measures your whole wardrobe before sewing anything, and has to re-measure when you buy new clothes. TurboQuant is a standard size chart that fits everyone well enough, decided once.

The plan for `turbovec`

A small library, later, with one job: store a matrix of vectors in a few bits per coordinate and answer top-k inner-product queries fast.

encode(X: float32[n, d]) -> codes (uint8 packed), scale, plus the rotation seed
search(q: float32[d], k) -> indices, approximate scores

Start with int8 (a weekend), then TurboQuant 4-bit and 3-bit, and measure recall at 10 against exact search on real memory embeddings. The Store protocol in hmem is the seam: swap the numpy dot product for a turbovec index and nothing above it changes.

The read budget stays flat

One more scale rule, from the very first blog. The prompt gets a fixed token budget of memories, not a fixed number of rows. Store 8 memories or 8,000 per user: the read cost per message does not move. That is the whole reason memory beats history, and it is the number that keeps the cost table sane.

memory layer, 200k daily active, small assistant model      ~$312/day
resending 10 sessions of history, same model                ~$5,400/day

Closing

Scale is three moves: write later, shard by user, shrink the vectors. The first two are plumbing. The third is where the bytes are, and a random rotation plus a size chart shrinks them nine times without a training step.

← All articles