memo · ADR 0017

17.Chunking depends on length. A short note is one chunk.!

The problem

Retrieval returns chunks and a citation points at a chunk, so the chunk is the unit of both recall and provenance. Every tutorial picks 512 tokens with overlap and moves on. A typed voice note is often forty words, and splitting that in two gives two vectors that each describe half a thought and a citation that points at half a sentence.

The options

Fixed token windows with overlap, the default everywhere; paragraph and sentence boundaries with a length cap; one chunk per note, always; semantic chunking that splits where adjacent sentences stop being similar.

The move

Two regimes. Up to about 200 tokens, which is most notes, the note is exactly one chunk. Above that, split on paragraphs, then sentences, packed into chunks of about 400 tokens with 60 tokens of overlap carried from the previous chunk, and a tail under 50 tokens merged into its predecessor. Tokens are approximated as characters divided by four. Each chunk stores its note, an ordinal and its character offsets, so a citation can highlight the passage rather than name the note.

What we accepted

Overlap duplicates text, so retrieval de-duplicates by note before fusion. Chunk boundaries change if the strategy changes, so chunks are never deleted on re-chunk, only marked superseded, and an old citation keeps resolving to the text it was grounded in. The token approximation is crude for code, URLs and non-Latin scripts, and is named as such so nobody trusts it in a different corpus.

When we would revisit

When recall at ten is low and the failures cluster on long notes. That is the signal for semantic chunking or a smaller target. If citations are often right note, wrong passage, the overlap or the target size is at fault, not the model.

Back to memo