Document Chunking

A practical example

Example: a 40-page security whitepaper split by section heading lets an assistant retrieve the complete 'data retention' section, while fixed 500-character splits cut that same section into fragments that answer questions poorly.

What to evaluate before investing

• Ask whether chunk size and overlap are configurable per source, or fixed by the vendor.
• Test retrieval with questions that require one specific section of a long document — does the answer cite the right passage?
• Ask how tables, lists, and headers are handled, since naive splitting often breaks their structure and meaning.

Limitations and tradeoffs

Tradeoff: small chunks retrieve precisely but lack surrounding context, while large chunks preserve context but dilute relevance and consume more of the model's window.

There is no universal setting; it must be tuned per content type.