Chunking
Split documents into small passages so retrieval is precise.
You cannot embed a whole 50-page manual as one vector — it would be too coarse to match a specific question. Chunking splits documents into smaller passages before embedding.
Chunking tips
- Aim for passages of a few hundred tokens.
- Split on natural boundaries: paragraphs, headings, sections.
- Use a small overlap between chunks so a sentence split across a boundary is not lost.
- Keep useful metadata (source, title, page) with each chunk.
Example
{
"document": "handbook.md",
"chunk_size_tokens": 300,
"overlap_tokens": 40,
"chunks": [
{"id": "handbook-0", "text": "Section 1: Onboarding ..."},
{"id": "handbook-1", "text": "Section 2: Time off ..."}
]
}When to use it
- A documentation site splits each help article into 300-token overlapping chunks so retrieval matches a specific paragraph, not the whole page.
- A legal platform chunks contracts by clause boundaries rather than token count to ensure each retrieved chunk is a complete, coherent clause.
- An academic search tool chunks research papers into section-level blocks (abstract, intro, methods) to return the relevant section to each query.
More examples
Fixed-size token chunking
Splits text into fixed 200-token chunks with a 20-token overlap to preserve context at boundaries.
import tiktoken
enc = tiktoken.encoding_for_model('gpt-4o-mini')
CHUNK_SIZE = 200
OVERLAP = 20
def chunk_tokens(text, size=CHUNK_SIZE, overlap=OVERLAP):
tokens = enc.encode(text)
chunks = []
start = 0
while start < len(tokens):
end = min(start + size, len(tokens))
chunks.append(enc.decode(tokens[start:end]))
start += size - overlap
return chunks
with open('/tmp/article.txt') as f:
text = f.read()
chunks = chunk_tokens(text)
print(f'{len(chunks)} chunks, first 80 chars: {chunks[0][:80]}')Sentence-boundary chunking
Splits on sentence boundaries rather than mid-word so each chunk ends at a natural break point.
import re
import tiktoken
enc = tiktoken.encoding_for_model('gpt-4o-mini')
MAX_TOKENS = 150
def sentence_chunks(text, max_tokens=MAX_TOKENS):
sentences = re.split(r'(?<=[.!?]) +', text)
chunks, current = [], ''
for s in sentences:
candidate = (current + ' ' + s).strip()
if len(enc.encode(candidate)) <= max_tokens:
current = candidate
else:
if current:
chunks.append(current)
current = s
if current:
chunks.append(current)
return chunks
text = 'RAG retrieves documents. It then augments the prompt. Finally the model generates an answer.'
for i, chunk in enumerate(sentence_chunks(text)):
print(f'Chunk {i}: {chunk}')Recursive character text splitter
Uses LangChain's recursive splitter which tries paragraph, then sentence, then word boundaries in order.
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=300,
chunk_overlap=30,
separators=['\n\n', '\n', '. ', ' ', '']
)
with open('/tmp/article.txt') as f:
text = f.read()
chunks = splitter.split_text(text)
print(f'{len(chunks)} chunks')
print('First chunk:', chunks[0][:100])
Discussion