Indexing
Embed every chunk and store it in a vector database, ready to search.
Indexing is the one-time (or periodic) job of preparing your knowledge base for retrieval. It happens before any user asks a question.
Indexing pipeline
- Load your documents.
- Chunk them into passages.
- Embed each chunk with an embedding model.
- Upsert the vectors, text, and metadata into a vector database.
When documents change, re-index the affected ones so the knowledge stays current.
Example
for (const doc of documents) {
const chunks = chunk(doc.text, { size: 300, overlap: 40 });
for (const [i, text] of chunks.entries()) {
const vector = await embed(text);
await vectorDb.upsert({
id: `${doc.id}-${i}`,
vector,
metadata: { source: doc.id, title: doc.title, text }
});
}
}When to use it
- A support team indexes all 5,000 help articles nightly so the RAG pipeline always searches the latest content with minimal rebuild time.
- A legal-discovery tool indexes case documents into a vector store at upload time so any query is answered instantly without per-query embedding.
- A news aggregator re-indexes embeddings every hour so breaking stories appear in search results within minutes of publication.
More examples
Embed and store chunks in ChromaDB
Embeds each chunk and upserts it into a ChromaDB collection, creating a searchable vector index.
import chromadb
from openai import OpenAI
client = OpenAI()
db = chromadb.Client()
collection = db.get_or_create_collection('knowledge-base')
chunks = [
'Refund policy: full refund within 30 days.',
'Shipping: 5-7 business days standard.',
'Returns require original packaging.',
]
for i, chunk in enumerate(chunks):
vec = client.embeddings.create(
model='text-embedding-3-small', input=chunk
).data[0].embedding
collection.add(ids=[f'chunk-{i}'], embeddings=[vec], documents=[chunk])
print(f'Indexed {len(chunks)} chunks')Batch-index documents efficiently
Passes all chunks to the embeddings API in one call for efficiency, avoiding one API request per chunk.
from openai import OpenAI
client = OpenAI()
docs = [
'Chunk one text here.',
'Chunk two text here.',
'Chunk three text here.',
]
# Embed all chunks in a single API call
response = client.embeddings.create(
model='text-embedding-3-small',
input=docs
)
vectors = [item.embedding for item in response.data]
# vectors[i] corresponds to docs[i]
for doc, vec in zip(docs, vectors):
print(f'Doc: "{doc[:30]}..." -> {len(vec)}-dim vector')Index with metadata for filtering
Stores metadata alongside each embedding so you can filter search results by category or source later.
import chromadb
from openai import OpenAI
client = OpenAI()
db = chromadb.Client()
collection = db.get_or_create_collection('docs')
chunks_with_meta = [
{'text': 'Refund policy details.', 'source': 'faq', 'category': 'billing'},
{'text': 'How to reset password.', 'source': 'faq', 'category': 'account'},
{'text': 'Shipping to EU countries.', 'source': 'faq', 'category': 'shipping'},
]
for i, item in enumerate(chunks_with_meta):
vec = client.embeddings.create(
model='text-embedding-3-small', input=item['text']
).data[0].embedding
collection.add(
ids=[f'doc-{i}'],
embeddings=[vec],
documents=[item['text']],
metadatas=[{'source': item['source'], 'category': item['category']}]
)
print('Indexed with metadata')
Discussion