Skip to main content

Chunking Strategies

Bad chunks hand the retriever bad evidence, and the model answers from it anyway.


What it is

Chunking is the process of breaking a large document into smaller, manageable pieces before embedding and storing them in a vector database. Because LLMs have context window limits and embedding models have maximum token constraints, you cannot process a 1,000-page manual all at once.

A chunking strategy determines how to break the text. You can split by character count, by word, by paragraph, or by semantic structure (like Markdown headers or code functions).

graph TD
    A[Large Document] --> B{Chunking Strategy}
    B -->|Fixed Size| C[Chunk 1: 500 tokens]
    B -->|Fixed Size| D[Chunk 2: 500 tokens]
    B -->|Semantic| E[Chunk 1: Function A]
    B -->|Semantic| F[Chunk 2: Function B]

Why it matters in production

If you blindly chop text every 500 characters, you will split sentences in half and sever code blocks from their declarations. When the retriever fetches that chunk later, the LLM will lack the necessary context to answer the user's question, leading to hallucinations.

In production, bad chunking is a common cause of RAG failure. A FixedSizeChunkStrategy that respects the unit of meaning keeps the text's sense intact, which is what makes vector search accurate.


How Agenthood implements it

Agenthood plans to implement chunking via the ChunkStrategy interface, specifically utilizing a FixedSizeChunkStrategy for code and markdown parsing.

This is implemented in src/rag/ChunkStrategy.ts (shipped in Phase 1):

export interface ChunkStrategy {
  chunk(text: string, metadata: FileMetadata): DocumentChunk[];
}

export class FixedSizeChunkStrategy implements ChunkStrategy {
  chunk(text: string, metadata: FileMetadata): DocumentChunk[] {
    const size = metadata.type === 'code' ? 250 : 500;
    return splitByTokens(text, size, metadata);
  }
}

Code cannot be split at an arbitrary offset, because a function has to stay whole.


Hands-on example

Though the formal pipeline is under development, you can test semantic chunking concepts using standard text processing:

# A future command to test chunking
# No dedicated CLI subcommand: use the API directly via the runtime

Or conceptually in TypeScript:

// Example of a naive vs semantic split
const text = "export class Agent { ... }";
// Bad: text.slice(0, 10) -> "export cla"
// Good: extract syntax tree node

Further reading