rajdeep@portfolio — zsh
$
building0%

press any key to skip

Mastering Text Chunking: The Key to Unlocking LLM Power and Efficiency

Rajdeep Sengupta

Published on : Feb 28, 2025

Text Chunking Methods: How They Work & When to Use Them

Text chunking is an essential step in processing large documents, whether for natural language processing (NLP), search indexing, or working with large language models (LLMs). Chunking methods break text into manageable pieces while maintaining context and coherence. In this blog, we'll explore different chunking techniques, their applications, and how to implement them in Python and LangChain.

Why Chunk Text?

  • Optimizing NLP Models: Most LLMs have token limits (e.g., GPT-4 Turbo has a 128k token limit, Claude 2 has a 200k context size, and Gemini models support large contexts), requiring text to be split into smaller chunks.
  • Efficient Information Retrieval: Chunking ensures that relevant sections of a document are indexed properly.
  • Preserving Context: Some chunking strategies retain meaningful context rather than splitting arbitrarily.
  • Reducing API Costs: Sending only relevant chunks to an LLM can save on API usage costs.

1. Fixed-Length Chunking

How It Works

This method splits text into chunks of a fixed number of characters or words.

When to Use It

  • When uniform chunk sizes are needed.
  • When splitting logs, structured reports, or articles with a known format.
  • When content order doesn't matter significantly.

Implementation

python

from textwrap import wrap

def fixed_length_chunking(text, chunk_size):
    return wrap(text, width=chunk_size)

# Example
text = "This is a sample text that needs to be chunked into smaller pieces for processing."
chunks = fixed_length_chunking(text, 50)
print(chunks)

2. Sentence-Based Chunking

How It Works

This method splits text at sentence boundaries to maintain semantic integrity.

When to Use It

  • When preserving sentence structure is important.
  • For tasks like summarization, question answering, or chatbots.

Implementation

python

import re

def sentence_based_chunking(text):
    sentences = re.split(r'(?<=[.!?]) +', text)
    return sentences

# Example
text = "This is the first sentence. Here is another one! And yet another?"
chunks = sentence_based_chunking(text)
print(chunks)

3. Token-Based Chunking

How It Works

Instead of splitting by characters or sentences, this method breaks text into chunks based on token count.

When to Use It

  • When working with LLMs that have token constraints.
  • When ensuring each chunk fits within a model’s limit.

Implementation using LangChain

python

from langchain.text_splitter import TokenTextSplitter

def token_based_chunking(text, chunk_size):
    splitter = TokenTextSplitter(chunk_size=chunk_size)
    return splitter.split_text(text)

# Example
text = "This is a long text that needs to be tokenized for an NLP model."
chunks = token_based_chunking(text, 100)
print(chunks)

4. Recursive Chunking

How It Works

This approach first tries to split by large semantic units (like paragraphs), then smaller units (like sentences) if necessary.

When to Use It

  • When preserving semantic structure is crucial.
  • For document-based question answering or semantic search.

Implementation using LangChain

python

from langchain.text_splitter import RecursiveCharacterTextSplitter

def recursive_chunking(text, chunk_size):
    splitter = RecursiveCharacterTextSplitter(chunk_size=chunk_size)
    return splitter.split_text(text)

# Example
text = "Paragraph one is long.\n\nHere is another paragraph.\n\nYet another paragraph."
chunks = recursive_chunking(text, 200)
print(chunks)

5. Semantic Chunking with Embeddings

How It Works

This technique uses text embeddings to identify semantic breakpoints rather than arbitrary character or word limits.

When to Use It

  • When high semantic accuracy is needed.
  • For intelligent document segmentation and retrieval.

Implementation

python

from sentence_transformers import SentenceTransformer
import numpy as np

def semantic_chunking(text, model_name, threshold=0.5):
    sentences = sentence_based_chunking(text)
    model = SentenceTransformer(model_name)
    embeddings = model.encode(sentences)
    chunks = []
    temp_chunk = []

    for i in range(len(sentences)):
        if len(temp_chunk) == 0:
            temp_chunk.append(sentences[i])
        else:
            sim = np.dot(embeddings[i], embeddings[i-1]) / (np.linalg.norm(embeddings[i]) * np.linalg.norm(embeddings[i-1]))
            if sim > threshold:
                temp_chunk.append(sentences[i])
            else:
                chunks.append(" ".join(temp_chunk))
                temp_chunk = [sentences[i]]

    if temp_chunk:
        chunks.append(" ".join(temp_chunk))

    return chunks

# Example
text = "Machine learning is a fascinating field. Neural networks are powerful models. They can be used for deep learning."
chunks = semantic_chunking(text, 'all-MiniLM-L6-v2')
print(chunks)

6. Agent-Based Chunking

How It Works

Agent-based chunking dynamically determines the chunking strategy based on context, such as document type, user queries, or content complexity.

When to Use It

  • When different sections of a document require different chunking strategies.
  • When working with AI agents that dynamically adjust retrieval methods.

Implementation Concept

python

def agent_based_chunking(text, strategy):
    if strategy == "semantic":
        return semantic_chunking(text, 'all-MiniLM-L6-v2')
    elif strategy == "token":
        return token_based_chunking(text, 100)
    elif strategy == "recursive":
        return recursive_chunking(text, 200)
    else:
        return fixed_length_chunking(text, 500)

# Example
text = "This is a document requiring different chunking methods."
chunks = agent_based_chunking(text, "semantic")
print(chunks)

What is Text Overlapping?

Text overlapping occurs when multiple pieces of text overlap or share similar portions within a dataset, making it challenging to distinguish unique content. This issue is common in natural language processing (NLP), text mining, and machine learning tasks. Overlapping text can affect the accuracy of models, lead to redundancy in data representation, and sometimes create bias in training datasets.

Examples of Text Overlapping:

  • Duplicate or near-duplicate sentences in a dataset.
  • Repeated phrases or paragraphs across multiple documents.
  • Overlapping word sequences in generated text from models like GPT.
  • Similar textual content in summarization or paraphrasing tasks.

Why Does Text Overlapping Matter?

In various applications, text overlapping can have both advantages and drawbacks. In information retrieval, overlapping keywords can enhance search relevance. However, in AI-generated content, excessive overlapping may lead to redundancy, reducing the diversity of responses.

Key areas where text overlapping is crucial:

  1. Text Generation – Avoiding repetitive outputs while ensuring coherence.
  2. Text Summarization – Extracting distinct yet relevant content.
  3. Plagiarism Detection – Identifying content similarity across sources.
  4. Data Preprocessing – Ensuring high-quality datasets with minimal redundancy.

Additional Tips:

Choosing the right text chunking method depends on the use case:

  • Fixed-length chunking: When uniform chunk sizes are needed.
  • Sentence-based chunking: When preserving sentence integrity is important.
  • Token-based chunking: When working with LLM token limits.
  • Recursive chunking: When hierarchical segmentation is required.
  • Semantic chunking: When chunking must align with meaning and context.
  • Agent-based chunking: When an adaptive, context-aware approach is needed.

Each approach has trade-offs, and sometimes combining methods yields the best results. Try these techniques in your NLP workflows to improve efficiency and accuracy!