AI Text Chunker

What does the AI Text Chunker do?

The AI Text Chunker breaks large documents, prompts, Markdown files, and source code into model-friendly chunks optimized for Large Language Model (LLM) context windows, Retrieval-Augmented Generation (RAG) pipelines, and dense vector embeddings. It offers configurable token and character limits, semantic boundary preservation, and sliding window overlap to prevent context truncation.

Core Concepts

Understanding text chunking in modern AI and RAG architectures:

• Semantic Hierarchy Preservation: The recursive chunker splits text using natural structural hierarchy (\n\n for paragraphs → \n for code lines → . for sentences → for words) before falling back to character-level breaks. This ensures complete semantic thoughts stay intact.
• Context Window Sliding Overlap: When documents are split into discrete chunks, sentences at the seam can lose meaning. Sliding overlap retains trailing tokens from the preceding chunk into the subsequent chunk, ensuring vector search matches boundary terms.
• RAG & Embedding Sizing: Dense embedding models (e.g. OpenAI text-embedding-3, Voyage AI voyage-3, Cohere embed-v3) achieve maximum semantic fidelity when chunks range between 256 and 512 tokens.
• WebMCP Integration: Enables AI coding agents and browser extensions to partition large codebases or transcripts into manageable prompt payloads via local Model Context Protocol (MCP) tool execution.

How to use the tool?

1. Enter Document or Code: Paste your long text, documentation, or source code into the editor, click Upload, or click Sample.
2. Select Sizing Unit: Choose Tokens (BPE Subwords) to target LLM context window boundaries or Characters for fixed storage sizes.
3. Configure Chunk Size & Overlap: Set your target chunk size (e.g., 300 tokens) and sliding overlap (e.g., 30 tokens).
4. Choose Splitting Strategy: Select Recursive (Semantic) for articles, Sentence Grouping for chat transcripts, or Fixed Size Window for uniform batches.
5. Extract & Copy: Click Chunk Text to generate chunks with real-time token counts. Copy individual chunks or click Download JSON to export all partitioned chunks.

Context-Aware Practical Workflow Guides

IDE Shortcuts & RAG Indexing Prep

• VS Code: Use Ctrl+K Ctrl+0 (Cmd+K Cmd+0 on macOS) to fold code sections into logical blocks before chunking for API tool documentation.
• Python LangChain / LlamaIndex: Test your chunk boundary parameters visually in this tool before committing RecursiveCharacterTextSplitter(chunk_size=300, chunk_overlap=30) to production ingest pipelines.

Related Developer Utilities

• AI Token Counter: Calculate token totals, subword statistics, and prompt costs across leading LLMs.
• JSON Formatter: Validate and format structured JSON metadata for vector database insertion.
• YAML to JSON Converter: Convert YAML configuration files into JSON payloads.
• Text Diff Tool: Compare text alterations and chunk boundary alignments side-by-side.

WebMCP (Model Context Protocol) Integration

This tool natively exposes a WebMCP tool interface for AI agents, allowing AI assistants running in the browser to split long documents locally.

WebMCP Tool Definition

{
  "name": "ai_chunk_text",
  "description": "Splits text, documents, or code into model-friendly chunks using configurable token or character limits with sliding window overlap.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "rawText": {
        "type": "string",
        "description": "The input text, document, prompt, or code to split into chunks."
      },
      "chunkSize": {
        "type": "number",
        "default": 500,
        "description": "Maximum chunk size in tokens or characters."
      },
      "unit": {
        "type": "string",
        "enum": ["tokens", "characters"],
        "default": "tokens",
        "description": "Measurement unit: 'tokens' or 'characters'."
      },
      "overlap": {
        "type": "number",
        "default": 50,
        "description": "Sliding overlap between adjacent chunks."
      },
      "strategy": {
        "type": "string",
        "enum": ["recursive", "fixed", "sentence", "paragraph"],
        "default": "recursive",
        "description": "Chunking algorithm: 'recursive', 'fixed', 'sentence', or 'paragraph'."
      }
    },
    "required": ["rawText"]
  }
}

REST API Integration

blueutils.com provides a free REST API endpoint (POST https://blueutils.com/api/ai/text-chunker) for programmatic chunking.

API Request Parameters

API Request Payload Examples

cURL

curl -X POST https://blueutils.com/api/ai/text-chunker \
  -H "Content-Type: application/json" \
  -d '{
    "rawText": "Retrieval-Augmented Generation (RAG) is an architectural pattern that enhances the capabilities of Large Language Models...",
    "chunkSize": 300,
    "unit": "tokens",
    "overlap": 30,
    "strategy": "recursive"
  }'

Python

import requests

url = "https://blueutils.com/api/ai/text-chunker"
payload = {
    "rawText": "Retrieval-Augmented Generation (RAG) is an architectural pattern that enhances LLMs...",
    "chunkSize": 300,
    "unit": "tokens",
    "overlap": 30,
    "strategy": "recursive"
}
headers = {"Content-Type": "application/json"}
response = requests.post(url, json=payload, headers=headers)
print(response.json())

Java

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;

public class Main {
    public static void main(String[] args) throws Exception {
        String jsonPayload = """
            {
                "rawText": "Retrieval-Augmented Generation (RAG) is an architectural pattern...",
                "chunkSize": 300,
                "unit": "tokens",
                "overlap": 30,
                "strategy": "recursive"
            }
            """;

        HttpClient client = HttpClient.newHttpClient();
        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://blueutils.com/api/ai/text-chunker"))
            .header("Content-Type", "application/json")
            .POST(HttpRequest.BodyPublishers.ofString(jsonPayload))
            .build();

        HttpResponse<String> response = client.send(request, HttpResponse.BodyHandlers.ofString());
        System.out.println(response.body());
    }
}

API Response Parameters

API Response Payload Examples

Success Response (HTTP 200 OK)

{
  "isValid": true,
  "chunks": [
    {
      "id": 1,
      "text": "Retrieval-Augmented Generation (RAG) is an architectural pattern that enhances the capabilities of Large Language Models...",
      "characters": 1280,
      "tokens": 284
    },
    {
      "id": 2,
      "text": "Embedding Model Context Window Boundaries: Modern embedding models such as OpenAI text-embedding-3 perform best with chunks...",
      "characters": 1150,
      "tokens": 265
    }
  ],
  "stats": {
    "totalChunks": 2,
    "unit": "tokens",
    "chunkSize": 300,
    "overlap": 30,
    "strategy": "recursive",
    "totalTokens": 549,
    "totalCharacters": 2430,
    "avgChunkTokens": 275,
    "avgChunkCharacters": 1215
  }
}

Validation Failure Response (HTTP 400 Bad Request)

{
  "isValid": false,
  "error": "Invalid input: Text payload cannot be empty."
}

Rate Limit Exceeded Response (HTTP 429 Too Many Requests)

{
  "error": "API rate limit exceeded. Please wait or contact support@blueutils.com."
}

Why use an API to chunk text?

• Standardized Ingestion Pipelines: Enforce consistent token boundaries across multilingual documents in automated data processing microservices.
• Cost Reduction for Agent Workflows: Offload document pre-processing to an API to avoid consuming LLM completion tokens for basic text formatting.
• RAG Vector Accuracy: Consistent chunk lengths optimize embedding density and boost similarity scores in vector database retrieval.

Native Usage

How to split text into chunks using standard programming libraries without third-party frameworks:

Windows (PowerShell)

# Fixed-size chunking script in PowerShell
$text = Get-Content -Raw -Path "document.txt"
$chunkSize = 1000
for ($i = 0; $i -lt $text.Length; $i += $chunkSize) {
    $length = [Math]::Min($chunkSize, $text.Length - $i)
    $chunk = $text.Substring($i, $length)
    Write-Output "--- Chunk $(($i/$chunkSize)+1) ---"
    Write-Output $chunk
}

Linux / Unix (Bash)

# Split document into 50-line chunks using coreutils
split -l 50 document.txt chunk_ --additional-suffix=.txt

Python

Using Python standard library:

def chunk_text_by_words(text, max_words=200, overlap_words=20):
    words = text.split()
    chunks = []
    step = max(1, max_words - overlap_words)
    for i in range(0, len(words), step):
        chunk = " ".join(words[i:i + max_words])
        chunks.append(chunk)
    return chunks

with open("document.txt", "r", encoding="utf-8") as f:
    content = f.read()

chunks = chunk_text_by_words(content, max_words=250, overlap_words=25)
print(f"Generated {len(chunks)} chunks.")

Java

Using Java standard library:

import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;

public class TextChunker {
    public static List<String> chunkByCharacters(String text, int chunkSize, int overlap) {
        List<String> chunks = new ArrayList<>();
        int step = Math.max(1, chunkSize - overlap);
        for (int i = 0; i < text.length(); i += step) {
            int end = Math.min(text.length(), i + chunkSize);
            chunks.add(text.substring(i, end));
        }
        return chunks;
    }

    public static void main(String[] args) throws Exception {
        String content = Files.readString(Path.of("document.txt"));
        List<String> chunks = chunkByCharacters(content, 1000, 100);
        System.out.println("Generated " + chunks.size() + " chunks.");
    }
}