Text chunker for RAG with overlap
Split long documents into overlapping chunks that end on paragraph or sentence boundaries, so retrieval doesn't return half-sentences. No dependencies beyond the standard library.
import re
from dataclasses import dataclass
@dataclass(frozen=True, slots=True)
class Chunk:
index: int
text: str
start: int # character offset in the normalised text
# Preferred split points, strongest first
_BREAKS = ("\n\n", "\n", ". ", "? ", "! ", "; ", ", ", " ")
def _best_break(text: str, start: int, end: int) -> int:
"""Return the last natural boundary in the back half of the window."""
floor = start + (end - start) // 2
for sep in _BREAKS:
pos = text.rfind(sep, floor, end)
if pos != -1:
return pos + len(sep)
return end # no boundary found: hard cut
def chunk_text(text: str, max_chars: int = 1200, overlap: int = 200) -> list[Chunk]:
if not 0 <= overlap < max_chars:
raise ValueError("overlap must be >= 0 and smaller than max_chars")
text = re.sub(r"[ \t]+", " ", text).strip()
chunks: list[Chunk] = []
start = 0
while start < len(text):
end = min(start + max_chars, len(text))
if end < len(text):
end = _best_break(text, start, end)
piece = text[start:end].strip()
if piece:
chunks.append(Chunk(index=len(chunks), text=piece, start=start))
if end >= len(text):
break
# Step back by the overlap, but always move forward
next_start = max(end - overlap, start + 1)
# Begin the next chunk on a word boundary
space = text.find(" ", next_start, end)
start = space + 1 if space != -1 else next_start
return chunksUsage
from pathlib import Path
from chunker import chunk_text
doc = Path("handbook.md").read_text(encoding="utf-8")
for chunk in chunk_text(doc, max_chars=1000, overlap=150):
print(chunk.index, chunk.start, len(chunk.text))
# embed chunk.text and store it with the source id and chunk.startHow it works & gotchas
- 01Sizes are in characters, not tokens. Roughly 4 characters per token in English is a safe starting estimate; use your tokenizer if you need hard limits.
- 02The splitter only looks for a boundary in the back half of each window, so chunks never shrink below about half of
max_chars. - 03Overlap keeps a sentence that straddles a boundary retrievable from both sides. 10–20% of the chunk size is usually enough.
- 04
startis an offset into the whitespace-normalised text. Store it if you want to highlight the source passage later. - 05For Markdown or code, split on headings or functions first and run this inside each section. Structure-aware splits retrieve better.
Need this wired into a real system?
We build and harden production software end to end.
More code helpers
Stream LLM responses from a Next.js route handler
Send tokens to the browser as the model produces them instead of making users wait for the full completion. Uses a plain ReadableStream, so it works with any client that can read a fetch body.
Retry LLM calls with exponential backoff and jitter
LLM APIs return 429s and 5xx errors under load. This wrapper retries only the failures worth retrying, honours Retry-After and spreads retries out so many workers don't hit the API in lockstep.
Redis sliding-window rate limiter
Limit requests per user, API key or IP across every instance of your service. A single Lua script keeps the check-and-increment atomic, so concurrent requests can't slip past the limit.