AI & LLMsTypeScript

Stream LLM responses from a Next.js route handler

Send tokens to the browser as the model produces them instead of making users wait for the full completion. Uses a plain ReadableStream, so it works with any client that can read a fetch body.

#streaming#nextjs#openai#llm

app/api/chat/route.ts

import OpenAI from "openai";

const client = new OpenAI(); // reads OPENAI_API_KEY

type ChatMessage = { role: "system" | "user" | "assistant"; content: string };

export async function POST(req: Request) {
  const { messages } = (await req.json()) as { messages: ChatMessage[] };

  const completion = await client.chat.completions.create(
    {
      model: process.env.OPENAI_MODEL ?? "gpt-4o-mini",
      messages,
      stream: true,
    },
    // Stop generating (and paying for) tokens if the browser disconnects
    { signal: req.signal },
  );

  const encoder = new TextEncoder();

  const stream = new ReadableStream<Uint8Array>({
    async start(controller) {
      try {
        for await (const chunk of completion) {
          const text = chunk.choices[0]?.delta?.content;
          if (text) controller.enqueue(encoder.encode(text));
        }
        controller.close();
      } catch (err) {
        controller.error(err);
      }
    },
    cancel() {
      completion.controller.abort();
    },
  });

  return new Response(stream, {
    headers: {
      "Content-Type": "text/plain; charset=utf-8",
      "Cache-Control": "no-cache, no-transform",
      // Tell nginx and similar proxies not to buffer the response
      "X-Accel-Buffering": "no",
    },
  });
}

Usage

TypeScript

export async function streamChat(
  messages: { role: "user" | "assistant"; content: string }[],
  onToken: (text: string) => void,
  signal?: AbortSignal,
) {
  const res = await fetch("/api/chat", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ messages }),
    signal,
  });
  if (!res.ok || !res.body) throw new Error(`Chat failed: ${res.status}`);

  const reader = res.body.pipeThrough(new TextDecoderStream()).getReader();
  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    onToken(value);
  }
}

How it works & gotchas

01The handler streams plain text, not SSE. That keeps the client simple; switch to text/event-stream if you need to send structured events such as tool calls or usage.
02Passing req.signal to the SDK means a closed tab cancels the upstream request, so you stop paying for tokens nobody reads.
03X-Accel-Buffering: no matters behind nginx. Without it the proxy can hold the whole response and the stream arrives in one burst.
04Validate and cap messages before forwarding them. This example trusts the request body to keep the focus on streaming.

Need this wired into a real system?

We build and harden production software end to end.

Talk to an engineer

More code helpers

TypeScript

Retry LLM calls with exponential backoff and jitter

LLM APIs return 429s and 5xx errors under load. This wrapper retries only the failures worth retrying, honours Retry-After and spreads retries out so many workers don't hit the API in lockstep.

Python

Text chunker for RAG with overlap

Split long documents into overlapping chunks that end on paragraph or sentence boundaries, so retrieval doesn't return half-sentences. No dependencies beyond the standard library.

TypeScript

Redis sliding-window rate limiter

Limit requests per user, API key or IP across every instance of your service. A single Lua script keeps the check-and-increment atomic, so concurrent requests can't slip past the limit.