Streaming Responses
Stream tokens as they are generated so the UI feels instant.
By default you wait for the whole reply before you get anything back. With streaming, the API sends tokens as they are produced, so text appears word by word — like a live typing effect.
Why stream
- Perceived speed — users see output immediately.
- Long replies feel responsive instead of frozen.
- You can stop early if the answer is already good.
Technically the server sends a stream of small chunks (often Server-Sent Events). Your code appends each chunk's text as it arrives.
Example
const res = await fetch('https://api.example-llm.com/v1/chat', {
method: 'POST',
headers: { 'Authorization': `Bearer ${process.env.API_KEY}`, 'Content-Type': 'application/json' },
body: JSON.stringify({ model: 'llm-medium', stream: true, messages: [{ role: 'user', content: 'Write a haiku.' }] })
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
while (true) {
const { done, value } = await reader.read();
if (done) break;
process.stdout.write(decoder.decode(value)); // print each chunk as it streams in
}When to use it
- A chat interface renders each word as it is generated so the user sees a typing effect instead of a blank screen for 5 seconds.
- A document editor streams the AI's rewrite of a long paragraph, letting the user start reading before the full text is ready.
- A voice assistant streams text tokens into a text-to-speech pipeline so audio playback starts before generation finishes.
More examples
Enable streaming in Python SDK
Uses the SDK's stream context manager to iterate over text chunks as they arrive.
from openai import OpenAI
import sys
client = OpenAI()
with client.chat.completions.stream(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': 'Write a haiku about the ocean.'}]
) as stream:
for text in stream.text_stream:
sys.stdout.write(text)
sys.stdout.flush()Stream with raw chunk loop
Iterates raw chunks, checking choices[0].delta.content for the incremental text.
from openai import OpenAI
client = OpenAI()
stream = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': 'List five planets.'}],
stream=True
)
for chunk in stream:
delta = chunk.choices[0].delta.content
if delta:
print(delta, end='', flush=True)
print() # newline at endServer-Sent Events streaming in Node.js
Shows the async for-await loop in Node.js to consume each streaming chunk in order.
import OpenAI from 'openai';
const client = new OpenAI();
const stream = await client.chat.completions.create({
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: 'List five planets.' }],
stream: true
});
for await (const chunk of stream) {
const text = chunk.choices[0]?.delta?.content ?? '';
process.stdout.write(text);
}
Discussion