Streaming UX Best Practices
Streaming makes an app feel instant — but only if you handle partial text, errors, and cancellation gracefully.
Streaming turns a frozen spinner into live, word-by-word output. The perceived speed win is huge, but streaming shifts complexity onto the client, and a few details separate a polished experience from a janky one.
Show progress immediately
Render the first token the instant it arrives. Even a blinking cursor at the start tells the user the system is working. Time-to-first-token, not total time, is what users feel.
Handle partial and structured output carefully
Mid-stream text is incomplete by definition. If you are streaming JSON, do not try to parse it on every chunk — wait for the stream to finish, then parse and validate the whole object. For prose, buffer a few tokens before rendering so the UI isn't thrashed by single-character updates.
Make it cancellable
Users change their mind. A visible "stop" button that aborts the request saves tokens (you stop paying for output you'll never show) and feels respectful. Abort the underlying connection, don't just hide the text.
Fail gracefully mid-stream
A connection can drop after ten tokens. Keep what you've received, show a clear "response interrupted" state, and offer a retry — never leave a half-sentence with no indication anything went wrong.
Streaming changes how fast an app feels, not the total generation time or the token cost. It is a UX technique, not a performance optimization — reach for it to make waiting pleasant, not to make requests cheaper.
Example
async function streamAnswer(prompt, { onToken, signal }) {
const res = await fetch("/api/chat", {
method: "POST",
body: JSON.stringify({ prompt, stream: true }),
signal, // wire a 'Stop' button to an AbortController -> signal
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
let full = "";
try {
while (true) {
const { done, value } = await reader.read();
if (done) break;
const chunk = decoder.decode(value);
full += chunk;
onToken(chunk); // render immediately for perceived speed
}
} catch (err) {
if (err.name === "AbortError") return { full, aborted: true };
// Keep partial text, show an 'interrupted — retry?' state.
return { full, error: true };
}
return { full, done: true }; // now it is safe to parse/validate `full`
}When to use it
- A chat interface renders tokens as they arrive so users see the reply build word by word, reducing perceived wait time from 4 seconds to near-instant.
- An editor tool shows a streaming cancel button so users can stop a long generation mid-way if the direction is wrong, saving API cost.
- A log-analysis tool displays partial results as the LLM processes each log line, so operators can start acting on early findings immediately.
More examples
Stream and buffer by sentence
Buffers streamed tokens and flushes complete sentences to the output, avoiding half-rendered mid-sentence UI flashes.
from openai import OpenAI
import sys
client = OpenAI()
buffer = ''
stream = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role':'user','content':'Write three sentences about the ocean.'}],
stream=True
)
for chunk in stream:
token = chunk.choices[0].delta.content or ''
buffer += token
if '.' in buffer: # flush on sentence boundary
sentences = buffer.split('.')
for s in sentences[:-1]:
print(s.strip() + '.', flush=True)
buffer = sentences[-1]
if buffer.strip():
print(buffer.strip())Stream with stop-button simulation
Closes the stream after a character limit to simulate a user clicking a Stop button mid-generation.
from openai import OpenAI
import sys
client = OpenAI()
MAX_CHARS = 150 # simulate user pressing Stop after 150 characters
chars = 0
stream = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role':'user','content':'Explain quantum entanglement in detail.'}],
stream=True
)
for chunk in stream:
token = chunk.choices[0].delta.content or ''
sys.stdout.write(token)
sys.stdout.flush()
chars += len(token)
if chars >= MAX_CHARS:
print('\n[Generation stopped by user]')
stream.close()
breakStream to a Server-Sent Event endpoint
Wraps the OpenAI stream in a Flask SSE endpoint so a browser EventSource can receive tokens in real time.
from flask import Flask, Response, stream_with_context
from openai import OpenAI
app = Flask(__name__)
client = OpenAI()
@app.route('/stream')
def stream_endpoint():
def generate():
stream = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role':'user','content':'Tell me a short story.'}],
stream=True
)
for chunk in stream:
token = chunk.choices[0].delta.content or ''
if token:
yield f'data: {token}\n\n'
yield 'data: [DONE]\n\n'
return Response(stream_with_context(generate()), mimetype='text/event-stream')
Discussion