How Generation Works
Models generate text one token at a time, feeding each new token back in.
An LLM writes text one token at a time. It reads your prompt, predicts the most likely next token, appends it, then repeats — using the growing text as the new input. This is called autoregressive generation.
Because the model picks from a probability distribution over possible next tokens, the same prompt can produce different outputs — especially when sampling is turned up.
Example
{
"step_1": {"input": "The sky is", "top_next_tokens": {" blue": 0.62, " clear": 0.11, " grey": 0.08}},
"step_2": {"input": "The sky is blue", "top_next_tokens": {" today": 0.30, ".": 0.28, " and": 0.12}}
}When to use it
- A chat UI streams tokens to the screen one by one so users see text appear instead of waiting several seconds for the full reply.
- A game uses temperature and sampling to generate slightly different NPC dialogue each playthrough from the same prompt.
- A coding tool generates a whole function autoregressively from a comment, building valid syntax token by token.
More examples
Stream tokens as they arrive
Iterates over the token stream, printing each chunk as the model generates it.
from openai import OpenAI
import sys
client = OpenAI()
stream = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': 'Count from 1 to 5 slowly.'}],
stream=True
)
for chunk in stream:
text = chunk.choices[0].delta.content or ''
sys.stdout.write(text)
sys.stdout.flush()Greedy decoding simulation
Simulates greedy decoding — a deterministic strategy that always selects the most probable token.
vocab_probs = [
{'token': 'Paris', 'prob': 0.91},
{'token': 'Lyon', 'prob': 0.06},
{'token': 'Nice', 'prob': 0.03},
]
greedy = max(vocab_probs, key=lambda t: t['prob'])
print('Greedy pick:', greedy['token'])Manual autoregressive loop
Manually runs the autoregressive loop, appending one generated token to the context each iteration.
from openai import OpenAI
client = OpenAI()
context = 'Once upon a time'
for step in range(4):
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': context}],
max_tokens=1
)
context += r.choices[0].message.content
print(f'Step {step+1}: {context}')
Discussion