Tokens
LLMs read and write tokens — chunks of text roughly the size of a word piece.
LLMs do not see letters or whole words. They see tokens: common chunks of text. A token is often a word, part of a word, or punctuation.
Rough rule of thumb
- 1 token is about 4 characters of English.
- 100 tokens is about 75 words.
- Spaces and punctuation count too.
Why tokens matter
- Cost — you are billed per token, in and out.
- Limits — models have a maximum number of tokens they can handle at once (the context window).
- Speed — more tokens means slower responses.
Example
{
"text": "Tokenization isn't magic.",
"tokens": ["Token", "ization", " isn", "'t", " magic", "."],
"token_count": 6
}When to use it
- A SaaS product sets max_tokens=200 to keep AI summaries short and prevent unexpectedly large API bills.
- A developer notices a 10k-token prompt is slow and expensive, so they trim boilerplate to cut costs by 30%.
- A billing service counts tokens before each request to enforce per-user quotas without waiting for the API response.
More examples
Count tokens with tiktoken
Uses the tiktoken library to count tokens exactly as GPT models will see them.
import tiktoken
enc = tiktoken.encoding_for_model('gpt-4o')
text = 'Hello, world! How many tokens is this?'
tokens = enc.encode(text)
print(f'{len(tokens)} tokens')
print(tokens)Read token usage from API response
Reads the usage object from the response to see exactly how many tokens the request consumed.
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': 'Say hello in Spanish.'}]
)
u = response.usage
print(f'Prompt: {u.prompt_tokens} Completion: {u.completion_tokens} Total: {u.total_tokens}')Hard-cap output with max_tokens
Shows how max_tokens truncates the response and how to detect a cut-off via finish_reason='length'.
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': 'Explain quantum computing in detail.'}],
max_tokens=60
)
print(response.choices[0].message.content)
print('Finish reason:', response.choices[0].finish_reason)
Discussion