Choosing a Model
Balance capability, speed, and cost — and match the model to the task.
Providers offer a range of models. Bigger models are smarter but slower and pricier; smaller models are fast and cheap but less capable. Choosing well is a core engineering decision.
Questions to ask
- How hard is the task? Classification needs far less than open-ended reasoning.
- How fast must it respond?
- What is the budget per request at your volume?
- How big a context window do you need?
A common pattern
Use a small model for easy, high-volume steps and escalate to a large model only for the hard part.
Example
{
"route": {
"classify_intent": "llm-small",
"draft_reply": "llm-medium",
"complex_reasoning": "llm-large"
}
}When to use it
- A startup runs all classification tasks on gpt-4o-mini to keep costs low, reserving gpt-4o only for complex reasoning tasks that failed on the smaller model.
- A latency-sensitive autocomplete feature chooses a smaller, faster model because users expect sub-200ms responses even at the expense of some quality.
- A legal analysis tool uses the largest available model because accuracy on edge-case contract clauses matters more than cost or speed.
More examples
Route tasks to different model tiers
Routes simple classification to a cheap fast model and complex analysis to a larger one.
from openai import OpenAI
client = OpenAI()
def choose_model(task_type):
if task_type == 'classification':
return 'gpt-4o-mini' # fast, cheap
elif task_type == 'analysis':
return 'gpt-4o' # more capable
return 'gpt-4o-mini' # default
for task in ['classification', 'analysis']:
model = choose_model(task)
r = client.chat.completions.create(
model=model,
messages=[{'role':'user','content':f'Perform {task} on: "I love this product"'}],
max_tokens=50
)
print(f'{task} [{model}]: {r.choices[0].message.content.strip()}')Benchmark two models on accuracy
Runs the same test set on two models to measure accuracy differences before committing to one.
from openai import OpenAI
client = OpenAI()
test_cases = [
{'input': 'Is 997 a prime number?', 'expected': 'yes'},
{'input': 'Is 100 a prime number?', 'expected': 'no'},
]
for model in ['gpt-4o-mini', 'gpt-4o']:
correct = 0
for tc in test_cases:
r = client.chat.completions.create(
model=model,
messages=[{'role':'user','content':tc['input'] + ' Answer yes or no only.'}]
)
if tc['expected'] in r.choices[0].message.content.lower():
correct += 1
print(f'{model}: {correct}/{len(test_cases)} correct')Measure latency per model
Times each model on the same prompt to compare latency before choosing one for a latency-sensitive feature.
import time
from openai import OpenAI
client = OpenAI()
for model in ['gpt-4o-mini', 'gpt-4o']:
start = time.time()
client.chat.completions.create(
model=model,
messages=[{'role':'user','content':'Say hello.'}],
max_tokens=10
)
elapsed = time.time() - start
print(f'{model}: {elapsed:.2f}s')
Discussion