Capabilities, Limits & Hallucinations
What LLMs are great at, where they fail, and why they sometimes make things up.
Knowing the sharp edges of LLMs is what separates a reliable AI feature from a demo that embarrasses you in production.
Good at
- Summarising, rewriting, translating, and classifying text.
- Drafting and explaining code.
- Extracting structured data from messy input.
- Answering questions when given the relevant source material.
Weak at
- Facts outside their training data — they have a knowledge cutoff and do not know recent or private information unless you supply it.
- Exact arithmetic and counting.
- Anything requiring guaranteed correctness without checks.
Hallucinations
A hallucination is when a model states something false with total confidence — an invented citation, a fake API method, a made-up date. It happens because the model generates plausible text, not verified text.
Example
{
"prompt": "What is the population of the city of Xqzzt?",
"risky_answer": "Xqzzt has about 2.3 million residents.",
"problem": "The city may not exist. The model produced a confident but fabricated number."
}When to use it
- A product team adds a disclaimer to their AI chatbot warning users to verify medical advice, because the model can hallucinate drug names.
- A search engine uses an LLM to rewrite query results in plain language while a traditional database handles the actual lookup.
- A content moderation pipeline uses an LLM to catch nuanced hate speech that keyword lists miss, but routes edge cases to human reviewers.
More examples
Ground model on provided facts only
Grounds the model on a supplied fact so it cannot hallucinate an invented location.
from openai import OpenAI
client = OpenAI()
fact = 'The Eiffel Tower is located in Paris, France.'
prompt = f'Based ONLY on this fact: "{fact}"\nWhere is the Eiffel Tower?'
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': prompt}]
)
print(r.choices[0].message.content)Instruct model to admit uncertainty
Uses the system prompt to instruct the model to acknowledge uncertainty instead of confabulating.
from openai import OpenAI
client = OpenAI()
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[
{'role': 'system', 'content': 'If you are not certain, say "I don\'t know" rather than guessing.'},
{'role': 'user', 'content': 'What was the exact population of Rome in 100 AD?'}
]
)
print(r.choices[0].message.content)Restrict AI to a safe task whitelist
Restricts model usage to a whitelist of well-defined, low-hallucination-risk tasks.
from openai import OpenAI
ALLOWED = {'summarise', 'translate', 'classify', 'rewrite'}
def safe_call(task, text):
if task not in ALLOWED:
raise ValueError(f'Task "{task}" not permitted')
client = OpenAI()
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': f'{task.capitalize()}: {text}'}]
)
return r.choices[0].message.content
print(safe_call('summarise', 'Long article text...'))
Discussion