Safety & Moderation
Screen inputs and outputs to keep your AI feature safe and on-brand.
When users can send anything and the model can say anything, you need safety layers around your feature.
Input side
- Run user input through a moderation check to catch abusive or disallowed content.
- Rate-limit and authenticate to prevent abuse of your API budget.
Output side
- Moderate or filter model output before showing it.
- Constrain scope in the system prompt ("only answer questions about our product").
- Verify factual or action-triggering output before acting on it.
Example
const check = await moderate(userInput);
if (check.flagged) {
return 'Sorry, I can\'t help with that request.';
}
const reply = await chat({ messages });
const outCheck = await moderate(reply.content);
return outCheck.flagged ? 'I can\'t provide that.' : reply.content;When to use it
- A children's education platform runs every user message through the moderation API before it reaches the LLM to block inappropriate content.
- A public chatbot screens AI-generated replies before display to catch policy violations the model might produce despite a strong system prompt.
- A UGC platform runs both input and output moderation to satisfy legal requirements and prevent the model from amplifying harmful content submitted by users.
More examples
Moderate user input before sending to LLM
Runs the user message through the moderation API and blocks it before it reaches the LLM if flagged.
from openai import OpenAI
client = OpenAI()
def safe_chat(user_message):
mod = client.moderations.create(input=user_message)
if mod.results[0].flagged:
return 'Your message was flagged and could not be processed.'
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role':'user','content':user_message}]
)
return r.choices[0].message.content
print(safe_chat('Tell me about photosynthesis.'))Check moderation category scores
Extracts which specific moderation categories are triggered so you can log or handle each one separately.
from openai import OpenAI
client = OpenAI()
def get_moderation_detail(text):
mod = client.moderations.create(input=text)
result = mod.results[0]
flagged_categories = [
cat for cat, val in result.categories.__dict__.items() if val
]
return {
'flagged': result.flagged,
'categories': flagged_categories
}
print(get_moderation_detail('I love learning about science.'))Moderate both input and output
Applies moderation to both the incoming user message and the outgoing model reply for double-layer safety.
from openai import OpenAI
client = OpenAI()
def moderate(text):
return client.moderations.create(input=text).results[0].flagged
def safe_pipeline(user_input):
if moderate(user_input):
return 'Input rejected.'
reply = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role':'user','content':user_input}]
).choices[0].message.content
if moderate(reply):
return 'Output rejected by moderation.'
return reply
print(safe_pipeline('What are some fun science experiments for kids?'))
Discussion