A new API for content moderation has been released by AI startup Mistral. The new tool, which currently powers Mistral’s Le Chat chatbot, helps detect and filter potentially harmful content. It uses a specialized AI model called Ministral 8B that can work with multiple languages including English, French, and German.
The API checks content across nine different categories:
- Sexual content
- Hate speech and discrimination
- Violence and threats
- Dangerous and criminal content
- Self-harm
- Health-related content
- Financial content
- Legal content
- Personal information
Developers can use this tool to check both regular text and conversation-style content. Mistral says their system can be customized to match specific safety requirements and applications.
While AI-powered moderation tools can be helpful, they do have some challenges. For example, these systems sometimes incorrectly flag certain dialects like African American Vernacular English as inappropriate. They can also mistakenly mark posts about disabilities as negative content.
Mistral acknowledges that while their moderation system is accurate, it’s still being improved. The company hasn’t shared how their tool compares to similar services from companies like Jigsaw and OpenAI.
Additionally, Mistral announced a new batch processing feature that can reduce API costs by 25% when handling large volumes of requests. This feature is similar to what other AI companies like Anthropic, OpenAI, and Google offer.
