Skip to main content
This analyzer evaluates text against safety policies for dangerous content, harassment, hate speech, and sexually explicit material. It uses Google’s ShieldGemma family of safety classifiers as an LLM-as-a-judge, served through vLLM on the internal model service.

​
What it detects

Default categories (configurable per safety policy):
  • Dangerous content — instructions for violence, self-harm, dangerous medical practice, illegal activity.
  • Harassment — bullying, intimidation, doxxing.
  • Hate speech — discrimination, slurs, dehumanizing content.
  • Sexually explicit content — explicit material, sexual content involving minors.
Additional categories shipped in the default safety policy: misinformation, privacy violations, illegal content. You can author a custom safety policy to redefine the category set; see Safety policies.

​
How it works

For each enabled category, Agnes constructs a per-category prompt using your chosen safety policy and the ShieldGemma chat template:
ShieldGemma replies with a Yes / No plus a confidence score. Agnes parses the verdict, normalizes it to a confidence number, and reports the per-category breakdown. Content is flagged as unsafe if any category scores >= 0.5.

​
Available models

Selected with the model_id parameter:

​
Parameters

​
Outputs and metrics

​
Termination signals

Combine with max_violation_score for stricter control. Example: only terminate when is_safe == false AND max_violation_score >= 0.8.

​
Limits and cost

Cost varies by model size; the catalog notes “model inference cost”. Expect 9B to be roughly 4× the 2B and 27B to be roughly 12×.

​
Typical latency

50–200 ms depending on model and number of categories. Each enabled category runs an independent ShieldGemma forward pass.

​
When to use it

  • Best on outbound. This is the canonical “did my LLM produce something unsafe?” guardrail. Pair it with a strict safety policy on outbound, a permissive one on inbound.
  • Skip on highly templated outputs. If you fully control the model output (e.g. JSON schema, structured tools), the safety judge is often redundant — a YARA / regex check is enough.
  • Pick the smallest model that meets your accuracy bar. Most teams ship on shieldgemma-2b and only escalate to 9b for explicit high-risk surfaces.

​
Failure modes

  • Model service unavailable → analyzer_unavailable 503 with Retry-After. SDKs retry automatically.
  • Prompt longer than the model’s limit → the analyzer returns an error in its result. The prompt is not truncated automatically.

​
Next