LLM Safety · Monitoring

LLM Misuse Monitor

Self-hostable, async monitoring for a served LLM: is our model being misused or misbehaving? It observes and flags for human review and never blocks inline.

50 tests 5 detector types Docker Compose deploy
View on GitHub

Problem

Teams self-hosting an LLM rarely see how it's actually used. Single-message filters miss slow-burn misuse, where a user probes across many sessions, and they say nothing about outbound failures like PII or system-prompt leakage. The goal was a monitor that sits beside the model, adds nothing to request latency, and turns raw traffic into a short review queue.

  • Inbound: jailbreaks, prompt injection, cross-session probing.
  • Outbound: harmful content, PII leakage, system-prompt leaks.

Approach

FastAPI /chat proxy app traffic → Ollama (monitored model) Capture events + logprobs, off the request path Detector orchestrator tier-0 · Llama Guard 3 · Presidio · LLM Guard · logprobs Risk scoring weights × severity, exponential decay, slow-burn multiplier Cases open → reviewing → resolved, in the Streamlit triage dashboard
  • Capture off the critical path. Prompts and completions are stored, then scanned in a background task, so monitoring never slows a response.
  • Pluggable detectors. A dependency-free tier-0 (jailbreak keywords, PII regex, system-prompt leak), Llama Guard 3 on the same Ollama instance, Presidio NER for PII, an LLM Guard DeBERTa prompt-injection classifier, and a logprob-anomaly tier (perplexity, top-k entropy, distribution shift). Each detector declares the telemetry it needs and is skipped when that telemetry is absent.
  • Explainable risk scores. Per event, points = Σ detector_weight × severity_weight. Per user and session, scores decay exponentially (2 h session half-life, 168 h user), with a repeat-offense multiplier, normalized to 0–100: watch ≥ 40, flagged ≥ 70. Every update is logged, so every score can be explained.
  • Triage. Crossing the threshold, or a single high-severity hit, opens a case: open → reviewing → resolved_benign | resolved_abuse.

Results

  • 50 tests; detector tests that need optional extras skip cleanly when deps are missing.
  • One-command deploy with Docker Compose: a slim image, or all detectors baked in.
  • POST /ingest/events lets existing traffic be backfilled or integrated without proxying through the service.

What's next

Langfuse / Phoenix / OTel capture adapters, a Postgres backend, and a white-box tier: activations captured at the serving layer (vLLM hidden states) feeding linear-probe detectors. The event model already carries an internal_telemetry_ref, so that tier plugs in without changes to the core.

Stack

PythonFastAPIOllamaLlama Guard 3PresidioLLM GuardSQLiteStreamlitDocker