Skip to content

LLMTrace

Estimated time to read: 3 minutes

Zero-code LLM observability and security for production.

LLMTrace is a transparent proxy that sits between your application and any OpenAI-compatible API. Point your SDK at http://localhost:8080/v1 and you instantly get prompt-injection detection, PII scanning, cost accounting, latency/TTFT metrics, and a dashboard — without changing a single line of application code.

Install Quickstart View on GitHub

Why LLMTrace

Production LLM applications have three blind spots that existing APM tools don't cover:

Security: — prompt injection, jailbreaks, PII leakage, data exfiltration via tool calls.

Cost: — per-agent budgets, rate limits, anomaly detection on token spend.

Performance: — latency, streaming TTFT, error rates, provider flakiness.

LLMTrace solves these transparently. Nothing about your application code changes.

Use cases

Your codebase already calls OpenAI. Change the base_url, restart, and every request is now scanned for prompt injection, PII, and policy violations.

import openai
client = openai.OpenAI(base_url="http://localhost:8080/v1")

See: OpenAI SDK integration, Pre-request enforcement.

Isolate traces, budgets, and policies per API key or tenant header. Enforce per-tenant rate limits and block leaks from one customer reaching another.

See: Auth & multi-tenancy, Custom policies.

Per-agent budgets, rate limiters, and Prometheus metrics feed Grafana dashboards and Slack/PagerDuty alerts when spend or latency crosses a threshold.

See: Monitoring & observability, Dashboard.

Agent frameworks fan out to many provider calls. LLMTrace stitches them into a single trace with per-step security verdicts.

See: LangChain integration.

Run benchmarks against the security ensemble locally. Reproduce OWASP LLM Top 10 coverage numbers, tune thresholds, measure false-positive rate.

See: Benchmark methodology, Threshold tuning.

How it works

flowchart LR
    A["Your app"] -->|OpenAI-compatible request| B["LLMTrace Proxy"]
    B --> C{"Security Ensemble<br/>regex · DeBERTa · InjecGuard · PIGuard"}
    C -->|allow| D["LLM Provider"]
    C -->|block / flag| E["Policy Action"]
    D --> B
    B -->|trace + metrics| F[("Storage<br/>SQLite · PG · ClickHouse")]
    B --> G["Dashboard / REST API"]

Deep-dive: System architecture · Transparent proxy mechanics · LLM-as-judge.

Security performance

Tested on a 153-sample adversarial corpus across 12 attack categories (CyberSecEval2, BIPIA, TensorTrust, InjecAgent).

Metric Value
Accuracy 87.6%
Precision 95.5%
F1 Score 86.9%
Recall 79.7%

Methodology and reproduction: Benchmark methodology.

Install

cargo install llmtrace
llmtrace-proxy --config config.yaml

docker pull ghcr.io/epappas/llmtrace-proxy:latest
docker run -p 8080:8080 ghcr.io/epappas/llmtrace-proxy:latest

curl -o compose.yaml https://raw.githubusercontent.com/epappas/llmtrace/main/compose.yaml
docker compose up -d

helm install llmtrace ./deployments/helm/llmtrace

pip install llmtracing

Full details: Installation guide.

Where to next

Operators: — Installation → Configuration → Kubernetes → Monitoring.

Integrators: — Quickstart → OpenAI SDK → REST API.

Security engineers: — Ensemble detection → OWASP LLM Top 10 coverage → Custom policies.

Researchers / contributors: — Architecture → Research library → Contributing.