Datadog Agent Observability (formerly LLM Observability)
Datadog's AI agent and LLM monitoring with tracing, evals, experiments and datasets, linked to your full Datadog stack
Overview
Datadog's product for monitoring LLM apps and AI agents, now called Agent Observability (previously LLM Observability), traces every step of an agent run: LLM calls, tool calls, retrieval and workflow steps, with latency, token usage, cost and errors. It adds built-in and custom evaluations, human review and annotation, versioned datasets made from production traces, and experiments that compare prompts, models and configurations side by side.
The main reason to choose it is integration: agent traces connect to the rest of Datadog, so you can follow a slow answer from the user's browser session through your backend services and into the model call, in the same tool your SRE team already uses. Datadog moved to a simpler pricing model with a free tier and per-LLM-span billing, where tool, workflow and retrieval spans are free.
The Verdict
Who Should Use Datadog Agent Observability?
Best For
- Companies already standardized on Datadog
- Correlating agent behavior with backend and frontend performance
- Platform and SRE teams owning AI in production
- Enterprises that want one vendor for all monitoring
Not Ideal For
- Startups not using Datadog for anything else
- Teams that need self-hosting or open source
- Very high LLM call volumes on a tight budget
What's Great
- End-to-end correlation with APM, logs and real user monitoring
- Free tier and billing only on LLM spans
- Evaluations, datasets and experiments in one place
- Supports OpenAI, Anthropic, Gemini, Bedrock and major frameworks
- Enterprise-grade security and compliance
Watch Out For
- Datadog bills add up across products
- Less specialized eval workflows than eval-first tools
- No self-hosting
- Retention beyond 15 days is a paid add-on
Pricing
View all features & details
Key Features
- Full-stack agent tracing
- Latency, token, cost and error monitoring
- Built-in and custom evaluations
- Datasets from production traces
- Experiments comparing prompts and models
- Human review and annotation
Integrations
- OpenAI, Anthropic, Gemini, Vertex AI, Bedrock
- LangChain, CrewAI, Pydantic AI, Strands Agents
- Python, Node.js and Java SDKs
- OpenTelemetry and HTTP API
How It Compares
| Feature | Datadog | Langfuse | Braintrust | Arize AX |
|---|---|---|---|---|
| Full-stack APM correlation | Yes | No | No | No |
| Free tier | 40K LLM spans | 50K units | 1 GB data | 25K spans |
| Self-hosting | No | Yes | Enterprise | Enterprise |
| Eval depth | Good | Good | Best | Good |
Is this your tool? Add a badge