Traceloop OpenLLMetry
Open-source OpenTelemetry instrumentation for LLM apps (OpenLLMetry) plus Traceloop's evaluation platform, now part of ServiceNow
Overview
OpenLLMetry is an open-source set of OpenTelemetry instrumentations for LLM applications. With a line of setup it traces calls to more than 20 model providers, vector databases and frameworks, and because the output is standard OpenTelemetry, you can send it to whatever backend you already use: Datadog, Grafana, Honeycomb, New Relic or Traceloop's own platform. SDKs exist for Python, TypeScript, Go and Ruby.
Traceloop, the Israeli Y Combinator company behind it, adds a commercial platform on top with monitoring, built-in quality evaluators (faithfulness, relevance, safety), custom evaluators and quality gates for CI. Traceloop raised a $6.1M seed in 2025 and has since been acquired by ServiceNow. OpenLLMetry remains Apache 2.0 licensed on GitHub.
The Verdict
Who Should Use OpenLLMetry?
Best For
- Teams that already run OpenTelemetry
- Stacks with several model providers and frameworks
- Avoiding lock-in to one observability vendor
- ServiceNow customers building AI agents
Not Ideal For
- Teams wanting one all-in-one LLM platform with prompt management
- Buyers wary of roadmap changes after an acquisition
- Teams with no observability backend at all
What's Great
- Built on the OpenTelemetry standard, so data goes anywhere
- Broad coverage of providers, vector DBs and frameworks
- Simple setup in Python, TypeScript, Go and Ruby
- Built-in evaluators and CI quality gates in the platform
- SOC 2 and HIPAA support on the commercial platform
Watch Out For
- ServiceNow acquisition may shift the commercial roadmap
- Instrumentation only; you need a backend to view data
- Fewer prompt-management features than LangSmith or Langfuse
- Limited public reviews
Pricing
View all features & details
Key Features
- OpenTelemetry instrumentation for LLM calls
- Prompts, completions, tokens and latency in traces
- Built-in quality evaluators
- Custom evaluators and CI quality gates
- Drift and regression monitoring
Integrations
- OpenAI, Anthropic, Gemini, Bedrock, Ollama and more
- Pinecone, Chroma and other vector DBs
- LangChain, LlamaIndex, CrewAI
- Datadog, Grafana, Honeycomb, New Relic backends
How It Compares
| Feature | OpenLLMetry | OpenInference (Arize) | Langfuse | LangSmith |
|---|---|---|---|---|
| OpenTelemetry native | Yes | Yes | Accepts OTel | Partial |
| Open source | Yes (Apache 2.0) | Yes | Yes (MIT) | No |
| Backend choice | Any OTel backend | Phoenix or any | Langfuse | LangSmith |
| Languages | Python, TS, Go, Ruby | Python, TS, Java | Python, TS | Python, TS |
Is this your tool? Add a badge