Traceloop OpenLLMetry iconTraceloop OpenLLMetry

Open-source OpenTelemetry instrumentation for LLM apps (OpenLLMetry) plus Traceloop's evaluation platform, now part of ServiceNow

7.5k+ GitHub Stars
OpenTelemetry Standard
ServiceNow Acquired By

Overview

OpenLLMetry is an open-source set of OpenTelemetry instrumentations for LLM applications. With a line of setup it traces calls to more than 20 model providers, vector databases and frameworks, and because the output is standard OpenTelemetry, you can send it to whatever backend you already use: Datadog, Grafana, Honeycomb, New Relic or Traceloop's own platform. SDKs exist for Python, TypeScript, Go and Ruby.

Traceloop, the Israeli Y Combinator company behind it, adds a commercial platform on top with monitoring, built-in quality evaluators (faithfulness, relevance, safety), custom evaluators and quality gates for CI. Traceloop raised a $6.1M seed in 2025 and has since been acquired by ServiceNow. OpenLLMetry remains Apache 2.0 licensed on GitHub.

The Verdict

Who Should Use OpenLLMetry?

Best For

  • Teams that already run OpenTelemetry
  • Stacks with several model providers and frameworks
  • Avoiding lock-in to one observability vendor
  • ServiceNow customers building AI agents

Not Ideal For

  • Teams wanting one all-in-one LLM platform with prompt management
  • Buyers wary of roadmap changes after an acquisition
  • Teams with no observability backend at all

What's Great

  • Built on the OpenTelemetry standard, so data goes anywhere
  • Broad coverage of providers, vector DBs and frameworks
  • Simple setup in Python, TypeScript, Go and Ruby
  • Built-in evaluators and CI quality gates in the platform
  • SOC 2 and HIPAA support on the commercial platform

Watch Out For

  • ServiceNow acquisition may shift the commercial roadmap
  • Instrumentation only; you need a backend to view data
  • Fewer prompt-management features than LangSmith or Langfuse
  • Limited public reviews

Pricing

View all features & details

Key Features

  • OpenTelemetry instrumentation for LLM calls
  • Prompts, completions, tokens and latency in traces
  • Built-in quality evaluators
  • Custom evaluators and CI quality gates
  • Drift and regression monitoring

Integrations

  • OpenAI, Anthropic, Gemini, Bedrock, Ollama and more
  • Pinecone, Chroma and other vector DBs
  • LangChain, LlamaIndex, CrewAI
  • Datadog, Grafana, Honeycomb, New Relic backends

How It Compares

Feature OpenLLMetry OpenInference (Arize) Langfuse LangSmith
OpenTelemetry native Yes Yes Accepts OTel Partial
Open source Yes (Apache 2.0) Yes Yes (MIT) No
Backend choice Any OTel backend Phoenix or any Langfuse LangSmith
Languages Python, TS, Go, Ruby Python, TS, Java Python, TS Python, TS

Is this your tool? Add a badge