Comet Opik iconComet Opik

Open-source LLM and agent observability from Comet: tracing, LLM-as-a-judge evals, prompt optimization and guardrails, Apache-2.0 and self-hostable

23K+ GitHub Stars
30+ LLM-as-a-judge metrics
2024 Launched by Comet

Overview

Opik is Comet's open-source platform for tracing, evaluating and monitoring LLM apps, RAG systems and AI agents. It logs every LLM call, tool call and retrieval step as a trace tree, scores outputs with 30+ LLM-as-a-judge and heuristic metrics, runs experiments on datasets (including from PyTest in CI), and adds production dashboards, online evaluation rules and guardrails for PII and policy violations. Newer pieces include a Prompt Optimizer with six algorithms, Ollie (an assistant that analyzes traces and writes fixes plus a regression test into your codebase) and Cost Intelligence for Claude Code and Codex spend. Comet launched Opik in September 2024; the whole platform, backend included, is Apache-2.0 and can be self-hosted with Docker Compose or Helm, and it accepts traces over OpenTelemetry from any language. Also from Comet: Comet Experiment Management, its original ML experiment tracking product.

The Verdict

Who Should Use Comet Opik?

Best For

  • Teams that want a free, fully self-hosted LLM tracing and eval stack under Apache 2.0
  • Agent builders who need traces, evals and guardrails in one tool
  • Comet customers who want ML experiments and LLM work from one vendor
  • Small teams on a budget: Pro Cloud is $19/month with up to 50 members

Not Ideal For

  • Teams that want the most mature ecosystem of guides and case studies
  • Heavy cloud users on the free tier: 25K spans a month, no overage
  • Prompt optimization without labeled datasets ready

What's Great

  • Whole platform is Apache-2.0, backend included, with no commercial license needed to self-host
  • Broad integrations: LangChain, LlamaIndex, OpenAI, Anthropic, LangGraph, CrewAI, Google ADK, AutoGen and more
  • OpenTelemetry support lets Java, Ruby and .NET apps send traces
  • Can seed evaluation datasets from production traces
  • Built for scale: Comet says it handles 40M+ traces a day

Watch Out For

  • Newest of the big three: thinner set of guides, third-party integrations and case studies
  • The experiment-tracking tie-in matters less if you do not use Comet
  • Free Cloud stops at 25K spans a month and keeps data 60 days
  • Evaluation framework judged less complete than LangSmith's in one comparison
  • Few independent user reviews so far

Pricing

View all features & details

Tracing and Debugging

  • Trace trees for LLM calls, tool use, retrieval and multi-step agents
  • Python @track decorator for nested calls; conversation threads
  • OpenTelemetry ingestion from any language
  • Annotation by subject-matter experts, Diagnostics for silent errors, audit logs
  • Ollie: debugging assistant that writes fixes and regression tests

Evaluation and Optimization

  • Datasets, experiments and 30+ LLM-as-a-judge metrics (hallucination, moderation, RAG quality)
  • Heuristic and custom metrics; PyTest integration for CI
  • Test Suites and plain-text assertions; Agent Playground
  • Prompt and agent optimizer SDK with six algorithms

Production

  • Dashboards for feedback scores, trace counts and token usage
  • Online evaluation rules and alerts
  • Guardrails that block policy violations and PII exposure
  • Cost Intelligence for Claude Code and Codex usage

Deployment

  • Comet-hosted cloud
  • Self-hosted with Docker Compose or Kubernetes (Helm)
  • MCP server for IDE use

Real-World Usage

Adoption

  • Logos on the product page include Uber, Netflix, Autodesk, Etsy, NatWest and Stability AI
  • The GitHub repo is very active (pushed the day this page was checked)

History

  • Announced by Comet on September 16, 2024 as an open-source LLM evaluation platform
  • Grew from evals into tracing, monitoring, guardrails and agent optimization

How It Compares

Feature Comet Opik Langfuse Arize Phoenix LangSmith
License Apache 2.0 (whole platform) MIT core, enterprise features separate Elastic License 2.0 Proprietary
Free self-hosting Yes Yes Yes No (Enterprise only)
Free cloud tier 25K spans/month 50K observations/month 25K spans/month (Arize AX) 5K traces/month
Entry paid cloud plan $19/month See Langfuse pricing $50/month (AX Pro) $39/seat/month
Extras Prompt optimizer, guardrails, coding-agent spend Prompt management OTel-native tracing Deep LangChain integration

Sources: Comet pricing · Langfuse pricing · Phoenix license · LangSmith pricing · BigData Boutique

Is this your tool? Add a badge