Comet Opik
Open-source LLM and agent observability from Comet: tracing, LLM-as-a-judge evals, prompt optimization and guardrails, Apache-2.0 and self-hostable
Overview
Opik is Comet's open-source platform for tracing, evaluating and monitoring LLM apps, RAG systems and AI agents. It logs every LLM call, tool call and retrieval step as a trace tree, scores outputs with 30+ LLM-as-a-judge and heuristic metrics, runs experiments on datasets (including from PyTest in CI), and adds production dashboards, online evaluation rules and guardrails for PII and policy violations. Newer pieces include a Prompt Optimizer with six algorithms, Ollie (an assistant that analyzes traces and writes fixes plus a regression test into your codebase) and Cost Intelligence for Claude Code and Codex spend. Comet launched Opik in September 2024; the whole platform, backend included, is Apache-2.0 and can be self-hosted with Docker Compose or Helm, and it accepts traces over OpenTelemetry from any language. Also from Comet: Comet Experiment Management, its original ML experiment tracking product.
The Verdict
Who Should Use Comet Opik?
Best For
- Teams that want a free, fully self-hosted LLM tracing and eval stack under Apache 2.0
- Agent builders who need traces, evals and guardrails in one tool
- Comet customers who want ML experiments and LLM work from one vendor
- Small teams on a budget: Pro Cloud is $19/month with up to 50 members
Not Ideal For
- Teams that want the most mature ecosystem of guides and case studies
- Heavy cloud users on the free tier: 25K spans a month, no overage
- Prompt optimization without labeled datasets ready
What's Great
- Whole platform is Apache-2.0, backend included, with no commercial license needed to self-host
- Broad integrations: LangChain, LlamaIndex, OpenAI, Anthropic, LangGraph, CrewAI, Google ADK, AutoGen and more
- OpenTelemetry support lets Java, Ruby and .NET apps send traces
- Can seed evaluation datasets from production traces
- Built for scale: Comet says it handles 40M+ traces a day
Watch Out For
- Newest of the big three: thinner set of guides, third-party integrations and case studies
- The experiment-tracking tie-in matters less if you do not use Comet
- Free Cloud stops at 25K spans a month and keeps data 60 days
- Evaluation framework judged less complete than LangSmith's in one comparison
- Few independent user reviews so far
Pricing
View all features & details
Tracing and Debugging
- Trace trees for LLM calls, tool use, retrieval and multi-step agents
- Python
@trackdecorator for nested calls; conversation threads - OpenTelemetry ingestion from any language
- Annotation by subject-matter experts, Diagnostics for silent errors, audit logs
- Ollie: debugging assistant that writes fixes and regression tests
Evaluation and Optimization
- Datasets, experiments and 30+ LLM-as-a-judge metrics (hallucination, moderation, RAG quality)
- Heuristic and custom metrics; PyTest integration for CI
- Test Suites and plain-text assertions; Agent Playground
- Prompt and agent optimizer SDK with six algorithms
Production
- Dashboards for feedback scores, trace counts and token usage
- Online evaluation rules and alerts
- Guardrails that block policy violations and PII exposure
- Cost Intelligence for Claude Code and Codex usage
Deployment
- Comet-hosted cloud
- Self-hosted with Docker Compose or Kubernetes (Helm)
- MCP server for IDE use
Real-World Usage
Adoption
- Logos on the product page include Uber, Netflix, Autodesk, Etsy, NatWest and Stability AI
- The GitHub repo is very active (pushed the day this page was checked)
History
- Announced by Comet on September 16, 2024 as an open-source LLM evaluation platform
- Grew from evals into tracing, monitoring, guardrails and agent optimization
How It Compares
| Feature | Comet Opik | Langfuse | Arize Phoenix | LangSmith |
|---|---|---|---|---|
| License | Apache 2.0 (whole platform) | MIT core, enterprise features separate | Elastic License 2.0 | Proprietary |
| Free self-hosting | Yes | Yes | Yes | No (Enterprise only) |
| Free cloud tier | 25K spans/month | 50K observations/month | 25K spans/month (Arize AX) | 5K traces/month |
| Entry paid cloud plan | $19/month | See Langfuse pricing | $50/month (AX Pro) | $39/seat/month |
| Extras | Prompt optimizer, guardrails, coding-agent spend | Prompt management | OTel-native tracing | Deep LangChain integration |
Sources: Comet pricing · Langfuse pricing · Phoenix license · LangSmith pricing · BigData Boutique
Is this your tool? Add a badge