vLLM iconvLLM

Open-source LLM inference and serving engine from UC Berkeley with PagedAttention, OpenAI-compatible server and 200+ model architectures

96.8k+ GitHub Stars
2,000+ Contributors
v0.33 Latest (Nov 2026)

Overview

vLLM is a fast, easy-to-use open-source library for LLM inference and serving. It started in UC Berkeley's Sky Computing Lab, introduced PagedAttention for memory-efficient KV caching, and now has more than 2,000 contributors. It runs an OpenAI-compatible server (plus Anthropic Messages API and gRPC endpoints), supports more than 200 model architectures, and covers the main production techniques: continuous batching, chunked prefill, prefix caching, speculative decoding, disaggregated prefill and decode, multi-LoRA, structured outputs and tool calling. Quantization options include FP8, INT8, INT4, GPTQ, AWQ and GGUF, and it scales with tensor, pipeline, data, expert and context parallelism. It runs on NVIDIA, AMD and Intel GPUs and CPUs, with TPU, Gaudi, Ascend and others through plugins. Releases ship about every two weeks; v0.33 landed in November 2026. In January 2026 the vLLM creators founded Inferact, led by CEO Simon Mo, which raised a $150M seed at an $800M valuation co-led by a16z and Lightspeed to build a commercial serverless vLLM. Compare it with SGLang and TensorRT-LLM; for laptops see Ollama or llama.cpp.

The Verdict

Who Should Use vLLM?

Best For

  • Teams self-hosting open models on GPU servers
  • Batch and offline inference at any load
  • Anyone who needs new open models supported on release day

Not Ideal For

  • Single-user desktop use
  • Teams with no GPU or ops capacity (use a hosted API)

What's Great

  • Largest ecosystem and best day-one support for new models
  • Works well at any load for offline and batch inference
  • Broad feature set: quantization, LoRA, tool calling, structured output

Watch Out For

  • Worse P99 time-to-first-token than SGLang at high concurrency in one test
  • Fast release pace means frequent upgrades and config changes
  • Benchmark on your own workload; results vary by model and traffic

Pricing

View all features & details

Key Features

  • PagedAttention and continuous batching
  • Chunked prefill and prefix caching
  • Speculative decoding (EAGLE and others)
  • Disaggregated prefill and decode
  • OpenAI-compatible, Anthropic Messages and gRPC APIs
  • FP8, INT8, INT4, GPTQ, AWQ and GGUF quantization
  • Multi-LoRA, structured outputs, tool calling

Project and Company

  • Started at UC Berkeley Sky Computing Lab
  • Inferact: CEO Simon Mo; $150M seed (Jan 2026)
  • Backers include a16z, Lightspeed, Sequoia and Databricks Ventures

How It Compares

Feature vLLM SGLang Ollama
Target Production GPU serving Production GPU serving Local desktop and dev
License Apache-2.0 Apache-2.0 MIT
Strength Ecosystem, model coverage Shared-prefix latency Ease of setup
Company behind it Inferact RadixArk Ollama

Sources: GitHub · vllm.ai · Releases (Nov 2026) · TechCrunch (Jan 2026) · SiliconANGLE (Jan 2026)

Is this your tool? Add a badge