Hugging Face TRL
Hugging Face's open-source post-training library: SFT, DPO, GRPO, RLOO, KTO and reward trainers, with a CLI and PEFT, DeepSpeed and Unsloth support
Overview
TRL is Hugging Face's open-source library "to post-train foundation models": the step after pre-training where you teach a model to follow instructions, prefer good answers and reason. Its trainers wrap Transformers, so the same code scales from one GPU to many with Accelerate, DeepSpeed or FSDP, and PEFT adds LoRA and QLoRA. Since TRL v1 (March 31, 2026) the stable core follows semantic versioning: SFTTrainer, DPOTrainer, RewardTrainer, RLOOTrainer and GRPOTrainer, plus KTO and a distillation trainer. Newer methods such as Online DPO, ORPO, CPO and GKD live in an experimental layer that can change quickly. A command line (trl sft, trl dpo, trl kto) runs jobs without writing a script. Tools such as Axolotl and Unsloth build on TRL trainers, and Hugging Face's own no-code AutoTrain is no longer maintained and points users to TRL instead. Also from Hugging Face: the Hugging Face Hub, LeRobot, Transformers, and Argilla, which joined Hugging Face in 2024.
The Verdict
Who Should Use TRL?
Best For
- ML engineers who want full control over post-training in Python
- Reasoning and RL work with GRPO or RLOO
- Teams already on Transformers, PEFT and the Hugging Face Hub
Not Ideal For
- No-code users (TRL is a library, not an app)
- Anyone who wants one config file to manage everything (see Axolotl)
- Projects that need experimental trainers to stay stable
What's Great
- New post-training methods usually land here first
- Stable v1 core with semantic versioning since March 2026
- Works with PEFT, DeepSpeed, FSDP, Unsloth and the Hub
- Frequent releases (several a month in late 2026)
Watch Out For
- More manual setup than config-driven tools like Axolotl
- Experimental trainers do not follow semver and can break
- PPO is no longer in the main trainer list of the current docs
Pricing
View all features & details
Trainers
- Offline: SFT, DPO, KTO
- Online RL: GRPO, RLOO
- RewardTrainer and DistillationTrainer
- Experimental: Online DPO, ORPO, CPO, GKD and more
Integrations
- Transformers, Accelerate, PEFT (LoRA, QLoRA)
- DeepSpeed and FSDP for multi-GPU
- Unsloth for faster single-GPU runs
- AMD ROCm and AWS Trainium support
Real-World Usage
Used By
- Hugging Face's Open-R1 reasoning replication
- Axolotl and Unsloth build on TRL trainers
- AutoTrain's maintainers now point users to TRL
2026 Milestones
- TRL v1 stable release (Mar 31, 2026)
- Ulysses sequence parallelism (Mar 2026)
- v1.17 (Oct 2026)
How It Compares
| Feature | TRL | Axolotl | Unsloth |
|---|---|---|---|
| Interface | Python library + CLI | YAML config + CLI | Python, Studio web UI |
| Strongest at | Newest methods, full control | Multi-GPU, reproducible runs | Single-GPU speed and memory |
| RL methods | GRPO, RLOO, DPO, KTO | DPO, KTO, ORPO, GRPO | DPO, GRPO |
| Builds on TRL | Is TRL | Yes | Integrates with TRL |
Sources: GitHub · Docs · TRL v1 (Mar 2026) · AutoTrain Advanced
Is this your tool? Add a badge