Hugging Face TRL iconHugging Face TRL

Hugging Face's open-source post-training library: SFT, DPO, GRPO, RLOO, KTO and reward trainers, with a CLI and PEFT, DeepSpeed and Unsloth support

19.7k+ GitHub Stars
v1.17.1 Latest Release (Oct 2026)
Apache-2.0 License

Overview

TRL is Hugging Face's open-source library "to post-train foundation models": the step after pre-training where you teach a model to follow instructions, prefer good answers and reason. Its trainers wrap Transformers, so the same code scales from one GPU to many with Accelerate, DeepSpeed or FSDP, and PEFT adds LoRA and QLoRA. Since TRL v1 (March 31, 2026) the stable core follows semantic versioning: SFTTrainer, DPOTrainer, RewardTrainer, RLOOTrainer and GRPOTrainer, plus KTO and a distillation trainer. Newer methods such as Online DPO, ORPO, CPO and GKD live in an experimental layer that can change quickly. A command line (trl sft, trl dpo, trl kto) runs jobs without writing a script. Tools such as Axolotl and Unsloth build on TRL trainers, and Hugging Face's own no-code AutoTrain is no longer maintained and points users to TRL instead. Also from Hugging Face: the Hugging Face Hub, LeRobot, Transformers, and Argilla, which joined Hugging Face in 2024.

The Verdict

Who Should Use TRL?

Best For

  • ML engineers who want full control over post-training in Python
  • Reasoning and RL work with GRPO or RLOO
  • Teams already on Transformers, PEFT and the Hugging Face Hub

Not Ideal For

  • No-code users (TRL is a library, not an app)
  • Anyone who wants one config file to manage everything (see Axolotl)
  • Projects that need experimental trainers to stay stable

What's Great

  • New post-training methods usually land here first
  • Stable v1 core with semantic versioning since March 2026
  • Works with PEFT, DeepSpeed, FSDP, Unsloth and the Hub
  • Frequent releases (several a month in late 2026)

Watch Out For

  • More manual setup than config-driven tools like Axolotl
  • Experimental trainers do not follow semver and can break
  • PPO is no longer in the main trainer list of the current docs

Pricing

View all features & details

Trainers

  • Offline: SFT, DPO, KTO
  • Online RL: GRPO, RLOO
  • RewardTrainer and DistillationTrainer
  • Experimental: Online DPO, ORPO, CPO, GKD and more

Integrations

  • Transformers, Accelerate, PEFT (LoRA, QLoRA)
  • DeepSpeed and FSDP for multi-GPU
  • Unsloth for faster single-GPU runs
  • AMD ROCm and AWS Trainium support

Real-World Usage

Used By

  • Hugging Face's Open-R1 reasoning replication
  • Axolotl and Unsloth build on TRL trainers
  • AutoTrain's maintainers now point users to TRL

2026 Milestones

  • TRL v1 stable release (Mar 31, 2026)
  • Ulysses sequence parallelism (Mar 2026)
  • v1.17 (Oct 2026)

How It Compares

Feature TRL Axolotl Unsloth
Interface Python library + CLI YAML config + CLI Python, Studio web UI
Strongest at Newest methods, full control Multi-GPU, reproducible runs Single-GPU speed and memory
RL methods GRPO, RLOO, DPO, KTO DPO, KTO, ORPO, GRPO DPO, GRPO
Builds on TRL Is TRL Yes Integrates with TRL

Sources: GitHub · Docs · TRL v1 (Mar 2026) · AutoTrain Advanced

Is this your tool? Add a badge