The tutorial walks through a complete preference‑learning pipeline using the Anthropic HH‑RLHF dataset and Direct Preference Optimization (DPO). It begins by configuring a reproducible Colab environment, installing compatible versions of TRL, Transformers, PEFT, and related libraries, and then loads balanced samples from the helpful‑base, helpful‑rejection‑sampled, helpful‑online, and harmless‑base subsets. Conversational turns are parsed, and chosen–rejected pairs are filtered to ensure identical prefixes. The authors audit the data for length and structural biases, train a TF‑IDF + logistic‑regression diagnostic to surface any lexical shortcuts, and construct tokenizer‑aware length features before applying a token‑length filter (max prompt = 256 tokens, max total = 512 tokens).
A Qwen2.5‑0.5B‑Instruct model is fine‑tuned with DPO, optionally applying LoRA (r = 16, α = 32, dropout = 0.05) while keeping the base model frozen as the reference. Training runs for 30 steps with an effective batch size of 8 (batch = 1 × grad_accum), learning rate = 5e‑6, beta = 0.1, and warm‑up ratio = 0.1, using bf16/fp16 as available. After training, reward accuracy is evaluated per source, and the policy’s tendency to favor longer responses is examined via a length‑delta correlation. Sample generations are produced, and the final adapter and tokenizer are saved to the specified output directory.
- Model: Qwen2.5‑0.5B‑Instruct (0.5 B parameters)
- LoRA: r = 16, α = 32, dropout = 0.05, task_type = CAUSAL_LM
- DPO hyper‑params: β = 0.1, max_steps = 30, batch = 1, grad_accum = 8, lr = 5e‑6, warmup_ratio = 0.1
- Token limits: max_prompt_length = 256, max_length = 512
- Dataset subsets: helpful‑base, helpful‑rejection‑sampled, helpful‑online, harmless‑base (120 train / 30 test per subset)
- Environment: TRL ≥ 0.12, Transformers ≥ 4.45, PEFT, Accelerate, Datasets, scikit‑learn
Why this matters
The tutorial provides a reproducible framework for detecting and mitigating preference biases in RLHF‑style data, which is essential because unexamined length or lexical shortcuts can cause a model to optimize for superficial traits rather than true human preferences. By integrating dataset auditing, lexical diagnostics, and per‑source DPO evaluation, the approach isolates genuine alignment signals from confounding factors—a methodological advance that can be applied to larger models and alternative preference datasets. The use of LoRA enables efficient adaptation without full fine‑tuning, preserving computational accessibility while still allowing rigorous analysis of reward model behavior. This combination of bias checking and lightweight adaptation offers a practical path toward more trustworthy preference‑aligned language models.