Skip to content
Rabat, Morocco
Posts

RLHF in November 2026 — Where Human Feedback Counts

November 13, 2026
A hand marks a circle on one of two evaluation cards with an orange pencil
Two answers can look equally good until an evaluator checks a source or notices that one of them ignores a user constraint. Asking people to catch those differences takes time. November's research on reinforcement learning from human feedback asks a practical question. Which judgments can models supply cheaply, and which deserve closer review? In a common reinforcement learning from human feedback (RLHF) pipeline, people compare candidate responses, a reward model learns to predict their preferences, and reinforcement learning updates the language model using that reward. Direct Preference Optimization (DPO) trains on preferred and rejected responses directly, without a separately trained reward model or an RL loop. Both approaches depend on what the underlying preference labels capture. Annual Reviews' statistical overview of RLHF In a November 12 preprint, Banerjee and colleagues compared human-labeled and synthetic data for supervised fine-tuning followed by DPO. They matched estimated annotation budgets across data sources, allowing the cheaper source to supply more examples. In experiments across three model families, the authors reported strong win rates for synthetic data at estimated annotation budgets of $200–$400 on helpfulness and summarization tasks. Human-labeled data came close on helpfulness with a much larger budget. Synthetic data kept its lead on summarization across the budgets tested. A model judged most test outputs, and the authors compared its judgments with human ratings on 50 sampled comparisons per task. Banerjee et al.'s preprint and evaluation details AERO, posted on November 7 and accepted to NeurIPS 2026, takes a different approach. It uses a group of reward models to decide when to request a reference judgment. When they agree strongly, the system can use a synthetic label. When they disagree enough, it asks an outside evaluator. The authors report competitive alignment results with fewer calls to that evaluator. In every experiment, the evaluator was a deterministic reward model standing in for a human annotator. How much annotator time this would save in practice remains an open question. AERO paper and experimental setup Disagreement between evaluators is one reason to inspect a label. A preference can also conflict with an explicit criterion, even when reviewers agree on their pick. In a November 5 preprint, Liu and Yan describe CoDPO, which gives less weight to pairs whose preferred answer scores worse on a criterion such as correctness or response length. The authors report less drift on the measured criteria across four experimental domains while retaining most of the preference learning of standard DPO. CoDPO preprint A shopping agent may ask a clarifying question, look up several products, and then recommend an option. The recommendation can read well even when the quoted price is stale or the agent overlooked a requirement from an earlier turn. Rewarding the whole interaction gives evaluators a chance to catch those errors if they can see the relevant evidence. E-GenRM, a preprint posted on November 2, applies that idea to Taobao. The authors used shopping behavior as an implicit signal to construct multi-step examples and trained a reward model to support its judgments with evidence. In a 14-day online A/B test, they reported a 1.39-percentage-point gain in effective shopping assistance and a 2.33-point gain in user satisfaction for the agent post-trained with the reward model. Those figures describe the Taobao experiment, where the feedback included what shoppers did in the app. E-GenRM preprint For the shopping example, the catalog can verify the price and stock. A human reviewer can judge whether the agent asked the right follow-up question before recommending a product. Keeping those signals separate lets an evaluation reveal which part of the interaction improved. Read the companion article on recent agent launches and benchmarks

Enjoyed this blog?

Seriously, thank you for spending your time here. If you made it to the end, you’re the kind of person who cares about craft, not just quick fixes.If you’d like to support my work, pick one (no pressure). It genuinely helps me keep writing and improving these blogs.
Subscribe to get a heads-up when I publish the next one. I keep it low-volume, and I only send what I’d personally want to read.

Subscribe to Josias's Newsletter

I occasionally write about software engineering, technology, and share thoughts on innovation and engineering.
One last thing: if you know someone who’d benefit from this, sharing it with them is one of the best compliments you can give. I appreciate you.