melon-test

Reinforcement Learning

Slides

• 强化学习引入
• 蒙特卡洛与时序差分
• DQN
• Deepseek-R1 & DAPO
• 过程奖励模型PRM
• 强化学习面临的挑战
• DDPG
• RLHF
• GRPO

Books

• 深度强化学习：基础、研究与应用
• 强化学习 (by Sutton)

Papers

Surveys

Recommended:

• Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models
• Reinforcement Learning Enhanced LLMs: A Survey

Optional:

• A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challenges
• A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discovery
• A Comprehensive Survey of LLM Alignment Techniques: RLHF, RLAIF, PPO, DPO and More
• A Survey of Reinforcement Learning from Human Feedback

Basic RL Algorithms

Recommended:

• Trust Region Policy Optimization(TRPO)
• Proximal Policy Optimization Algorithms(PPO)
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models(GRPO)

Optional:

• Actor-Critic Algorithms
• A2C(from Hugging Face)
• A2C is a special case of PPO
• Asynchronous methods for deep reinforcement learning(A3C)
• Continuous control with deep reinforcement learning(DDPG)
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor(SAC)
• Mirror Descent Policy Optimization(MDPO)
• Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training(Off-Policy GRPO)
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale(tuned GRPO)
• What is the Alignment Objective of GRPO?

LLM Alignment Techniques

RLHF

• InstructGPT：Training language models to follow instructions with human feedback
• Safe RLHF: Safe Reinforcement Learning from Human Feedback

RLAIF

• Constitutional AI: Harmlessness from AI Feedback
• RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
• Generative Verifiers: Reward Modeling as Next-Token Prediction
• Self-Rewarding Language Models

More About Reward Models For Alignment

• Rule Based Rewards for Language Model Safety
• Beyond Scalar Reward Model: Learning Generative Judge from Preference Data
• Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown
• Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs
• Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts

DPO

• Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Analysis:

• Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback
• Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study
• Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks
• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint

Long CoT Related

Read the survey for a lot more information.

Aha Moment

• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
• Understanding R1-Zero-Like Training: A Critical Perspective
• Understanding Aha Moments: from External Observations to Internal Mechanisms

Process Reward Model

• Let’s Verify Step by Step
• Do We Need to Verify Step by Step? Rethinking Process Supervision from a Theoretical Perspective
• Process Reinforcement Through Implicit Rewards

Test-Time Scaling

Analysis:

• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• ECM: A Unified Electronic Circuit Model for Explaining the Emergence of In-Context Learning and Chain-of-Thought in Large Language Model

Vertical Scaling:

• s1: Simple test-time scaling

Parallel Scaling:

• Self-Consistency Improves Chain Of Thought Reasoning In Language Models
• Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification
• ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning

Reward-Model-Based RL Algorithms

• ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models
• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models(GRPO)
• Reinforce++: A simple and efficient approach for aligning large language models

Reward-Model-Free RL Algorithms (mostly about DPO, thus optional)

• Offline Reinforcement Learning for LLM Multi-Step Reasoning
• Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization
• Critical plan step learning boosts llm generalization in reasoning tasks
• Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability
• Focused-DPO: Enhancing Code Generation Through Focused Preference Optimization on Error-Prone Points
• Thinking Preference Optimization

Appendix: Math Provers

• Generative Language Modeling for Automated Theorem Proving(GPT-f)
• HyperTree Proof Search for Neural Theorem Proving
• DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search
• InternLM2.5-StepProver: Advancing Automated Theorem Proving via Expert Iteration on Large-Scale LEAN Problems
• HunyuanProver: A Scalable Data Synthesis Framework and Guided Tree Search for Automated Theorem Proving

2025

• STP: Self-play LLM Theorem Provers with Iterative Conjecturing and Proving
• BFS-Prover: Scalable Best-First Tree Search for LLM-based Automatic Theorem Proving
• Goedel-Prover: A Frontier Model for Open-Source Automated Theorem Proving
• Leanabell-Prover: Posttraining Scaling in Formal Reasoning
• REAL-Prover: Retrieval Augmented Lean Prover for Mathematical Reasoning
• Kimina-Prover Preview: Towards Large Formal Reasoning Models with Reinforcement Learning
• DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition

Appendix: Other DPO related Methods or Improvements

• ORPO: Monolithic Preference Optimization without Reference Model
• SLiC-HF: Sequence Likelihood Calibration with Human Feedback
• β-DPO: Direct Preference Optimization with Dynamic β
• sDPO: Don’t Use Your Data All at Once
• mDPO: Conditional Preference Optimization for Multimodal Large Language Models
• Statistical Rejection Sampling Improves Preference Optimization
• Generalized Preference Optimization: A Unified Approach to Offline Alignment
• Offline Regularised Reinforcement Learning for Large Language Models Alignment
• Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization
• Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
• Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences
• Self-Play Preference Optimization for Language Model Alignment
• A Minimaximalist Approach to Reinforcement Learning from Human Feedback
• Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive
• Token-level Direct Preference Optimization
• A General Theoretical Paradigm to Understand Learning from Human Preferences

Appendix：Some LLM technical reports with related RL algorithms

(Here RLHF and RLAIF utilize PPO by default)

2022

• InstructGPT: proposed RLHF

2023

Model	Algorithm
GPT4	RLHF + Rule-Based Reward Model(RBRM)
Llama2	RLHF + Rejection Sampling
qwen	RLHF
zephyr-7B	direct distillation of LM Alignment
starling-7B	RLAIF
Gemini	RLHF, continuously updating Reward Model

2024

Model	Algorithm
DeepSeekMath	proposed GRPO
Claude3	RLAIF(see Constitutional AI)
InternLM2	COOL RLHF
Reka	RLHF
Llama3	rejection-sampling + DPO with some tricks
Phi-3	DPO
Zephyr 141B-A39B	ORPO
DeepSeek-V2	GRPO
Qwen2	DPO
Nemotron-4 340B	DPO + RPO
ChatGLM	ChatGLM-RLHF
Hermes 3	DPO + LoRA
Gemma2	RLHF
Qwen2.5	DPO for offline and GRPO for online
Hunyuan-Large	DPO
Phi-4	DPO
DeepSeek-V3	GRPO

2025

Model	Algorithm
MiniMax-01	DPO for offline, modified GRPO for online
Kimi-k1.5	MDPO for CoT
DeepSeek-R1	GRPO for CoT
Qwen3	GRPO for CoT
Phi-4-reasoning	GRPO for CoT