#GRPO
LightNews LightNews
arxiv-cs-cl.bsky.social
Besher Hassan, Xiuying Chen
GRASP LoRA: GRPO Guided Adapter Sparsity Policy for Cross Lingual Transfer
https://arxiv.org/abs/2601.06702
January 13, 2026 at 2:38 PM
arxiv-cs-cl.bsky.social
Shubhashis Roy Dipta, Khairul Mahbub, Nadia Najjar
GanitLLM: Difficulty-Aware Bengali Mathematical Reasoning through Curriculum-GRPO
https://arxiv.org/abs/2601.06767
January 13, 2026 at 2:36 PM
cscl-bot.bsky.social
Shubhashis Roy Dipta, Khairul Mahbub, Nadia Najjar: GanitLLM: Difficulty-Aware Bengali Mathematical Reasoning through Curriculum-GRPO https://arxiv.org/abs/2601.06767 https://arxiv.org/pdf/2601.06767 https://arxiv.org/html/2601.06767
January 13, 2026 at 6:30 AM
cscl-bot.bsky.social
Besher Hassan, Xiuying Chen: GRASP LoRA: GRPO Guided Adapter Sparsity Policy for Cross Lingual Transfer https://arxiv.org/abs/2601.06702 https://arxiv.org/pdf/2601.06702 https://arxiv.org/html/2601.06702
January 13, 2026 at 6:30 AM
csar-bot.bsky.social
Dimple Vijay Kochar, Nathaniel Pinckney, Guan-Ting Liu, Chia-Tung Ho, Chenhui Deng, Haoxing Ren, Brucek Khailany: GRPO with State Mutations: Improving LLM-Based Hardware Test Plan Generation https://arxiv.org/abs/2601.07593 https://arxiv.org/pdf/2601.07593 https://arxiv.org/html/2601.07593
January 13, 2026 at 6:29 AM
bluesky.awakari.com
Building a Reasoning LLM with GRPO. If your AI agent only knows how to “search and find,” it’s already behind. The future of AI isn’t just about retrieval; it’s about Logical… Continue reading on Medium »

Interest | Match | Feed
Origin
medium.com
January 12, 2026 at 1:31 PM
cscv-bot.bsky.social
Wang, Lu, Xu, Chen, Yang, Wang, Chen, Chen, Hu, Wu, Shao, Lu, Luo: TAGRPO: Boosting GRPO on Image-to-Video Generation with Direct Trajectory Alignment https://arxiv.org/abs/2601.05729 https://arxiv.org/pdf/2601.05729 https://arxiv.org/html/2601.05729
January 12, 2026 at 6:30 AM
paper.bsky.social
2601.05242
言語モデルがますます高度になるにつれ、ユーザーは正確な応答だけでなく、様々なシナリオにおける多様な人間の嗜好に沿った行動も提供することを期待している。これを実現するため、強化学習（RL）パイプラインでは複数の報酬を取り入れ始めた。各報酬は異なる選好を捉え、モデルをこれらの望ましい行動へと...
言語モデルがますます高度になるにつれ、ユーザーは正確な応答だけでなく、様々なシナリオにおける多様な人間の嗜好に沿った行動も提供することを期待している。 これを実現するため、強化学習（RL）パイプラインでは複数の報酬を取り入れ始めた。各報酬は異なる選好を捉え、モデルをこれらの望ましい行動へと導く。 しかしながら、近年の研究では、その適性を検証することなく、マルチ報酬設定下でグループ相対政策最適化（GRPO）を適用することがデフォルトとなっている。 本論文では、異なるロールアウト報酬の組み合わせを正規化するためにGRPOを直接適用すると、それらが同一のアドバンテージ値に収束し、訓練信号の分解能が低下し、最適でない収束をもたらし、場合によっては早期の訓練失敗を引き起こすことを実証する。 次に、これらの課題を解決する新たなポリシー最適化手法として、グループ報酬分離正規化ポリシー最適化（GDPO）を導入する。本手法は個々の報酬の正規化を分離することで、報酬間の相対的な差異をより忠実に保持し、より正確なマルチ報酬最適化を可能とすると同時に、訓練の安定性を大幅に改善する。 GDPOとGRPOを、ツール呼び出し、数学的推論、コーディング推論の3つのタスクで比較し、正しさの指標（正確性、バグ比率）と制約順守の指標（形式、長さ）の両方を評価する。 あらゆる設定において、GDPOは一貫してGRPOを上回る性能を示しており、多報酬強化学習最適化におけるその有効性と汎用性を実証している。
January 11, 2026 at 12:06 AM
paper.bsky.social
[24/30] 118 Likes, 6 Comments, 1 Posts
2601.05242, cs․CL | cs․AI | cs․LG, 08 Jan 2026

🆕GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization

Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjie Liu, Min-Hung Chen, Hongxu Yin, Yu-Ch...
As language models become increasingly capable, users expect them to provide not only accurate responses but also behaviors aligned with diverse human preferences across a variety of scenarios. To achieve this, Reinforcement learning (RL) pipelines have begun incorporating multiple rewards, each capturing a distinct preference, to guide models toward these desired behaviors. However, recent work has defaulted to apply Group Relative Policy Optimization (GRPO) under multi-reward setting without examining its suitability. In this paper, we demonstrate that directly applying GRPO to normalize distinct rollout reward combinations causes them to collapse into identical advantage values, reducing the resolution of the training signal and resulting in suboptimal convergence and, in some cases, early training failure. We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues by decoupling the normalization of individual rewards, more faithfully preserving their relative differences and enabling more accurate multi-reward optimization, along with substantially improved training stability. We compare GDPO with GRPO across three tasks: tool calling, math reasoning, and coding reasoning, evaluating both correctness metrics (accuracy, bug ratio) and constraint adherence metrics (format, length). Across all settings, GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.
January 11, 2026 at 12:06 AM
abls.bsky.social
Eu fico grata que o pessoal do subredit da crônica do matador do rei mantém aquela merda ativa, ficam postando teorias e tbm serve como um grpo de terapia kkkk amo demais, faz tudo por mim
January 10, 2026 at 7:44 PM
sungkim.bsky.social
They find that the common practice of applying GRPO to multi-reward optimization normalizes distinct reward combinations into identical advantage values. This collapses the training signal, reduces reward-level resolution, and leads to suboptimal convergence.
January 10, 2026 at 2:20 PM
sungkim.bsky.social
GDPO, not GRPO

NVIDIA introduces Group reward-Decoupled Normalization Policy Optimization (GDPO), a new multi-reward RL algorithm that consistently improves per-reward convergence over GRPO across a wide range of tasks.
January 10, 2026 at 2:20 PM
pekka.bsky.social
As a technical note, they seem to have tried using GRPO replay buffer, but apparently it just collects data that's never used because their training is too short to see any replayed prompts (7B & 8B models) and/or has too few steps for reaching target temperature that would enable it (all models).
January 8, 2026 at 11:45 PM
pekka.bsky.social
I tasked agentic Gemini to figure it out by giving it their released data and code. It reported that their GRPO trainer has hardcoded temperature annealing from T=1 down to T=0.3 over 3000 steps. So their supposed temperature variants are probably all trained the same, except for some random noise.
January 8, 2026 at 11:44 PM
pekka.bsky.social
They describe how GRPO was used to generate 8 answers per problem. The idea is to nudge model weights towards the best of them.

What if you do that with temperature=0? Yep, their result files are just sets of 8 identical* responses.

They tried other temperatures, but oddly chose to report that.
January 8, 2026 at 11:42 PM
pekka.bsky.social
So 2/3 of their benchmarks are already toast, and I'll get back to the remaining one (math) later. (spoiler: it doesn't get any better.)

But I'll take a more technical detour first.

Let's get to what their training was.

"We fine-tune reasoning models with GRPO"

First again: not reasoning models.
January 8, 2026 at 11:41 PM
cslg-bot.bsky.social
Chi Liu, Xin Chen: Adaptive-Boundary-Clipping GRPO: Ensuring Bounded Ratios for Stable and Generalizable Training https://arxiv.org/abs/2601.03895 https://arxiv.org/pdf/2601.03895 https://arxiv.org/html/2601.03895
January 8, 2026 at 6:34 AM
cslg-bot.bsky.social
Amir Hossein Yari, Fajri Koto: AMIR-GRPO: Inducing Implicit Preference Signals into GRPO https://arxiv.org/abs/2601.03661 https://arxiv.org/pdf/2601.03661 https://arxiv.org/html/2601.03661
January 8, 2026 at 6:33 AM
yutakashino.bsky.social
CME 295 - Transformers & Large Language Models cme295.stanford.edu/syllabus/ この講義は良い！このくらいの適度に粗い粒度で通しでやるモダンLLM解説，実はあまり無いんだよね．flash attn.やGRPOの実装まで踏み込む講義は結構あるけど…
Syllabus | CME 295 - Transformers & Large Language Models
Here, you will find slides and recordings of class lectures, along with suggested readings.
cme295.stanford.edu
January 8, 2026 at 5:56 AM
paper.bsky.social
2601.02256
視覚生成は主に三つのパラダイムによって支配されている：自己回帰（AR）、拡散、および視覚自己回帰（VAR）モデルである。ARや拡散モデルとは異なり、VARは生成過程において異質な入力構造を扱うため、深刻な非同期的な政策の矛盾が生じる。この問題は強化学習（RL）シナリオにおいて特に深刻となり、不安定...
視覚生成は主に三つのパラダイムによって支配されている：自己回帰（AR）、拡散、および視覚自己回帰（VAR）モデルである。 ARや拡散モデルとは異なり、VARは生成過程において異質な入力構造を扱うため、深刻な非同期的な政策の矛盾が生じる。 この問題は強化学習（RL）シナリオにおいて特に深刻となり、不安定な学習と最適でない整合性をもたらす。 この問題を解決するため、我々はこれらの競合を明示的に管理することでグループ相対政策最適化（GRPO）を強化する新たな枠組みを提案する。 当社の手法は、三つの相乗効果のある構成要素を統合しています： 1) 初期段階の生成を導くための安定化中間報酬； 2) 正確な信用帰属のための動的時間ステップ再加重スキーム；および 3) 報酬フィードバック学習（ReFL）の原理から導出された新規マスク伝播アルゴリズム。空間的・時間的に最適化効果を分離するよう設計されている。 当社の手法は、標準的なGRPOベースラインと比較してサンプル品質と目的関数の整合性において大幅な改善を示し、VARモデルに対する堅牢かつ効果的な最適化を可能にします。
January 8, 2026 at 12:17 AM
paper.bsky.social
[2/30] 30 Likes, 2 Comments, 1 Posts
2601.02256, cs․CV | cs․LG, 05 Jan 2026

🆕VAR RL Done Right: Tackling Asynchronous Policy Conflicts in Visual Autoregressive Generation

Shikun Sun, Liao Qu, Huichao Zhang, Yiheng Liu, Yangyang Song, Xian Li, Xu Wang, Yi Jiang, Daniel K. Du, Xinglong Wu, Jia...
Visual generation is dominated by three paradigms: AutoRegressive (AR), diffusion, and Visual AutoRegressive (VAR) models. Unlike AR and diffusion, VARs operate on heterogeneous input structures across their generation steps, which creates severe asynchronous policy conflicts. This issue becomes particularly acute in reinforcement learning (RL) scenarios, leading to unstable training and suboptimal alignment. To resolve this, we propose a novel framework to enhance Group Relative Policy Optimization (GRPO) by explicitly managing these conflicts. Our method integrates three synergistic components: 1) a stabilizing intermediate reward to guide early-stage generation; 2) a dynamic time-step reweighting scheme for precise credit assignment; and 3) a novel mask propagation algorithm, derived from principles of Reward Feedback Learning (ReFL), designed to isolate optimization effects both spatially and temporally. Our approach demonstrates significant improvements in sample quality and objective alignment over the vanilla GRPO baseline, enabling robust and effective optimization for VAR models.
January 8, 2026 at 12:16 AM
yutakashino.bsky.social
Geminiはモデルとしては有能なんだろうけどGRPOで失敗している感じで，性格がとても悪いし途中で仕事を放棄するから，全然信頼できない．GPTは何でもできる風をして一番最初の最初から勘違いして暴走するので，人間の時間を浪費する感じで，これも信頼できない．
January 7, 2026 at 1:40 AM
topickapp.bsky.social
https://note.com/holy_fox/n/n976faac80012
ArrowIdeative-13b-NeoBase-ZERO-llm-jpは、GRPOのみを用いた強化学習で作られた純国産LLMです。
ベースモデルと指示追従モデルの中間的な性質を持ち、プロンプトエンジニアリングが有効。
本モデルはllm-jp-3.1-13bをベースにしたモデルです。
ArrowIdeative-13b-instruct-ZERO-llm-jpについて｜Holy_fox
概要 ArrowIdeative-13b-NeoBase-ZERO-llm-jpは、日本語のベースモデルに対して、完全にGRPO（Group Relative Policy Optimization）のみを用いた強化学習を施して作られた純国産LLMです。また、これはおそらく世界初の「GRPO単独の事後学習で作られた日本語直感モデル」であり、 ベースモデルと指示追従（Instruct）モデルの中間的な性質を備えています。そのため、簡単に言えば「一定のプロンプトエンジニアリングが有効なベースモデル」という位置づけです。 モデルの詳細については、以下のリンクをご覧ください。 D
note.com
January 6, 2026 at 10:42 AM
manoelhortaribeiro.bsky.social
That’s not true! All these models also are trained to reason before answering like R1… with GRPO
January 5, 2026 at 11:56 PM
tilmanbayer.bsky.social
What's more, Guo et al. stressed that e.g. the "wait" "was virtually absent during the initial training stages, appeared sporadically between steps 4,000 and 7,000 and exhibited a marked increase in frequency after step 8,000."

Yet all your experiments stopped at 500 or 1000 steps...?
"Extended Data Fig. 1 | Evolution of reasoning-related linguistic features in model outputs across training steps. a, Frequency of representative reflective terms in model-generated outputs throughout the training process. Reflective terms—including ‘wait’, ‘mistake’, ‘however’, ‘but’, ‘retry’, ‘error’, ‘verify’, ‘wrong’, ‘evaluate’ and ‘check’—were identified and curated by a panel of three human experts. Each expert independently proposed a set of words indicative of reflective reasoning, which were subsequently consolidated through consensus into a final vocabulary list. b, Frequency of the term ‘wait’ in model outputs over the course of training. This term was virtually absent during the initial training stages, appeared sporadically between steps 4,000 and 7,000 and exhibited a marked increase in frequency after step 8,000. These trends suggest the emergence of temporal reasoning or self-monitoring behaviour as training progresses." "5 Methods We fine-tune reasoning models with GRPO across evaluation domains (§5.1); collect and annotate reasoning traces during training (§5.2); and estimate model uncertainty to trigger entropy-based interventions (§5.3). 5.1 Models and Training Motivated by claims of mid-trace “Aha!” behavior in DeepSeek-R1 (Guo et al., 2025), we adopt Group Relative Policy Optimization (GRPO) (Shao et al., 2024) as our fine-tuning method. GRPO is an RLHF-style algorithm (Ouyang et al., 2022) that optimizes chain-of-thought generation by comparing groups of sampled completions and ex- tends PPO (Schulman et al., 2017) with group-normalized advantages and KL regularization to a frozen reference policy. Full implementation details appear in App. A.4. We fine-tune Qwen2.5 (Qwen Team, 2024) and Llama 3.1 (Grattafiori et al., 2024) models on each domain for up to 1,000 steps. Our primary experiments use Qwen2.5-1.5B trained for 1,000 steps (≈2.5–3 epochs per domain), while larger models (Qwen2.5-7B and Llama 3.1-8B) are evaluated at 500 steps due to compute constraints. To verify that models improve during training, we evaluate at multiple checkpoints and report accuracy at initialization (Step 0) and at the final evaluated checkpoint (Step 950 for the 1.5B runs; Step 500 for 7B/8B). Table 1 summarizes coverage and accuracy gains. We use lightweight, task-specific prompts that"
January 5, 2026 at 11:29 PM