Qwen-GRPO Math Reasoner — RLAIF Fine-Tuning Pipeline

This project fine-tunes the Qwen2.5-0.5B-Instruct model on the GSM8K dataset to solve grade-school math problems[cite: 1]. The methodology utilizes the verifiers package to programmatically reward the model for correct calculations and strict tag formatting[cite: 1]. Optimized for a Kaggle environment, this targeted training successfully boosted the model's format compliance to 100% and its mathematical accuracy to 48%[cite: 1].

---

🏛️ Training Architecture

The pipeline is built on a Reinforcement Learning from AI Feedback (RLAIF) approach using Group Relative Policy Optimization (GRPO)[cite: 1]:

1. Dataset & Tokenization

• Prompt Filtering: Extracts questions from the openai/gsm8k dataset and strictly enforces a 256-token prompt budget to maintain memory efficiency[cite: 1].
• Left-Padding: Configures the tokenizer to use left-padding, ensuring batched autoregressive generation aligns correctly for GRPO[cite: 1].
• System Prompting: Instructs the model to output step-by-step reasoning inside <thinking> tags, followed strictly by the final numeric answer inside <answer> tags[cite: 1].

2. Reward Verification System

• Format Reward: Uses the verifiers XML parser to check for the presence of valid <thinking> and <answer> tags[cite: 1].
• Correctness Reward: Programmatically compares the extracted numeric answer to the deterministic GSM8K ground truth, normalizing numbers to avoid false penalties on string formatting (e.g., stripping commas and symbols)[cite: 1].
• No LLM Judge: By using deterministic programmatic verification, the training avoids the overhead and latency of using an LLM-as-a-judge[cite: 1].

---

🛠️ Technology Stack

Component	Technologies
Base Model	Qwen/Qwen2.5-0.5B-Instruct[cite: 1]
Dataset	openai/gsm8k[cite: 1]
Training Framework	trl (GRPOTrainer), transformers, accelerate[cite: 1]
Verification Engine	verifiers (XMLParser)[cite: 1]
Hardware	Kaggle T4 x2 (2 x 16GB Turing GPUs)[cite: 1]
Precision	fp16 mixed precision[cite: 1]

---

💡 Challenges & Architectural Decisions

1. Overcoming Free-Tier RAM Limitations

The Struggle: Initially, we attempted to run the fine-tuning pipeline on the free tier of Google Colab. However, compiling the model with mixed precision and retaining multiple generated completions in memory simultaneously for the policy-gradient update quickly exceeded the standard RAM limit. This caused the kernel to repeatedly crash with Out-Of-Memory (OOM) errors before the first epoch could finish.

The Decision: We migrated the entire environment to a Kaggle T4 x2 runtime, giving us access to two 16GB Turing GPUs[cite: 1]. To ensure it fit safely within this new memory ceiling, we configured the GRPOTrainer with a per-device batch size of 1, offset by gradient_accumulation_steps=8, and explicitly enabled gradient_checkpointing=True to trade compute for massive activation-memory savings[cite: 1].

2. Dependency Hell & Library Conflicts

The Struggle: Kaggle's base image ships with a pre-installed wandb package whose compiled protobuf files frequently sit out of sync with the newer protobuf versions pulled in by fresh transformers and trl installations[cite: 1]. Even when not using external logging, the library auto-probed for WandB on import, crashing the entire script with an ImportError[cite: 1].

The Decision: Instead of chasing down matching version pins and compiling dependencies from scratch, we opted for a clean bypass[cite: 1]. Since we set report_to="none" for this lightweight run, we executed a silent, forced uninstall of wandb right at the start of the environment setup, cleanly sidestepping the crash[cite: 1].