Blog

RLVR: how verifiable rewards train reliable, auditable LLMs

Si Chen, Head of Strategy
February 2026 · Updated October 2026

RLVR trains LLMs on pass/fail checks such as unit tests and JSON schemas; learn how it works, where it fits vs RLHF, and how to implement it.

Optimizing a language model against a learned reward model has a known ceiling: past a certain point, the proxy reward keeps rising while true output quality declines (Gao et al., 2023). Reinforcement learning with verifiable rewards (RLVR) replaces that learned proxy with a deterministic check, rewarding a large language model (LLM) only when its output passes a test such as an exact-match math answer, a unit test, or a JSON schema. This article explains what RLVR is, how it compares with reinforcement learning from human feedback (RLHF), how the training loop works, and how to implement it with verifiers you can audit.

What is RLVR?

RLVR (reinforcement learning with verifiable rewards) is a post-training method that rewards a language model only when its output passes an automated, deterministic check, such as a correct math answer, passing unit tests, or a valid JSON schema. Because the reward comes from a rule rather than human preference, it is reproducible, scalable, and auditable.

The term was introduced in the Tülu 3 post-training recipe from the Allen Institute for AI (Lambert et al., 2024). Rule-based accuracy rewards are also central to reasoning models such as DeepSeek-R1 (DeepSeek-AI et al., 2025). The reward comes from a verifier: code that inspects an output and returns pass or fail.

  • Exact-match answer checks: The verifier parses the final answer from a math or logic response and compares it with a reference answer. Only exact matches earn reward (Cobbe et al., 2021).
  • Unit tests: Generated code runs against test cases in a sandbox. Functional correctness is usually reported as pass@k across k samples (Chen et al., 2021).
  • Schema and field validators: These enforce a machine-readable structure plus cross-field rules, such as line items summing to an invoice total.
  • Instruction-constraint checks: These verify explicit, countable instructions such as "answer in under 100 words" or "use exactly three bullet points" (Lambert et al., 2024).
  • Citation resolution: This confirms that cited sources resolve and contain the claimed passage, a retrieve-and-critique pattern explored in Self-RAG (Asai et al., 2023).

For tasks with no deterministic answer, such as tone or helpfulness, see our guide to rubric-based LLM evaluation.

RLVR vs RLHF: why verifiable rewards matter

RLHF trains a reward model on human preference rankings and then optimizes the policy against it. That suits open-ended qualities such as helpfulness and tone (Ouyang et al., 2022). RLVR drops the learned reward model for tasks where correctness can be computed. That changes the properties engineering and compliance teams care about most.

Dimension RLHF (human preference reward) RLVR (verifiable reward)
Reward source Learned reward model trained on ranked comparisons Deterministic verifier: tests, parsers, validators
Reproducibility Scores shift with the rater pool and reward-model version Same output plus same verifier version gives the same result
Over-optimization risk Proxy reward can rise while true quality falls (Gao et al., 2023) Limited by verifier coverage; gaps in the checks can still be exploited
Scalability Needs ongoing preference labeling Scales with compute once the verifier and golden set exist
Auditability A reward score doesn't explain why one output won Pass/fail logs for each check show which rule passed or failed
Best fit Helpfulness, tone, nuanced safety policy Math, code, structured output, instruction constraints

Most teams use both: RLVR where a check exists, preference-based methods where it doesn't. For the preference side, see our guide to reinforcement learning from human feedback.

RLVR limitations and failure modes

  • Reward hacking: The model learns to satisfy the verifier rather than the task, for example by hard-coding expected test outputs or exploiting a lenient parser (Skalse et al., 2022). The worked example below shows how a partial verifier inflates reward.
  • Verifier false positives: Any check that passes a wrong output turns that error into a training signal. False positives cost more than false negatives because they actively reward the wrong behavior.
  • Narrow task coverage: Deterministic verifiers mostly cover math, code and structured output. Tasks without a reference answer still need human or model-based judgment.
  • Contested reasoning gains: Yue et al. (2025) found RLVR-trained models beat their base models at pass@1, but base models matched or exceeded them at large k. That suggests RLVR mainly makes the model more likely to sample answers it could already find. Wen et al. (2025) counter that when chain-of-thought correctness is also scored (CoT-pass@k), RLVR does increase correct reasoning.

These failure modes are why verifier design still depends on expert review, the same principle behind human in the loop data pipelines and AI safety evaluation.

How does RLVR work?

Each training batch goes through four stages:

  • Sample: For each prompt, the policy generates a group of candidate responses, for example k = 8.
  • Verify: Each candidate runs through the verifier stack and gets a reward, usually 1 for pass and 0 for fail. Some setups add a separate format reward (DeepSeek-AI et al., 2025).
  • Score against the group: Group relative policy optimization (GRPO) normalizes each reward against the group's mean and standard deviation, so no separate value model is needed (Shao et al., 2024).
  • Update: The policy shifts probability toward above-average candidates. A Kullback-Leibler (KL) penalty keeps it close to the reference model.

One consequence matters in practice. If every candidate in a group passes, or every candidate fails, all advantages are zero and that prompt teaches the model nothing. Prompt difficulty therefore needs calibrating (step 4 below). At scale, this loop raised DeepSeek-R1-Zero's AIME 2024 pass@1 from 15.6% to 77.9% using only rule-based accuracy and format rewards (DeepSeek-AI et al., 2025).

Worked example: invoice extraction

Prompt: "Extract vendor, invoice_date, line_items, and total from this invoice as JSON." The reference total is $1,284.50 including tax. The policy produces 8 samples, and four verifiers run in sequence:

Sample Schema valid Date in ISO 8601 Line items + tax = total Total matches reference Reward
1 Pass Pass Pass Pass 1
2 Pass Pass Pass Pass 1
3 Pass Fail (03/04/26) Not checked Not checked 0
4 Fail (missing field) Not checked Not checked Not checked 0
5 Pass Pass Fail (tax dropped: $1,190.00) Fail 0
6 Pass Pass Pass Pass 1
7 Pass Pass Pass Fail (line item misread) 0
8 Fail (prose after JSON) Not checked Not checked Not checked 0
  • Group statistics: 3 of 8 samples pass, so the mean reward is 0.375 and the standard deviation is 0.484.
  • Advantages: Passing samples get +1.29 and failing samples get −0.77, pushing the policy toward samples 1, 2 and 6.
  • The partial-verifier trap: If the stack checked only the schema, 6 of 8 samples would score 1. The mean reward would double to 0.75, and sample 5's tax-dropping habit would be rewarded.

Closing verifier gaps like this is one of the most direct levers for reducing hallucinations in structured outputs.

RLVR use cases: where verifiable rewards fit

RLVR use case Verifier Pass criterion Human input required
Math and quantitative reasoning Answer parser plus exact match Final answer equals the reference Expert-written problems and reference solutions
Code generation Sandboxed unit tests All tests pass Engineer-written tests, including edge cases
Structured extraction (invoices, forms, claims) Schema plus cross-field validators Schema valid and all rules hold Golden documents with adjudicated field values
Text-to-SQL Run the query and compare result sets Result set equals the reference Analyst-written reference queries
Instruction following Constraint checkers Every stated constraint is satisfied Prompt authors who write checkable constraints
Agentic tool use Environment state check The task's end state is reached Task designers who define success states

Agent tasks extend RLVR from single answers to multi-step agentic workflows, where the verifier checks the final state of the environment.

How to implement RLVR: a seven-step checklist

The thresholds below are starting points; adjust them to your risk tolerance.

  1. Qualify the task: Sample 200 outputs and count how many a deterministic checker can score without human judgment. Below about 80%, route the remainder to RLHF or rubric-based evaluation.
  2. Build the golden set before the verifier: Assemble 300-1,000 expert-solved items, each annotated twice and adjudicated. Release only batches with Cohen's κ ≥ 0.80, the threshold of the "almost perfect" agreement band (0.81-1.00; Landis & Koch, 1977).
  3. Stress-test the verifier: Run it on the golden set plus adversarial near-misses. Require a false-positive rate ≤ 1% and a false-negative rate ≤ 5% before training starts.
  4. Filter prompts by difficulty: Draw 8 samples per prompt from the base model. Keep prompts with a pass rate between 1/8 and 7/8; the rest produce zero advantage under GRPO.
  5. Start with a minimal reward: Use binary correctness plus a format gate. Add partial credit only after showing it ranks golden-set outputs the same way experts do.
  6. Monitor for reward hacking: Track held-out pass@1, pass@8, verifier pass rate and response length at every checkpoint. Have experts audit 100-200 passing outputs per checkpoint. Pause training if training reward rises for two checkpoints while expert-audited accuracy stays flat or falls.
  7. Version and log everything: Store the verifier code hash, golden-set version and per-sample check results. These records support the MEASURE function of the NIST AI Risk Management Framework (NIST, 2023) and the technical documentation (Article 11 and Annex IV) and record-keeping (Article 12) duties for high-risk systems under the EU AI Act (European Parliament and Council, 2024).

Steps 2, 3 and 6 take the most expert labor. See how Appen's RL environment and reward design services support them.

How Appen supports RLVR

RLVR moves human effort away from labeling preferences and toward building the assets verifiers depend on. Appen supports research and engineering teams on the following:

  • Expert-written prompts and reference answers: Contributors are recruited by domain (mathematics, software engineering, finance), region and language, so prompt sets reflect the locales the model will face. Each item ships with a reference answer, a worked solution and a difficulty tag that feeds step 4.
  • Golden sets with measured agreement: Each golden item is annotated twice and adjudicated. Deliverables report Cohen's κ per batch, and batches below 0.80 are re-adjudicated before release.
  • Verifier stress tests: Contributors write adversarial near-misses, such as right format with the wrong value or code that passes the visible tests but fails edge cases. This measures the verifier's false-positive rate before any training compute is spent.
  • Quality control in ADAP: In the Appen Data Annotation Platform (ADAP), test questions with known answers gate entry to a task (quiz mode) and are embedded in live work (work mode). Contributors who fall below the accuracy threshold are flagged, and the job manager can remove them and their work from the job.
  • Checkpoint audits: Domain experts review samples of passing outputs at each checkpoint to catch reward hacking that automated metrics miss (step 6).
  • Off-the-shelf datasets: Where an existing Appen dataset matches the target language or domain, it can seed evaluation sets and shorten the time to a first working verifier. The catalog also lists RL tasks and verifiers, including 500 coding tasks each paired with an automated verifier.

Deliverables are versioned JSONL files containing the prompt, reference answer, verifier specification, adjudication record and contributor qualification metadata. That maps directly to the logging in step 7. Appen's off-the-shelf catalog covers 80+ languages, and Appen collects data across 500+ global locales.

Discuss your RLVR data requirements with an Appen specialist →

Frequently asked questions about RLVR

Is RLVR better than RLHF?

Neither replaces the other. Tülu 3 runs supervised fine-tuning, then direct preference optimization (DPO), then RLVR as the final stage (Lambert et al., 2024). DeepSeek-R1 uses rule-based rewards for reasoning data but relies on learned reward models for general helpfulness and harmlessness in its final reinforcement learning stage (DeepSeek-AI et al., 2025). Choose per task: verifiable checks where they exist, preference signals where they don't.

Does RLVR work outside math and code?

Increasingly, yes. Su et al. (2025) extended RLVR to free-form answers in fields such as medicine, chemistry, psychology, economics and education. They used expert-written reference answers and a generative model that scores how well a response matches them, and showed that a relatively small 7B model can be trained as this cross-domain reward model. The trade-off is that a model-based judge is less deterministic than a unit test, so measure its agreement with expert graders before training.

How much training data does RLVR need?

Often less than teams expect. Wang et al. (2025) reported that RLVR on a single training example raised Qwen2.5-Math-1.5B from 36.0% to 73.6% on the MATH500 benchmark. That result depends on a strong base model and a carefully chosen example. Production systems still need a broad, difficulty-calibrated prompt set, but verifier precision matters more than raw volume.

What is the difference between RLVR and a process reward model?

RLVR usually scores the outcome: is the final answer or end state correct? A process reward model (PRM) scores each reasoning step. Lightman et al. (2023) trained a PRM on PRM800K, about 800,000 human step-level labels, and found process supervision beat outcome supervision on MATH problems. Process rewards catch "right answer, wrong reasoning" cases that outcome checks miss, at much higher labeling cost.

Can an RLVR-trained model still hallucinate?

Yes. RLVR only constrains what the verifier checks. A model rewarded for correct final answers can still produce unsupported explanations, invented citations or confident errors on unfamiliar inputs. Useful mitigations include extending verifiers to cover citations and intermediate claims, and measuring calibration (whether stated confidence tracks accuracy) on test sets that differ from the training domain.

Have a question we didn't cover? Talk to an Appen RLVR specialist →

References

Tags:
RLHF
Alignment
Compliance
Talk to our team