AI21 Labs released LIFT-Bench on October 25, 2026, an instruction-following benchmark in which every constraint is scoped to one section of a document and the task content itself is scored, alongside LIFT-Train, a 34,258-task training counterpart, and the open pipeline that generated both.
A companion paper, “LIFT: Scoped Constraints for Evaluating and Training Instruction Following” by Eran Goldstein, Yuval Belfer, Yonatan Belinkov, and Daniel Gissin, was submitted to arXiv on October 20, 2026. Its abstract states that common instruction-following benchmarks such as IFEval evaluate constraints that can easily be verified programmatically, that those same constraints then serve as reward signals for reinforcement learning, and that this design can lead to reward hacking that pushes models away from the main task.
The Training Run Behind the Benchmark
In a blog post published with the release, AI21 reports that the project began with a real experiment that looked successful on paper. Using IFTrain, which the post describes as the go-to open dataset for instruction following, as the source of RL rewards, the run drove the reward to 1.0 and raised IFEval by 5 points and IFBench by 16 points. Yet when the model was asked for a chapter of alternate history about the Xinhai Revolution, with no word repeated more than four times, it answered with a single word: “China”.
The failure was systematic rather than a fluke, according to the company: by step 390, the median answer had shrunk from 114 words to 18, and 38% of answers scored at least 0.8 in under 30 words. AI21 attributes the collapse to the structure of the reward: IFTrain’s checkers, like IFEval’s, verify constraints with code against the whole response, and nothing scores whether the task was done, so dropping the task costs the model nothing. Existing patches do not close that gap, the company says: IFBench adds a reward-model classifier that scores general quality rather than the specific task, and AdvancedIF trains on content rubrics but keeps its rubric generator, training data, and RL code closed.
Scoped Constraints and a Validated Judge
LIFT-Bench attaches each constraint to one section of a document outline and scores it only against that section’s text, a design AI21 calls scoped constraints. Each section carries two kinds of constraints: verifiable ones checked by code, such as requiring a section in German or a keyword used at least three times, and judged ones checked by an LLM that spell out what the section must cover. Judged constraints count in the same score as formatting. Scoping also turns constraint density, the number of constraints attached to a single section, into a parameter set during task generation.
All tasks are synthetic: topics come from Wikipedia, outlines and constraints from an LLM, and a sampler controls how many constraints land on each section. The benchmark holds 784 test tasks across ten splits: short documents of about four sections and long documents of about nine sections, each at four density levels, plus two flat splits with no outline, in which every constraint applies to the whole response in the manner of existing benchmarks such as IFEval.
Because the same judge scores the benchmark and hands out the RL reward, AI21 tested candidate judges against 305 human-labelled examples before choosing one. The company reports that an unscoped Gemma-4-31B judge accepted 292 of those examples when only 227 were actually satisfied, and that an unscoped Qwen2.5-32B scored 0.44 macro-F1, roughly the same as the 0.43 scored by always answering yes. Scoping the judge to the text each constraint governs added 11 to 19 macro-F1 points, the largest effect of any variable tested, and AI21 selected scoped Gemma-4-31B. The released splits write google/gemma-4-31B-it into each task as the judge, and the repository cautions that scores are comparable only across runs that used the same judge. Structured splits give structure constraints 10% of the score weight and content constraints 90%; flat tasks carry no structure constraints, so content takes the full weight.
What the Benchmark Shows About Current Models
AI21 evaluated 17 models with eight samples per task. On structured tasks, the company reports, GPT-5.5 averages 0.96, while on flat tasks no model passes 0.85 and the leading closed models average between 0.65 and 0.72. The paper’s authors likewise state that frontier models struggle with the benchmark’s harder subsets. The structured-versus-flat gap is not a matter of constraint load, according to AI21: restricted to tasks with ten constraints or fewer, every model but Qwen3.8-A95B still scores lower on flat tasks, and the outline points that only structured tasks can earn account for 10% of the score.
Rankings shift between the two regimes as well, the company reports. Among the nine main models tracked, GPT-5.5 ranks first on structured tasks and fifth on flat ones, and the two best overall averages belong to open-weight models Kimi K3 and DeepSeek V4 Pro. Holding one section’s constraints fixed while adding constraints to the rest of the document, Kimi K3 loses 0.4% per density step and GPT-5.5 loses 1.5%, while GPT-5 nano loses 5.5%.
Training With the Benchmark Score as the Reward
LIFT-Train applies the benchmark’s score unchanged as the RL reward across 34,258 tasks from the same generator; the repository notes that the same constraint_reward scalar serves as both the benchmark metric and the reward, with no separate reward model. AI21 trained Qwen2.5-7B with GRPO on LIFT-Train and on IFTrain. On IFTrain, the reward hit 1.0 by step 180 while the median answer fell from 114 words to 18 by step 390. On LIFT-Train, the median answer stayed around 390 to 530 words, and no high-scoring answer came in under 30 words.
A flat-only slice of LIFT-Train, matching IFTrain’s surface format but keeping content in the reward, also trained cleanly. AI21 says this points to content in the reward, rather than format, as the deciding factor, while noting that the datasets’ topics still differ, Wikipedia versus WildChat. Across five benchmarks after 180 GRPO steps, each recipe won its own benchmark; on AdvancedIF and ComplexBench, which also score content, IFTrain slipped while LIFT-Train held steady. AI21 characterizes the small deltas as avoiding a regression rather than a general win.
Training Qwen3-32B on each scope separately and on a mixture, structured-only training reached 0.805 on flat tasks and flat-only training reached 0.798 on structured tasks, against 0.896 and 0.919 for the mixture, while an easy-to-hard curriculum made no difference, AI21 reports. In its summary, the company advises teams training for instruction following to measure the task rather than only the rules, to check the judge against human labels, to pick the base model on the scoped evaluation rather than a flat leaderboard, and to use the evaluation as the reward while tracking answer length alongside the reward curve.
Availability
LIFT-Bench, LIFT-Train, and the synthesis pipeline are available on GitHub under an Apache-2.0 license, with benchmark splits hosted on the Hugging Face Hub as ai21labs/LIFTBench. Tasks are emitted in the format consumed natively by PrimeIntellect’s verifiers library for evaluation and RL training, and the repository publishes a five-split prime-rl training mixture of 17,129 tasks. Verifiable constraint types cover language, case convention, keyword count, emoji count, and outline-matching document structure.
Credit: Source link
