In an October 20, 2026 post on Hugging Face, IBM Research reported that guidelines learned automatically from an agent’s own training runs and appended to its system prompt lifted a stock GPT-4.1 customer-service agent to 67.5% Pass^4 on τ²-bench’s retail domain, above every retail entry in the team’s October 12, 2026 leaderboard snapshot.
The result comes from ALTK-Evolve, the lab’s method for building what the authors describe as a learned playbook. The method analyzes an agent’s attempts on training tasks and uses the same model to distill successes and failures into short behavioral rules, covering situations such as authenticating a customer, confirming a transaction, or handling a return. According to the post, the same model powers both guideline extraction and the agent itself, and the process requires no weight updates or human annotation. The team previously demonstrated the approach on coding agents; the new study applies it to customer service. The post lists eleven IBM Research authors, among them Ashwath Vaithinathan Aravindan, Vatche Isahagian, and Vinod Muthusamy.
How the Study Was Run
τ²-bench, maintained by Sierra Research, is a simulation framework for evaluating customer-service agents across domains. According to the τ²-bench repository, each domain specifies a policy the agent must follow, a set of tools, and a set of evaluation tasks, and the benchmark’s core paper describes a dual-control environment. IBM’s study uses the retail and airline domains. Each task pairs the agent with an LLM-simulated customer, domain tools, and a written policy document; a task counts as solved only when the final database state matches the expected outcome, which the post describes as a strict binary reward.
The evaluation ran τ²-bench’s own implementation of a basic ReAct agent with a two-phase THINK→ACT loop, using gpt-4.1 as the agent model and gpt-5.5 as the customer-simulator model, with four trials per task. The only modification was a plain markdown list of learned guidelines appended to the system prompt after the domain policy. Published examples include authenticating the user before any order-related action, requiring explicit user approval before database-updating actions, providing itemized refund breakdowns, taking one action per step, and parsing combined name handles before user lookups.
The guidelines were extracted from the agent’s trajectories on τ²-bench train tasks using gpt-4.1 and applied unchanged on the held-out test set. Each guideline carries a support count, the number of distinct training tasks it was extracted from. The study compares three configurations: a baseline with no guidelines; sup3, which keeps only guidelines with support of at least three (31 in retail, 14 in airline); and lossless, which keeps every retained guideline (292 in retail, 101 in airline).
Held-Out Results in Retail and Airline
The study tracks two metrics across the four trials per task: Mean Accuracy, the average success rate, which the authors frame as the capability axis, and Pass^4, the percentage of tasks solved on all four trials, framed as the reliability axis and identified as the metric the τ²-bench leaderboard reports.
On 40 held-out retail test tasks, IBM reports the baseline agent scored 78.1% mean accuracy and 57.5% Pass^4. The sup3 core scored 78.8% and 60.0%, and the lossless configuration scored 82.5% and 67.5%, stated as gains of 4.4 and 10.0 percentage points over baseline.
On 20 held-out airline test tasks, the baseline scored 57.5% accuracy and 35.0% Pass^4. The sup3 core reached 60.0% accuracy and 45.0% Pass^4, while lossless reached 65.0% accuracy and 40.0% Pass^4. The two domains disagreed on dosage: retail improved most with the full lossless set, while in airline the compact sup3 core produced the best Pass^4. IBM states that support count is a knob worth tuning per domain rather than a value to hard-code.
Cost, a Frontier Rerun, and Leaderboard Status
Total LLM cost per conversation on the full base split rose with prompt size: in retail from $0.084 at baseline to $0.139 with lossless, stated as a 65% increase, and in airline from $0.081 to $0.107, stated as 32%; the sup3 variants cost $0.095 and $0.097 respectively. On matched runs solved under both baseline and guideline settings, conversations ran slightly shorter with guidelines: retail fell from 10.59 to 10.16 assistant turns under lossless, and airline from 9.15 to 8.82, with no variant lengthening them. IBM states that fewer turns mean lower end-to-end latency for the customer, and that the added input cost is largely reclaimable in production through prompt caching because the guideline block is identical on every call.
The team also reran the airline domain with gpt-5.5 as both agent and guideline-extraction model, re-mining guidelines from that agent’s own runs. No extracted guideline reached support of at least three, so the high-support core was sup2 with 103 guidelines, alongside a 146-guideline lossless set. On the same held-out airline tasks, the gpt-5.5 baseline scored 72.5% accuracy and 55.0% Pass^4; sup2 reached 80.0% and 70.0%; and lossless reached 82.5% and 70.0%, stated as gains of 10.0 accuracy points and 15.0 Pass^4 points. IBM reports the lossless result reproduced exactly across reruns.
Leaderboard submissions are scored on the full base split, train and test tasks combined, and reported at Pass^4, so those figures differ from the held-out numbers above. Against its October 12, 2026 snapshot, IBM reports its retail agent ranks first at 67.5%, ahead of RAFT-30B-A3B at 61.4%, a model it identifies as fine-tuned on synthetic τ²-bench retail data, by a stated 6.1 points. On airline, it reports the gpt-5.5 agent ranks first at 82.0%, above frontier entries it lists at 66–72%, while the gpt-4.1 sup3 variant placed eighth at 44%, ahead of GPT-5.2 running without reasoning and nearly matching Claude Sonnet 4.5. IBM states the approach improves stock agents without weight updates, human annotation, or test-set leakage.
The retail leaderboard submission was filed as a τ²-bench pull request on September 14, 2026, reporting Pass^1 of 81.14% through Pass^4 of 67.54% across 114 base-split tasks at four trials, at an average cost of $0.1084 per conversation on tau2-bench v1.0.1, and noting that the guidelines were learned from a retail train split that the base evaluation split includes. The airline leaderboard submission followed on September 21, 2026, reporting Pass^1 of 88.50% through Pass^4 of 82.00% across 50 tasks at an average cost of $0.1690 per conversation. Both submissions disclose modified prompts, state that they expect to be listed as Unverified, and remain open.
Credit: Source link
