Scale AI and Singapore’s Infocomm Media Development Authority (IMDA) introduced SG-LegalClarify on November 12, 2026, testing 19 language models on 1,860 scenarios across 10 domains of Singapore law. The benchmark examines both legal accuracy and whether models ask for missing facts before answering.
Scale describes SG-LegalClarify as the first research initiative under a collaboration formalized in April 2026 and publicly described on July 7, 2026. The project is intended to help governments assess AI systems against their own laws and institutions, rather than assume that results on international benchmarks transfer to every national setting.
The research paper presents Singapore as a demanding test because inherited English common law operates alongside local modifications and Singapore-specific statutes. A model relying on generic common-law reasoning can therefore miss the governing local rule. SG-LegalClarify also tests a separate skill: deciding whether a query contains enough legally material information to answer at all.
That distinction matters in ordinary queries. A driver asking about a speeding penalty may need to supply the vehicle type, actual speed and existing demerit points. An assistant that gives a penalty without those details can produce an authoritative-sounding but legally incorrect answer.
Paired Questions Test When Models Ask
The dataset begins with 1,100 English-language source questions: 340 fully specified and 760 underspecified. Each underspecified question has a fully specified counterpart containing its hidden facts. Twenty-three verified lawyers authored the questions, with seven also serving as reviewers. Their legal experience ranged from two to 37 years, with a median of six.
Each question has a lawyer-written rubric covering issue identification, rule recall, application and conclusion, with 12,546 criteria across the dataset. Of those criteria, 89.0% are mandatory for a substantively correct answer; the headline results count only mandatory items.
A sufficiency gate first makes each model choose between answering directly and gathering facts. On the fact-gathering path, a simulated user reveals hidden facts only when the model explicitly requests them. A three-model AI judge panel, selected through comparisons with human lawyer assessments, grades each criterion by majority vote.
Search Improved Scores but Clarification Remained Weak
The paper reports that Claude Fable 5 led closed-book testing by satisfying 69.8% of mandatory criteria, against an average of 53.8%. With web search, Kimi K3 led at 77.8% and the average rose to 63.0%; every model improved, although the gains varied. Rule recall was the weakest grading component, with pass rates of 43.2% without search and 56.5% with it.
Scale’s summary reports that approximately 39% of closed-book scenarios with missing facts received no clarifying question, while models asked unnecessary questions in approximately 28% of scenarios that already contained enough information. When facts were missing, models asked about an average of only 39% of the relevant facts identified by legal experts.
In a descriptive comparison of web-enabled trials where models requested every relevant missing fact, the paper reports that 55.6% reached the correct legal conclusion, 32.1% reached an incorrect conclusion and 12.3% provided no conclusion. Complete information gathering therefore did not ensure correct application of the law.
Evaluation Limits and Release Plans
The authors identify several limits to the results: human validation of the AI helpers was limited, provider differences prevented identical search-tool access, and upfront routing may have altered clarification behavior. Not every annotated missing variable is necessary to resolve an instance, so incomplete fact-request coverage does not automatically indicate inadequate clarification.
The researchers plan to release 50% of the source questions in each category while retaining the rest for private evaluation, keeping every underspecified–fully specified pair in the same split. Model outputs on the public split and the evaluation pipeline are also planned releases.
The paper’s ethics statement says the scores should not be interpreted as evidence that a model is safe for unsupervised legal advice. For governments evaluating legal AI, the findings make clarification part of the accuracy question: a system must recognize which facts local law makes material, then apply that law correctly after obtaining them.
Credit: Source link
