Meet LegalBench: A Collaboratively Constructed Open-Source AI Benchmark for Evaluating Legal Reasoning in English Large Language Models

American attorneys and administrators are reevaluating the legal profession due to advances in large language models (LLMs). According to its supporters, LLMs might change how attorneys approach jobs like brief writing and corporate compliance. They may eventually contribute to resolving the long-standing access to justice dilemma in the United States by increasing the accessibility of legal services. This viewpoint is influenced by the finding that LLMs have unique qualities that make them more equipped for legal work. The expenditures associated with manual data annotation, which often add the expense to the creation of legal language models, would be reduced by the models’ ability to learn new jobs from small amounts of labeled data. 

They would also be well suited for the rigorous study of law, which includes deciphering complex texts with plenty of jargon and engaging in inferential procedures that integrate several modes of thinking. The fact that legal applications frequently involve high risk dampens this enthusiasm. Research has demonstrated that LLMs can produce offensive, deceptive, and factually wrong information. If these actions were repeated in legal contexts, they might cause serious damages, with historically marginalized and under-resourced people bearing disproportionate weight. Thus, there is an urgent need to build infrastructure and procedures for measuring LLMs in legal contexts due to the safety implications. 

However, practitioners who want to judge whether LLMs can use legal reasoning confront major obstacles. The small ecology of legal benchmarks is the first obstacle. For instance, most current benchmarks concentrate on tasks that models learn by adjusting or training on task-specific data. These standards do not capture the characteristics of LLMs that inspire interest in law practice—specifically, their capacity to complete various tasks with just short-shot prompts. Similarly, benchmarking initiatives have centered on professional certification examinations like the Uniform Bar Exam, although they don’t always indicate real-world applications for LLMs. The second issue is the discrepancy between how attorneys and established standards define “legal reasoning.” 

Currently used benchmarks broadly classify any work requiring legal information or laws as assessing “legal reasoning.” Contrarily, attorneys are aware that the phrase “legal reasoning” is wide and encompasses various sorts of reasoning. Various legal responsibilities call for different abilities and bodies of knowledge. It is challenging for legal practitioners to contextualize the performance of contemporary LLMs within their sense of legal competency since existing legal standards need to identify these differences. The legal profession does not employ the same jargon or conceptual frameworks as legal standards. Given these restrictions, they think that to rigorously assess the legal reasoning skills of LLMs, the legal community will need to become more involved in the benchmarking process.

To do this, they introduce LEGALBENCH, which represents the initial stages in creating an interdisciplinary collaborative legal reasoning benchmark for English.3 The authors of this research worked together over the past year to construct 162 tasks (from 36 distinct data sources), each of which tests a particular form of legal reasoning. They drew on their various legal and computer science backgrounds. So far as they are aware, LEGALBENCH is the first open-source legal benchmarking project. This method of benchmark design, in which subject matter experts actively and actively participate in the development of evaluation tasks, exemplifies one kind of multidisciplinary cooperation in LLM research. They also contend that it demonstrates the crucial part that legal practitioners must perform in evaluating and advancing LLMs in law. 

They emphasize three aspects of LEGALBENCH as a research project: 

1. LEGALBENCH was built using a combination of pre-existing legal datasets that had been reformatted for the few-shot LLM paradigm and manually made datasets that were generated and supplied by legal experts who were also listed as authors on this work. The legal experts engaged in this cooperation were invited to provide datasets that either test an intriguing legal reasoning talent or represent a practically valuable application for LLMs in law. As a result, strong performance on LEGALBENCH assignments offers relevant data that attorneys may use to confirm their opinion of an LLM’s legal competency or to find an LLM that could benefit their workflow. 

2. The tasks on the LEGALBENCH are arranged into a detailed typology that outlines the kinds of legal reasoning needed to complete the assignment. Legal professionals can actively participate in debates about LLM performance since this typology draws from frameworks common to the legal community and uses vocabulary and a conceptual framework they are already familiar with. 

3. Lastly, LEGALBENCH is designed to serve as a platform for more study. LEGALBENCH offers substantial assistance in knowing how to prompt and assess various activities for AI researchers without legal training. They also intend to expand LEGALBENCH by continuing to solicit and include work from legal practitioners as more of the legal community continues to interact with LLMs’ potential effect and function.

They contribute to this paper: 

1. They offer a typology for classifying and characterizing legal duties according to the necessary justifications. This typology is based on the frameworks attorneys use to explain legal reasoning. 

2. Next, they give an overview of the activities in LEGALBENCH, outlining how they were created, significant heterogeneity dimensions, and constraints. In the appendix, a detailed description of each assignment is given. 

3. To analyze 20 LLMs from 11 different families at various size points, they employ LEGALBENCH as their last step. They give an early investigation of several prompt-engineering tactics and make remarks about the effectiveness of various models. 

These findings ultimately illustrate several potential research topics that LEGALBENCH may facilitate. They anticipate that a variety of communities will find this benchmark fascinating. Practitioners may use these activities to decide whether and how LLMs might be included in current processes to enhance client results. The varied sorts of annotation that LLMs are capable of and the various types of empirical scholarly work they permit can be of interest to legal academics. The success of these models in a field like law, where special lexical characteristics and challenging tasks may reveal novel insights, may interest computer scientists. 

Before continuing, they clarify that the goal of this work is not to assess whether computational technologies should replace solicitors and legal staff or to comprehend the advantages and disadvantages of such a replacement. Instead, they want to create artifacts to help the impacted communities and pertinent stakeholders better grasp how well LLMs can do certain legal responsibilities. Given the spread of these technologies, they think the solution to this issue is crucial for assuring the secure and moral use of computational legal tools.

Check out the Paper and Project Page. All Credit For This Research Goes To the Researchers on This Project. Also, don’t forget to join our 29k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.