LangChain’s Align Evals closes the evaluator trust gap with prompt-level calibration

As enterprises increasingly turn to AI models to ensure their applications function well and are reliable, the gaps between model-led evaluations and human evaluations have only become clearer. 

To combat this, LangChain added Align Evals to LangSmith, a way to bridge the gap between large language model-based evaluators and human preferences and reduce noise. Align Evals enables LangSmith users to create their own LLM-based evaluators and calibrate them to align more closely with company preferences. 

“But, one big challenge we hear consistently from teams is: ‘Our evaluation scores don’t match what we’d expect a human on our team to say.’ This mismatch leads to noisy comparisons and time wasted chasing false signals,” LangChain said in a blog post. 

LangChain is one of the few platforms to integrate LLM-as-a-judge, or model-led evaluations for other models, directly into the testing dashboard. 

The company said that it based Align Evals on a paper by Amazon principal applied scientist Eugene Yan. In his paper, Yan laid out the framework for an app, also called AlignEval, that would automate parts of the evaluation process. 

Align Evals would allow enterprises and other builders to iterate on evaluation prompts, compare alignment scores from human evaluators and LLM-generated scores and to a baseline alignment score. 

LangChain said Align Evals “is the first step in helping you build better evaluators.” Over time, the company aims to integrate analytics to track performance and automate prompt optimization, generating prompt variations automatically. 

How to start 

Users will first identify evaluation criteria for their application. For example, chat apps generally require accuracy.

Next, users have to select the data they want for human review. These examples must demonstrate both good and bad aspects so that human evaluators can gain a holistic view of the application and assign a range of grades. Developers then have to manually assign scores for prompts or task goals that will serve as a benchmark. 

Developers then need to create an initial prompt for the model evaluator and iterate using the alignment results from the human graders. 

“For example, if your LLM consistently over-scores certain responses, try adding clearer negative criteria. Improving your evaluator score is meant to be an iterative process. Learn more about best practices on iterating on your prompt in our docs,” LangChain said.

Growing number of LLM evaluations

Increasingly, enterprises are turning to evaluation frameworks to assess the reliability, behavior, task alignment and auditability of AI systems, including applications and agents. Being able to point to a clear score of how models or agents perform provides organizations not just the confidence to deploy AI applications, but also makes it easier to compare other models. 

Companies like Salesforce and AWS began offering ways for customers to judge performance. Salesforce’s Agentforce 3 has a command center that shows agent performance. AWS provides both human and automated evaluation on the Amazon Bedrock platform, where users can choose the model to test their applications on, though these are not user-created model evaluators. OpenAI also offers model-based evaluation.

Meta’s Self-Taught Evaluator builds on the same LLM-as-a-judge concept that LangSmith uses, though Meta has yet to make it a feature for any of its application-building platforms. 

As more developers and businesses demand easier evaluation and more customized ways to assess performance, more platforms will begin to offer integrated methods for using models to evaluate other models, and many more will provide tailored options for enterprises.