CoSy (Concept Synthesis): A Novel Architecture-Agnostic Machine Learning Framework to Evaluate the Quality of Textual Explanations for Latent Neurons

Modern Deep Neural Networks (DNNs) are inherently opaque; we do not know how or why these computers arrive at the predictions they do. This is a major barrier to the broader use of Machine Learning techniques in many domains. An emerging area of study called Explainable AI (XAI) has arisen to shed light on how DNNs make decisions in a way that humans can comprehend. XAI has expanded its scope to include examining the functional purpose of each model component to explain the models’ global behavior, as opposed to just explaining how DNNs make decisions locally for specific inputs using saliency maps.

The second global explainability technique, mechanistic interpretability, is followed by methods that characterize the particular ideas neurons, which are the basic computational units in a neural network, have learned to recognize. This allows one to examine how these broad ideas impact the predictions made by the network. Labeling neurons using notions humans can understand in prose is a common way to explain how a network’s latent representations work. A neuron is given a written description according to the notions it has learned to detect or is strongly triggered by. These techniques have progressed from describing labels to offering more in-depth compositional and open-vocabulary explanations. However, the absence of a generally acknowledged quantitative metric for open-vocabulary neuron descriptions remains a substantial obstacle. The result was that many approaches came up with their evaluation standards, making it hard to conduct thorough, general-purpose comparisons.

To fill this void, researchers from ATB Potsdam, University of Potsdam, TU Berlin, Fraunhofer Heinrich-Hertz-Institute, and BIFOLD present CoSy, a groundbreaking quantitative evaluation approach for assessing computer vision (CV) models’ use of open-vocabulary explanations for neurons. This innovative method, leveraging modern developments in Generative AI, allows for the creation of synthetic visuals corresponding to the given concept-based textual descriptions. By combining data points typical for specific target explanations, the researchers have paved the way for a new era of AI evaluation. Unlike current ad hoc approaches, CoSy enables quantitative comparisons of several concept-based textual explanation methods and tests using the activations of the neurons. This breakthrough eliminates the need for human intervention, empowering users to assess the accuracy of individual neuron explanations. 

By conducting a thorough meta-analysis, the team has proven that CoSy ensures an accurate explanation evaluation. The study demonstrates through multiple studies that the last levels, where learning of high-level concepts takes place, are the best places to apply concept-based textual explanation methods. In these layers, INVERT, a technique that inverts the process of generating an image from a neural network’s internal representation, and CLIP-Dissect, a method that dissects the internal representations of a neural network, give notions of high-quality neurons. In contrast, MILAN and FALCON give explanations of lower-quality neurons that can provide concepts that are near to random, which could cause incorrect conclusions about the network. Therefore, it is clear from the data that evaluation is crucial when employing textual explanation approaches based on concepts.

The researchers highlight that the generative model is a major drawback of CoSy. For instance, the ideas produced may not be incorporated into the training of the text-to-image model. Analyzing pre-training datasets and model performance could help overcome this lack, which results in poorer generative performance. Worse yet, the model can only come up with vague ideas like ‘white objects,’ which are not specific enough to provide a comprehensive understanding. More complex, niche or limited models may be useful in both situations. Looking Ahead In the underexplored field of evaluating non-local explanation approaches, where CoSy is still in its infancy, there is a lot of promise.

The team is optimistic about the future of CoSy and envisions its application in various fields. They hope that future work will focus on defining explanation quality in a way that considers human judgment, a crucial aspect when judging the plausibility or the quality of an explanation in relation to the outcome of a downstream job. They intend to broaden the scope of their evaluation framework’s application to other fields, such as healthcare and natural language processing. The prospect of evaluating huge, opaque, autointerpretable language models (LLMs) developed recently is particularly intriguing. The researchers also believe that applying CoSy to healthcare datasets, where explanation quality is crucial, could be a significant step forward. These future applications of CoSy hold great promise for the advancement of AI research.  

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our 43k+ ML SubReddit | Also, check out our AI Events Platform