LongICLBench Benchmark: Evaluating Large Language Models on Long In-Context Learning for Extreme-Label Classification

The processing of long textual sequences, which is critical for numerous applications, including question-answering systems and document summarization, has shown remarkable progress in large language models (LLMs). These models can understand and generate text based on large contexts. Still, their effectiveness in comprehending extremely long sequences and performing tasks with numerous possible outcomes remains an area requiring further exploration.

Existing research includes advancements in Transformer models for handling long sequences, notably through AliBi and RoPE embeddings, facilitating context window extensions during inference. Innovations like LongRoPE aim to expand context windows up to 2M tokens. Techniques such as sliding memory windows and segmentation address the computational challenges of long inputs. Models like RWKV and Mamba introduce RNN-like architectures to process extended sequences with reduced complexity efficiently. In contrast, selective state-space models emerge as promising alternatives for natural long-range computations.

Researchers from the University of Waterloo, Carnegie Mellon University, and Vector Institute, Toronto, have introduced LongICLBench, a benchmark specifically developed for evaluating LLMs in processing long-context sequences for extreme-label classification tasks. Its uniqueness lies in the comprehensive testing across six datasets with varied difficulty levels and extensive input lengths, offering a nuanced perspective on LLMs’ performance in real-world scenarios.

The methodology centres on evaluating 13 long-context LLMs against LongICLBench, which comprises six datasets: GoEmotion, BANKING77, TacRED, Few-NERD, DialogRE, and Discovery, covering input lengths from 2K to 50K tokens and label ranges from 28 to 174 classes. The benchmark tests each model’s ability to process extensive sequences and recognize vast label spaces accurately. Performance is measured regarding models’ comprehension of the entire input to make correct predictions, highlighting their capabilities and limitations in long in-context learning. This structured evaluation offers a detailed understanding of current LLM performance across complex classification tasks.

Models showed varying performance across datasets, with a significant decline in accuracy as task complexity increased. For instance, all models struggled on the Discovery dataset featuring 174 labels, with performance nearing zero accuracies in the most challenging scenarios. However, on less complex tasks such as BANKING77, with input lengths ranging from 2K to 14K tokens, models like GPT4-turbo and RWKV-5-World achieved higher accuracies at 84.4% and 32.6%, respectively. The detailed analysis revealed a general trend: while LLMs can process longer contexts up to 20K tokens with relative success, their ability to understand and reason over these sequences diminishes significantly with further complexity and input length.

To conclude, the research introduced LongICLBench, a novel benchmark for evaluating the efficacy of Large Language Models (LLMs) in long in-context learning for extreme-label classification tasks. Rigorous testing across various models and datasets revealed that while LLMs perform adequately on simpler tasks, their ability to process and understand longer, more complex sequences still needs to be improved. These findings underscore the need for continued development in LLM capabilities, highlighting the benchmark’s role in advancing our understanding of LLM performance in handling real-world, complex tasks.

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.

If you like our work, you will love our newsletter with 24k+ members…

Don’t Forget to join our 40k+ ML SubReddit