CodeEditorBench: A Machine Learning System for Evaluating the Effectiveness of Large Language Models (LLMs) in Code Editing Activities

Coding-related jobs have led to the rapid advancement of Large Language Models (LLMs), with a focus on code editing. LLMs created specifically for coding jobs are applied to a variety of activities, including code optimisation and repair. As programming tools, they are becoming more and more popular, but most evaluation techniques concentrate on code production, ignoring the crucial role that code editing plays in software development.

In recent research, a team of researchers from the Multimodal Art Projection Research Community, University of Waterloo, HKUST, University of Manchester, Tongji University, and Vector Institute has introduced CodeEditorBench, an assessment system that has been designed to evaluate LLMs’ effectiveness in a range of code editing activities, such as requirement switching, debugging, translating, and polishing. 

In contrast to other benchmarks that primarily concentrate on code creation, CodeEditorBench emphasises real-world applications and pragmatic elements of software development. The team has selected a variety of coding scenarios and challenges from five distinct sources, covering a broad spectrum of programming languages, degrees of difficulty, and editing assignments. By doing this, they have made sure that the evaluation takes into account the variety and complexity of difficulties found in actual coding environments.

The team has found some intriguing trends in their review, which included 19 distinct LLMs. In the CodeEditorBench framework, closed-source models, specifically, Gemini-Ultra and GPT-4 have demonstrated better performance than open-source models. This emphasises how important model architecture and training data are to deciding performance, particularly when varying prompt sensitivity and problem categories. 

The team has summarized their primary contributions as follows.

1. The goal of CodeEditorBench is to offer a uniform approach for evaluating LLMs. Tools for additional analyses, training, and visualisation have been included in this framework. To promote more research into LLM features, the team has shared that all evaluation-related data will be openly accessible. To improve the assessment’s comprehensiveness, more evaluation measures will be added in the future. 

2. The main aim is to map the current state of LLMs. OpenCIDS-33B is the most effective base model available to the public, followed by OpenCI-DS-6.7B and DS-33B-INST. Models like Gemini, GPT, and GLM that are not publicly accessible usually perform better than those that are. OpenCIDS-33B and DS-33B-INST, two instruction-tuned models with over 30 billion parameters, close this performance difference. 

3. The goal of CodeEditorBench is to draw attention to the shortcomings of LLMs, especially when it comes to rewriting and revising code. Though it performs admirably in three of the four categories, GPT4’s code-polishing abilities are noticeably lacking. In a similar vein, Gemini Ultra is not up to the challenge of changing code requirements. The team has recognized these constraints to tackle these particular issues in LLM training and development.

In conclusion, CodeEditorBench’s main objective is to spur advances in LLMs by providing a strong platform for thoroughly assessing code editing capabilities.

Check out the Paper, Project, and Github. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter. Join our Telegram Channel, Discord Channel, and LinkedIn Group.

If you like our work, you will love our newsletter..

Don’t Forget to join our 40k+ ML SubReddit