JetBrains Researchers Introduce CoqPilot: A Plugin for LLM-Based Generation of Proofs

In recent years, formal software verification has gained prominence, especially in fields where software reliability is critical, such as aerospace engineering, finance, and healthcare. Proof assistants like Coq have been instrumental in ensuring the correctness of software by enabling developers to create mathematical proofs to verify their code. However, writing such formal proofs is a labor-intensive and time-consuming task, requiring considerable expertise. This challenge has led to the need for automated tools that can streamline proof generation, reduce errors, and speed up the process.

JetBrains Researchers have introduced CoqPilot, a VS Code extension that automates the generation of Coq proofs. CoqPilot collects incomplete proof segments, known as proof holes, marked with the admit tactic in Coq files and uses LLMs along with traditional methods to generate possible solutions. It then verifies if the generated proof is correct, automatically replacing the proof hole when successful. The focus of CoqPilot is twofold: to provide a seamless experience for developers working with Coq by integrating multiple generation methods and to create a platform for experimentation with LLM-based Coq proof generation. CoqPilot requires minimal setup, making it accessible for users interested in formal verification without requiring extensive tool configuration.

Technically, CoqPilot’s architecture is modular, designed to accommodate a variety of proof generation methods. It integrates popular LLMs like GPT-4 and GPT-3.5, as well as automation tools such as CoqHammer and Tactician, allowing users to combine multiple approaches. CoqPilot provides services like proof verification and completion using different model parameters, including prompt structure and temperature settings for LLMs. Its modular nature makes it easy to adapt to new models or even different languages beyond Coq. CoqPilot also handles proof generation in a user-friendly manner, allowing proof holes to be solved automatically and, if necessary, utilizing multiple rounds of error handling and retries to improve the generated proof’s correctness.

The importance of CoqPilot lies in its ability to significantly improve the efficiency of proof generation for Coq users. In their evaluation, JetBrains researchers experimented with several LLMs, including GPT-4, GPT-3.5, Anthropic Claude, and LLaMA-2, comparing their performance in generating Coq proofs. The results were promising: GPT-4, combined with CoqPilot, successfully generated 34% of the proofs, while a collective effort using multiple models proved 39% of the theorems in their dataset. Furthermore, CoqPilot’s integration with tools like Tactician and CoqHammer further boosted its performance, with an overall success rate of 51% when all available tools were utilized. These results demonstrate CoqPilot’s potential to streamline the proof-writing process, allowing developers to focus on higher-level concerns while the plugin handles more repetitive tasks.

In conclusion, CoqPilot represents a significant advancement in automating the proof generation process for Coq users. By leveraging LLMs and integrating various proof generation tools, CoqPilot not only reduces the time and effort required for formal verification but also enhances the quality of proofs. Its modular architecture and support for a range of tools make it an excellent choice for developers and researchers looking to automate formal verification processes. With its ability to work seamlessly with various models and tools, CoqPilot provides a robust solution for the challenges associated with generating formal proofs, making it an invaluable tool for those working in software reliability and formal verification domains.

Check out the GitHub Repo and Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and join our Telegram Channel and LinkedIn Group. If you like our work, you will love our newsletter.. Don’t Forget to join our 55k+ ML SubReddit.

[Trending] LLMWare Introduces Model Depot: An Extensive Collection of Small Language Models (SLMs) for Intel PCs