This AI Research Introduces Breakthrough Methods for Tailoring Language Models to Chip Design

ChipNeMo explores the utilisation of LLMs for industrial chip design, employing domain adaptation techniques rather than relying on off-the-shelf LLMs. These techniques involve custom tokenisation, domain-adaptive pretraining, supervised fine-tuning with domain-specific guidance, and domain-adapted retrieval models. The study evaluates these methods through three LLM applications in chip design, resulting in notable performance enhancements compared to general-purpose models. It enables substantial model size reduction with equal or improved performance across various design tasks while highlighting the potential for further refinement in domain-adapted LLM approaches.

The study explores domain-specific applications of LLMs in chip design, emphasising the presence of proprietary data in various domains. It delves into retrieval augmented generation to enhance knowledge-intensive NLP and code generation tasks, incorporating sparse and dense retrieval methods. Prior research in chip design has leveraged fine-tuning open-source LLMs on domain-specific data for improved performance in tasks like Verilog code generation. It also calls for further exploration and enhancement of domain-adapted LLM approaches in chip design.

Electronic Design Automation (EDA) tools have enhanced chip design productivity, yet some time-consuming language-related tasks still need to be completed. LLMs can automate code generation, engineering responses, analysis, and bug triage in chip design. Previous research has explored LLM applications for generating RTL and EDA scripts. Domain-specific LLMs demonstrate superior performance in domain-specific chip design tasks. The aim is to enhance LLM performance while reducing model size. 

The chip design data underwent processing through customised tokenisers, optimising its suitability for analysis. Domain-adaptive continued pretraining procedures were carried out to fine-tune pretrained foundation models, aligning them with the chip design domain. Supervised fine-tuning leveraged domain-specific and general chat instruction datasets to refine model performance. Domain-adapted retrieval models, encompassing both sparse retrieval techniques like TF-IDF and BM25, as well as dense retrieval methods using pretrained models, were harnessed to enhance information retrieval and generation. 

Domain adaptation techniques in ChipNeMo yielded remarkable performance enhancements in LLMs for chip design applications, spanning tasks like engineering chatbots, EDA script generation, and bug analysis. These techniques not only significantly reduced model size but also maintained or improved performance across various design assignments. Domain-adapted retrieval models outshone general-purpose models, showcasing notable improvements—2x better than unsupervised models and a remarkable 30x boost compared to Sentence Transformer models. Rigorous evaluation benchmarks, encompassing multiple-choice queries and code generation assessments, provided quantifiable insights into model accuracy and effectiveness. 

In conclusion, Domain-adapted techniques, such as custom tokenisation, domain-adaptive pretraining, supervised fine-tuning with domain-specific instructions, and domain-adapted retrieval models, marked a substantial enhancement in LLM performance for chip design applications. ChipNeMo models, exemplified by ChipNeMo-13B-Chat, exhibited comparable or superior results to their base models, narrowing the performance gap with more potent LLaMA2 70B models in engineering assistant chatbot, EDA script generation, and bug analysis tasks. 

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to join our 32k+ ML SubReddit, 40k+ Facebook Community, Discord Channel, and Email Newsletter, where we share the latest AI research news, cool AI projects, and more.

If you like our work, you will love our newsletter..

We are also on Telegram and WhatsApp.