Аpplication of large language models for detecting software code vulnerabilities

Abstract

The article examines the application of large language models (LLMs), particularly GPT-4, for automated detection of vulnerabilities in software code. Modern static code analysis methods have limitations in detecting complex vulnerabilities and contextual security threats, which necessitates the development of new approaches based on artificial intelligence technologies. To assess the ability of large language models (LLMs) to detect vulnerabilities in program code, a series of experiments was conducted using several generations of GPT models, including: Ada (350 million parameters), Curie (6.7 billion), Davinci (175 billion), and the latest GPT-4 model (≈1.7 trillion parameters). Based on a comparative analysis with the traditional static analyzer Snyk, it is demonstrated that GPT-4 is capable of detecting a greater number of vulnerabilities, covering a wider spectrum of threats, as well as proposing relevant fixes for their elimination. The study utilized 64 code fragments in eight programming languages, covering 33 vulnerability categories according to the Common Weakness Enumeration (CWE) classification. The experimental research was conducted by comparing the results of code analysis using GPT-4 and Snyk on identical test data sets. The GPT-4 model demonstrated high efficiency in automatic code refactoring, specifically achieving a 94% reduction in the level of critical vulnerabilities compared to the initial state of the software code. Metrics of threat detection accuracy, the frequency of false positive and false negative results, as well as the model's ability to provide detailed explanations of identified security issues were analyzed separately. A particular advantage of GPT-4 was its ability to work in automatic refactoring mode, which was also built into system context variations. The research results indicate the feasibility of using LLMs as an additional tool for ensuring software code security. Possibilities for integrating large language models into modern continuous integration and deployment (CI/CD) environments are proposed, and prospects for hybrid application together with classical automated software security testing tools are outlined.