MICHELLE HAMPSON 06 JUL 2024
Collected at: https://spectrum.ieee.org/chatgpt-for-coding
This article is part of our exclusive IEEE Journal Watch series in partnership with IEEE Xplore.
Programmers have spent decades writing code for AI models, and now, in a full circle moment, AI is being used to write code. But how does an AI code generator compare to a human programmer?
A study published in the June issue of IEEE Transactions on Software Engineering evaluated the code produced by OpenAI’s ChatGPT in terms of functionality, complexity and security. The results show that ChatGPT has an extremely broad range of success when it comes to producing functional code—with a success rate ranging from anywhere as poor as 0.66 percent and as good as 89 percent—depending on the difficulty of the task, the programming language, and a number of other factors.
While in some cases the AI generator could produce better code than humans, the analysis also reveals some security concerns with AI-generated code.
Yutian Tang is a lecturer at the University of Glasgow who was involved in the study. He notes that AI-based code generation could provide some advantages in terms of enhancing productivity and automating software development tasks—but it’s important to understand the strengths and limitations of these models.
“By conducting a comprehensive analysis, we can uncover potential issues and limitations that arise in the ChatGPT-based code generation… [and] improve generation techniques,” Tang explains.
To explore these limitations in more detail, his team sought to test GPT-3.5’s ability to address 728 coding problems from the LeetCode testing platform in five programming languages: C, C++, Java, JavaScript, and Python.
Leave a Reply