How Good Is ChatGPT at Coding, Really?

🌈 Abstract

This article discusses the capabilities and limitations of the AI code generator ChatGPT in comparison to human programmers. It presents the findings of a study published in the IEEE Transactions on Software Engineering journal, which evaluated the functionality, complexity, and security of code produced by ChatGPT.

🙋 Q&A

[01] Evaluating ChatGPT's Code Generation Capabilities

1. What were the key findings of the study on ChatGPT's code generation abilities?

• ChatGPT had a wide range of success in producing functional code, with success rates ranging from 0.66% to 89% depending on factors like task difficulty, programming language, and more.
• In some cases, ChatGPT could produce better code than humans, but the analysis also revealed security concerns with AI-generated code.
• ChatGPT performed better on coding problems that existed on LeetCode before 2021, compared to newer problems introduced after 2021. This suggests ChatGPT lacks the critical thinking skills to address novel problems.
• ChatGPT was able to generate code with smaller runtime and memory overheads than at least 50% of human solutions to the same problems.
• While ChatGPT was good at fixing compilation errors, it generally struggled to correct its own mistakes in understanding the meaning of algorithm problems.

2. What are some of the security concerns with AI-generated code identified in the study? The study found that ChatGPT-generated code did have a fair amount of vulnerabilities, such as missing null tests, though many of these were easily fixable.

3. How did the complexity of ChatGPT-generated code compare to human-written code? The results show that generated code in C was the most complex, followed by C++ and Python, which had a similar complexity to human-written code.

[02] Improving AI-Based Code Generation

1. What recommendations did the researchers provide for developers using ChatGPT for code generation? The researchers recommend that developers provide additional information to help ChatGPT better understand problems and avoid vulnerabilities, especially when encountering more complex programming tasks. For example, developers can provide relevant knowledge in the prompt and inform ChatGPT of potential vulnerabilities to be aware of.