Developers embrace AI tools despite accuracy concerns: survey

New insights from Stack Overflow have revealed that while AI tools have seen significant adoption within IT teams, there remain considerable concerns regarding their productivity and the complexity and accuracy of these tools. Over 1,700 developers were surveyed to gain a deeper understanding of their experiences and opinions on the use of AI-based coding assistants.

According to the survey, an impressive 76% of respondents are currently using or planning to use AI code assistants. Among the roles that predominantly utilise these tools are academic researchers (87%), AI developers (76%), frontend developers (75%), mobile developers (60%), and data scientists (67%). Despite this widespread adoption, there is an evident apprehension about the accuracy of these tools. Approximately 38% of developers reported that AI code assistants frequently provided inaccurate information, with half or more of the outputs being incorrect.

A notable trend emerged from teams with a higher number of AI tool users: a greater degree of complexity and a marked lack of trust in the outputs were cited as significant challenges. This complexity and trust issue might contribute to the ongoing discourse about the reliability and utility of AI in coding environments.

However, this apparent inaccuracy does not seem to significantly deter the satisfaction of users. On the contrary, the data shows a positive correlation between tool use and reported productivity. A remarkable 95% of those who addressed both satisfaction and productivity noted at least a slight increase in productivity due to these tools. This indicates that despite frequent inaccuracies, developers find value and satisfaction in using AI-assisted tools.

The survey also highlighted a long-standing issue in the measurement of productivity. A substantial 74% of respondents working in organisations that utilise these AI tools admitted to being unsure of how their productivity is quantified by their employers. This uncertainty was more pronounced among developers working for smaller organisations, suggesting a possible gap in the clarity of performance metrics in less structured environments.

Interestingly, while AI coding assistants may not always yield accurate results or definitive productivity improvements, they appear to contribute to freeing up developers' time for more creative and personal projects. This inferred benefit points to a holistic enhancement of the developer experience, which could be a significant factor in the overall satisfaction reported by users.

The tools that developers found most satisfying align closely with those perceived as the easiest to use. Codeium (84%), GitHub Copilot (76%), and ChatGPT (61%) enjoyed the highest 'ease of use' ratings, which also corresponded to their satisfaction ratings at 86%, 72%, and 65%, respectively. The major critiques developers had regarding these AI coding assistants involved their struggle with context, complexity, and obscurity in code generation, highlighting areas where further refinement is needed.