A recent report from Artificial Analysis suggests a significant performance drop in OpenAI’s GPT-4o, bringing its capabilities closer to the less powerful GPT-4o-mini. This comes shortly after OpenAI announced an upgrade promising improved creative writing, better file handling, and more insightful responses.
OpenAI’s announcement, shared on X (formerly Twitter), highlighted GPT-4o’s enhanced writing abilities, claiming “more natural, engaging, and tailored writing to improve relevance & readability.” They also touted improvements in handling uploaded files, offering “deeper insights & more thorough responses.” However, Artificial Analysis’s findings cast doubt on these claims.
Artificial Analysis expressed concern over the model’s performance in a post on X, stating, “We have completed running our independent evals on OpenAI’s GPT-4o release yesterday and are consistently measuring materially lower eval scores than the August release of GPT-4o.” Their Quality Index for GPT-4o dropped from 77 to 71, matching the score of GPT-4o-mini.
Further analysis revealed a decline in GPT-4o’s performance on specific benchmarks. The GPQA Diamond benchmark saw a decrease from 51% to 39%, while MATH benchmarks fell from 78% to 69%.
Interestingly, the researchers observed a significant increase in the model’s response speed. Output tokens per second jumped from approximately 80 to 180. They noted, “We have generally observed significantly faster speeds on launch day for OpenAI models (likely due to OpenAI provisioning capacity ahead of adoption), but previously have not seen a 2x speed difference.”
This increased speed, coupled with the performance decline, led Artificial Analysis to speculate, “Based on this data, we conclude that it is likely that OpenAI’s Nov 20th GPT-4o model is a smaller model than the August release.” They advised developers against switching from the August version without thorough testing, especially given that OpenAI hasn’t adjusted pricing.
GPT-4o, initially released in May 2024 as an improvement over GPT-3.5 and GPT-4, boasts state-of-the-art performance in voice, multilingual, and vision tasks, according to OpenAI. These capabilities make it suitable for complex applications such as real-time translation and conversational AI. The recent performance dip raises questions about the direction of OpenAI’s development and the potential impact on these applications.
This performance regression raises questions about the trade-offs between speed and accuracy in large language models. While faster responses are desirable, maintaining performance on key benchmarks is crucial for user satisfaction and the continued development of reliable AI applications. Whether OpenAI will address these concerns and restore GPT-4o’s previous performance remains to be seen.
