Artificial intelligence (AI) has made remarkable strides, generating photorealistic images, crafting novels, and even predicting protein structures. However, new research reveals a surprising deficiency: AI often struggles with the fundamental task of telling time.
A study from the University of Edinburgh, soon to be published and currently available on arXiv, tested seven prominent multimodal large language models (MLLMs). These AI models, capable of interpreting and generating various media, were tasked with answering time-related questions based on images of clocks and calendars. The results highlight a significant gap in AI’s ability to perform this seemingly basic skill.
The research focused on the ability of MLLMs to perform temporal inference, a critical aspect of real-world applications ranging from event scheduling to autonomous systems. Despite advancements in MLLMs, research has primarily concentrated on object detection, image captioning, and scene understanding, leaving temporal inference relatively unexplored.
Testing the Time-Telling Abilities of AI
The study evaluated several leading MLLMs, including OpenAI’s GPT-4o and GPT-o1, Google DeepMind’s Gemini 2.0, Anthropic’s Claude 3.5 Sonnet, Meta’s Llama 3.2-11B-Vision-Instruct, Alibaba’s Qwen2-VL7B-Instruct, and ModelBest’s MiniCPM-V-2.6. These models were presented with images of analog clocks, featuring various designs including Roman numerals, diverse dial colors, and some even missing the seconds hand. Additionally, they were given calendar images spanning ten years.
For the clock images, the LLMs were asked to identify the time displayed. The calendar tasks ranged from simple questions like “What day of the week is New Year’s Day?” to more complex queries such as “What is the 153rd day of the year?”
The Challenges of Analog Clocks and Calendars for AI
Reading analog clocks and comprehending calendars requires complex cognitive processes. These tasks demand precise visual recognition of elements like clock hand positions and calendar layouts, combined with non-trivial numerical reasoning, such as calculating day offsets.
AI’s Performance: A Significant Gap Remains
The results revealed that AI systems struggled with these tasks. Their accuracy in reading analog clocks was below 25%. The AI models encountered difficulties with Roman numerals, stylized hands, and even clocks missing the seconds hand, suggesting challenges in detecting hands and interpreting angles on the clock face.
While Google’s Gemini-2.0 performed best on the clock task, GPT-o1 achieved 80% accuracy on the calendar task, significantly outperforming its competitors. However, even the most successful MLLM still made errors approximately 20% of the time.
Implications for Real-World Applications
This research underscores a notable discrepancy between human abilities and current AI capabilities. As Rohit Saxena, a co-author of the study and PhD student at the University of Edinburgh’s School of Informatics, noted, these basic skills, mastered by most people at a young age, pose a significant challenge for AI. Addressing these shortcomings is crucial for successfully integrating AI systems into time-sensitive real-world applications, including scheduling, automation, and assistive technologies.
In conclusion, while AI can perform complex tasks, its struggle with basic time-telling reveals a critical area for future development. As AI continues to evolve, overcoming this limitation will be essential for its seamless integration into our daily lives. For now, while AI might help with your homework, don’t rely on it to meet your deadlines.
