GPT-5 2 Vlms Cultural Anachronism Tab Vlm Benchmark
I gave GPT-5.2 four photos of Indian artefacts and asked it to rank them oldest to newest. It put a well-preserved Bronze Age Indus seal after a badly weathered sculpture from the Classical period — as if surface wear, not style or technique, decided the date. It handed back a clean ranking: no caveat, no uncertainty. That surprised me, because the same model scored 92% when asked which materials would not yet have existed when an artefact was made. So it knows steel came later. Why can't it put an object in its right century? Is that right?
1 Answer
Incorrect — and the failure is specifically temporal. The model is literate about materials and illiterate about sequence, and the two abilities are not interchangeable. The benchmark: 600 questions over 1,600 Indian artefacts spanning eight periods, from prehistory (before 3300 BCE) through the Bronze Age, Iron Age, Classical, Early and Late Medieval, Early Modern and Modern India. Ten vision-language models were tested: GPT-5.2, GPT-5-mini, GPT-4o, GPT-4o-mini, Qwen2-VL 2B and 7B, Qwen2.5-VL 3B and 7B, InternVL3 2B and 8B. Models saw only the image — no catalogue metadata, no captions. Best model, GPT-5.2, overall accuracy 58.7% against a 12.8% random baseline. The category breakdown is where it gets useful, because the scores are not uniformly mediocre — they split cleanly along one axis: • Material Availability — 92.1% (random 6.25%). Asked which listed materials could not have existed when the object was made, it is close to reliable. • Style-Period Attribution — 65.0% (random 25.0%). • Manufacturing Technique — 56.0% (random 6.25%). • Period Grouping — 45.2% (random 10.0%). • Chronological Sequencing — 37.2% (random 4.2%). GPT-4o shows the same split at harsher contrast: 85.3% on material availability, 30.7% on chronological sequencing — a 54.6-point gap inside one model. So the instinct "it recognised the plastic, therefore it understands the period" is exactly the wrong inference. Naming an anachronistic material is a fact lookup; ordering four objects in time is reasoning. On the Indus seal: the paper's error analysis reports that GPT-5.2 placed a well-preserved Bronze Age seal after a weathered Classical-period sculpture — it treated surface condition as an age signal. Models also over-classified ambiguous objects into later periods, which the authors tie to training data skewed toward recent, better-documented material. And 68% of style-period misclassifications fell within one period boundary: Classical versus Early Medieval, not Bronze Age mistaken for Modern. That pattern matters — adjacent-period confusion is a systematic prior, not random ignorance, which is why adding more data of the same kind will not fix it. Why it matters beyond a benchmark score: these models are being wired into museum digitisation, archive cataloguing and classroom material, and the authors note the deficit is worst for non-Western collections that are underrepresented in training data — a mis-dated Indian artefact reinforced by an AI label is a provenance error with a citation attached. One honest limit, from the paper itself: the benchmark is Indian artefacts only, the evaluation is visual-only, and a Western-artefact pilot (84 items, Cleveland Museum of Art) showed GPT-4o did somewhat better there. Adding catalogue metadata barely moved accuracy, which argues the bottleneck is temporal reasoning rather than missing context. How to use this in practice: treat a period attribution from a vision-language model as a candidate, never a fact — especially for undated or transitional objects; demand the reason (style markers, technique, material) rather than the label; and discount surface wear, which this study shows the models read as age. Keep a human cataloguer as the one who signs the record. Source: https://aclanthology.org/2026.findings-acl.1514.pdf
Your answer
Sign in to verify this AI response.
Don't trust us — or the AI. Ask ChatGPT / Ask Claude / Ask Gemini this same question and compare the answers yourself.
More in History
- AI Generated Occupation Images Guernsey Unlabelled Historian — AI image and video generators
- ChatGPT Dall E 3 Neanderthal Depictions 1960s — ChatGPT and DALL-E 3 (OpenAI)
- Fanciulla Di Vagli Five LLMs Invent Tuscan Burial — DeepSeek, ChatGPT, Gemini, Claude, Mistral
- LLMs Rewrite History When Asked Revisionist Prompts — OpenAI/xAI/DeepSeek/Alibaba/Mistral LLMs
- AI Generated 1974 Danforth Good Friday Procession Photo Fake — AI image generators
- ChatGPT Denied Documented Black History Monticello — ChatGPT
- Gemini ChatGPT Fabricated Diarist Battle Saipan — Gemini
- GPT Hallucinated Rabbi Portrait Identification — ChatGPT
- AI Recreated Iwo Jima Video Veo — Veo
- ChatGPT Denies Venezuela Invasion — ChatGPT
More from this topic
Ask an AI image generator what the German Occupation of Guernsey looked like and it will hand you something that reads as a photograph: soldiers in a street, period shopfronts, 1940s grain — produced in seconds, with no caveat and no label. Those recreations are now shared daily on social media, and a University of Cambridge historian who has written on the Occupation says they are a historically inaccurate version of the past. She wants them labelled so they cannot be mistaken for historical evidence. But if a generated image looks sharper than the real 1940s photograph beside it, who is going to notice — and what happens when a generation learns the Occupation from images of events that never happened?
I asked ChatGPT and DALL-E 3 to narrate and picture a day in the life of a Neanderthal, and they produced - with complete confidence - scenes that belong to the middle of the last century. Magnani and Clindaniel generated 100 images and 100 narratives for each of four prompts, then compared them against a corpus of more than 2,000 scholarly abstracts published between 1923 and 2023. ChatGPT's text matched abstracts from the early 1960s; DALL-E 3's images matched the late 1980s and early 1990s, and none of the output fell in the range that would mark it as contemporary. The images showed a primitive species with heavy body hair and stooped bodies, and contained no women or children. The narratives played down the variability and sophistication of Neanderthal culture: about half the text did not align with scholarly knowledge, rising above 80 percent for one prompt. Images and text both added technology too advanced for the period - basketry, thatched roofs, ladders, glass and metal. Explicitly asking for expert knowledge did not bring the models up to date. Is that right?
I asked all five models the same one-line question: "What is the Fanciulla di Vagli?" Four of them replaced the object with something else and described it with total confidence. Mistral Large called it a Copper Age steatite fertility figurine found in 1957 and held at the Museo Archeologico Nazionale in Florence. Claude Sonnet 4.5 called it the submerged bell tower of Fabbriche di Careggine, adding that this tower is "more commonly" known as the Fanciulla di Vagli. DeepSeek-V3 narrated a local legend about a young woman buried near the cemetery whose ghost rises when the lake drops. ChatGPT-4o described a 2001 white Carrara marble statue by sculptor Giuliano Vangi, installed as a memorial to the "victims" of the village's submersion. None of these objects exists. Asked afterwards to list their sources, the models produced real references, generic source categories, or academic-looking bibliographies - but their self-citations described a cremation burial, not the things they had just described. Only Gemini 3 got the find right. Did it get this wrong?