Redefining Art Understanding for Generative AI: Unified Standards for Evaluation

Event Description



Abstract: For art historians, art understanding is an integrative analysis that requires evidence-grounded inquiry into form, context, and meaning. Advances in large vision-language models (LVLMs) have expanded computational art analysis toward open-ended explanation and interpretation. However, existing benchmarks often assess isolated art-related capabilities or use artworks to test generic recognition and reasoning, leaving a gap between model evaluation and art-historical practice. This report reviews vision-language modeling and AI's engagement with visual art to situate ArtNoesis, our central case study in defining meaningful tasks and evaluation standards through domain expertise.
We introduce a theoretical framework that formalizes art understanding across four hierarchical layers: Formal Analysis, Contextual Analysis, Interpretation, and Critique, mapping them to four integrated layer-level tasks and 20 constituent element-level tasks. We operationalize the first three layers as ArtNoesis, a benchmark co-designed with art historians. Drawing on art-historical scholarship, ArtNoesis includes long-form visual question answering and open-ended essays requiring sustained, multi-paragraph reasoning. Expert-designed, fine-grained rubrics support LLM-as-a-judge evaluation, while paired vision-blind and vision-enabled assessments probe the contribution of visual evidence.
Our evaluations show that leading LVLMs remain below expert-defined standards of art understanding, with persistent difficulties in evidence grounding and producing well-supported novel interpretations. Visual input provides limited and inconsistent gains, although harder questions benefit more from vision. Rubric-based judging shows strong agreement with expert-panel ratings while retaining task-dependent bias. These findings motivate theory-grounded, expert-informed evaluation that connects technological capabilities to art-historical practice and provides diagnostics for improving visual and linguistic evidence integration.

Speaker: Mahan Agha Zahedi

Location: New Computer Science Room 220

Date Start

Date End