Confusion about results on reasoning-image task

[yayafengzi]

The reasoning-image task only have two choices, so "random choose" can get 50 scores. However, performance of the best model "InternVL-Chat-V1-5-RAG" is

Does it means this task is too difficult?

[Weiyun1025]

Thank you for your interest in our benchmark.

The main reason for the underperformance of these models, as compared to random guessing, is their inability to understand images presented as options. For example, when we provide a question in the format <question>\nA.<image>\nB.<image>, the models often do not respond with either A or B, but instead generate a lengthy, irrelevant passage.

We argue that this task is not hard, as humans can easily achieve near-perfect performance. On the contrary, we think this task effectively demonstrates the current limitations of multimodal models in understanding multiple images.