Can you do better than top-level AI models on these basic vision tests?
What are the eight visual acuity tests created?
The eight visual acuity tests created by the researchers include: 1) identifying how often two colored lines intersect, 2) identifying which letter in a long word has been circled, 3) counting how many nested shapes exist in an image, and other tasks involving basic 2D shapes and patterns. These tests are designed to evaluate the performance of Vision Language Models (VLMs) on simple visual analysis tasks, ensuring minimal reliance on pre-existing images or memorization.
How do these tests minimize VLMs solving by memorization?
These tests minimize VLMs solving by memorization by using custom-generated code to create simple visual acuity tests with objectively correct answers, rather than relying on pre-existing images or tests found on the public internet. This ensures that the VLMs have not been previously exposed to the specific test examples during their training, reducing the chances of solving through memorization.