Researchers at Bern University of Applied Sciences have demonstrated that while large language models (LLMs) can struggle with visual CAPTCHA challenges, their performance can be significantly improved when paired with specialized tools, though some models exhibit an unexpected tendency to override correct answers. The study focused on three types of CAPTCHAs: rotation puzzles, open-circle identification, and insect selection from a grid.
For the rotation CAPTCHAs, which involve aligning a spun image segment, the researchers developed a script utilizing 1970s circle-detection mathematics and signal-matching techniques. This script consistently solved all ten test puzzles in 0.006 seconds, outperforming several prominent LLMs. Gemini 3.1 Pro achieved a 70% success rate, taking 67 seconds, while GPT-4o and Grok each solved only one puzzle.
The researchers then integrated their fast script as a helper tool for the LLMs. GPT-4o and Grok, when provided with the correct answer by the script, achieved perfect scores. However, Gemini 3.1 Pro, despite receiving the correct answer, chose to overrule the script in some instances. For example, one puzzle was best aligned at 129 degrees, but the CAPTCHA system only accepted answers in 45-degree increments, making 135 degrees the correct submission. Gemini, perceiving the 135-degree alignment as slightly imperfect, rejected the script's correct output, leading to a 20% reduction in its score. This behavior suggests a potential issue where LLMs supervising tools might introduce errors by second-guessing accurate machine outputs.
Similar challenges were observed with other CAPTCHA types. In the open-circle puzzle, where users must click a ring with a gap, LLMs generally failed to provide accurate coordinates for clicking. However, when prompted to explain their reasoning, the models often correctly described the target, such as identifying a "cyan ring in the lower central area." This indicates that the models could visually discern the target but struggled with the precise action of clicking it.
For the insect selection grid, Grok initially made errors, mislabeling squares (e.g., calling square 4 "number 5") and attempting to select a non-existent square 10 in a nine-square grid. Activating its reasoning mode, without altering the visual input, significantly improved its performance to near-perfect.
The researchers also noted an underlying aspect of these specific CAPTCHA types. They are often employed by websites that cannot rely on JavaScript, typically because visitors disable it to avoid web tracking and fingerprinting. Without JavaScript, sites cannot use behavioral signals like mouse movements or click timing to distinguish humans from bots. This leaves a purely visual shape-matching problem, which, as demonstrated by the custom script, can be solved efficiently by relatively simple computer vision techniques on older hardware. The study highlights that the security of these CAPTCHAs largely depends on a geometry problem solvable at high speeds by modern computing devices.
