The article discusses how a benchmark called VisualWebArena was created by experts at Carnegie Mellon University to evaluate the performance of multimodal web agents on realistic and visually stimulating challenges. The benchmark tests agents’ abilities to read image-text inputs, understand natural language instructions, and perform activities on websites.
According to the research, current text-based autonomous agents aren’t as capable as the powerful Vision-Language Models (VLMs) when it comes to handling VisualWebArena tasks – these agents achieved a mere 16.4% success rate, significantly lower than the 88.7% of human performance.
The team suggests that a new VLM agent using the Set-of-Marks prompting strategy shows better promise, especially on graphically complex web pages. By addressing the shortcomings of the previous text-based agents, this new VLM agent can improve the performance of autonomous agents in visually complex web contexts.
This benchmark is the framework for improving autonomous agents for online tasks. For a more in-depth analysis, you can check the research paper and the code on Github.