Google and Artificial Analysis launch multimodal agent benchmark with 200 tasks

Google announced the launch of the Multimodal Knowledge Agent benchmark on its Gemini blog, developed in partnership with Artificial Analysis.confirmed This new evaluation framework is designed to measure how artificial intelligence agents reason through visually complex files to solve real-world knowledge work tasks.This new test checks if AI can handle visual tasks like reading blueprints and spreadsheets, not just text.The MKA benchmark introduces a 200-task evaluation set with multimodal inputs and expert rubrics to assess agentic reasoning in visual domains.

The evidence

The benchmark addresses a gap where most existing tests can be solved using text alone, ignoring the visual nature of professional work. Google states that day-to-day tasks in many industries involve cross-referencing architectural blueprints, tracing engineering schematics and synthesizing findings into reports.reported To reflect this, specialists from 10 verticals and 83 occupations created a 200-task evaluation set. Each task mirrors a multi-step project that typically takes a seasoned professional three to eight hours to complete.

The evaluation pairs realistic prompts with bundles of multimodal input files. An expert-written rubric grades the output, consisting of roughly 70 yes/no checks. These checks are weighted 75 per cent on factual accuracy and 25 per cent on visual design. According to the blog post, an agentic autorater built on a non-Google model grades each deliverable.reported This automated system opens files, renders pages and inspects spreadsheet formulas against the rubric.

Human domain experts agreed with the autorater’s grades 96 per cent of the time. To prevent data leakage, Artificial Analysis runs the full 200-task set privately. However, a separate set of 10 public tasks is available on Hugging Face for inspection.reported All models in the evaluation run in the same open-source agent harness called Stirrup, using identical tools, sandboxes and turn budgets.

The decision

The technical report accompanying the benchmark highlights extreme differences in model efficiency. On the same tasks, the models tested used between 30 and 130 turns on average to complete each task.reported Google reports that strong perception abilities are necessary to solve these tasks, noting how models attempt to compensate when vision is removed.reported The results indicate that frontier models operate in very different ways regarding resource consumption and task completion strategies.

What to make of it

Gemini 4 Carbon achieved the second-highest quality score among the models tested in this benchmark.reported However, it completed tasks at an average cost six times lower than the model that scored highest in quality. Google describes Gemini 4 Carbon as being on the pareto frontier of quality and price for the MKA evaluation. This suggests a trade-off where slightly lower peak quality may yield significantly higher economic efficiency for certain workloads.

In practice

Gemini 4 Carbon is available today via the Gemini API in Google AI Studio. Developers can access updated documentation and a developer guide to integrate the model. Users can also explore the benchmark directly on the Artificial Analysis platform to evaluate their own models. The private nature of the main 200-task set means external users cannot run the full evaluation themselves, but the public 10-task set on Hugging Face allows for format inspection and limited testing.