The development of autonomous agents increasingly relies on Multimodal Language Models (MLMs) to perform tasks described in natural language with GUI environments, such as websites, desktop computers, or mobile phones. Existing benchmarks for MLM agents in interactive environments are limited by their focus on a single environment, lack of detailed and generalized evaluation methods, and thecomplexities of constructing tasks and evaluators. To overcome these limitations, we introduce CRAB, the first cross-environment agent benchmark framework, incorporating a graph-based fine-grained evaluation method and an efficient task generation method. Our framework supports multiple devices and can be easily extended to any environment with a Python interface. Leveraging CRAB, we develope CRAB Benchmark-v0 comprising 120 tasks in computer desktop and mobile phone environments. We evaluated 6 advanced MLMs using different single and multi-agent system configurations on this benchmark. The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%.

“本文聚焦研究较少的古汉语嵌套命名实体识别任务,以《史记》作为原始语料,针对古文意义丰富而导致的实体分类模糊问题,分别构建了基于字词本义和语境义2个标注标准的古汉语嵌套命名实体数据集,探讨了数据集的实体分类原则和标注格式,并用RoBERTa-classical-chinese+GlobalPointer模型进行对比试验,标准一数据集F1值为80.42%,标准二F1值为77.43%,以此确定了数据集的标注标准。之后对比了六种预训练模型配合GlobalPointer在古汉语嵌套命名实体识别任务上的表现。最终试验结果:RoBERTa-classical-chinese模型F1值为84.71%,表现最好。”