MEGA-Bench Leaderboard
🚀 Introduction
MEGA-Bench is a comprehensive benchmark scaling multimodal evaluation to 500+ real-world tasks!
We aim to provide cost-effective and accurate evaluation for multimodal models, covering a wide range of real-world tasks. You don't have to run models on dozens of benchmarks -- MEGA-Bench delivers a comprehensive performance report in a single benchmark.
🧐 Highlights of MEGA-Bench
- 505 diverse tasks evaluating multimodal models across 8 grand application types, 7 input visual formats, 6 output formats, and 10 general multimodal skills, covering single-image, multi-image, and video tasks
- Moves beyond multiple-choice questions, offering diverse output formats like numbers, code, LATEX, phrases, free-form responses, and more. We developed 45 customized metrics to accurately evaluate these diverse outputs
- Focuses on task diversity rather than repetitive examples, ensuring cost-efficient evaluation
- Provides fine-grained capability reports across application type, input/output formats, and required skills
🔨 Systematic Annotation Process
- Guided by an initial application-driven taxonomy tree
- 16 expert annotators contributing to a 2-round process to develop 505 tasks
- Utilizes advanced tools for task design, review, and quality control
- Ensures high-quality data through continuous refinement and balanced task distribution
📊🔍 Results & Takeaways from Evaluating Top Models
️🔥📝 2025.01
- Gemini 2.0 Experimental (1206) and Gemini 2.0 Flash Experimental outperform GPT-4o and Claude 3.5 Sonnet.
- We add Grok-2-vision-1212 to the single-image leaderboard. The model seems to use a lot of tokens per image, and cannot run many of our multi-image and video tasks.
- We will evaluate o1 series models when there is budget.
📝 2024.11
- GPT-4o (0513) and Claude 3.5 Sonnet (1022) lead the benchmark. Claude 3.5 Sonnet (1022) improves over Claude 3.5 Sonnet (0620) obviously in planning tasks (application dimension) and UI/Infographics inputs (input format dimension).
- Qwen2-VL stands out among open-source models, and its flagship model gets close to some proprietary flagship models
- Chain-of-Thought (CoT) prompting improves proprietary models but has limited impact on open-source models
- Gemini 1.5 Flash performs the best among all the evaluated efficiency models, but struggles with UI and document tasks
- Many open-source models face challenges in adhering to output format instructions
🎯 Interactive Visualization
Visit our project page to explore the interactive task taxonomy and radar maps, offering deep insights into model capabilities across multiple dimensions. Discover a comprehensive breakdown far beyond single-score evaluations.
📚 More Information
- Our evaluation pipeline is available on our GitHub repo.
- Check full details of our paper at https://arxiv.org/abs/2410.10563
- Hugging Face Datasets: https://huggingface.co/datasets/TIGER-Lab/MEGA-Bench
Table 1: MEGA-Bench full results. The number in the parentheses is the number of tasks of each keyword.
The Core set contains $N_{\text{core}} = 440$ tasks evaluated by rule-based metrics, and the Open-ended set contains $N_{\text{open}} = 65$ tasks evaluated by a VLM judge (we use GPT-4o-0806).
Different from the results in our paper, we only use the Core results with CoT prompting here for clarity and compatibility with the released data.
$\text{Overall} \ = \ \frac{\text{Core} \ \cdot \ N_{\text{core}} \ + \ \text{Open-ended} \ \cdot \ N_{\text{open}}}{N_{\text{core}} \ + \ N_{\text{open}}}$
* indicates self-reported results from the model authors.
10 | 58.00* | 57.12 | 64.56 | 58.82 | 64.68 | 40.44 | 66.94 | 53.27 | 57.89 | 58.84 | 64.63 |
Data Sources
The data source of MEGA-Bench tasks have three main types:
- Purely Self-designed: The task is designed entirely by the annotator, and the annotator looks for the image or video resources from the Internet or even using code/simulator.
- Inspired and adapted from existing benchmarks: The task is inspired by existing benchmarks or datasets. The annotator collects the raw image/video data from existing datasets but does not use the original annotation. The annotator redesigns/repurposes the data by writing concrete task descriptions and creating new questions and answers, or using scripts to re-process the data for the designed task.
- Directly converted from existing benchmarks: The task is directly converted from existing benchmarks or datasets. The annotator randomly samples a subset from the existing benchmark, directly using its image/video and the annotation without redesign.
In our annotation process, the first two task types are encouraged. The task reviewers strictly control the number of the third type and reject the task if an annotator submits many tasks of the third type.
Please refer to Table 17 of our paper for the detailed data source of all tasks in MEGA-Bench.
Submit on MEGA-Bench Leaderboard
Our evaluation pipeline is released on our GitHub repository.
The evaluation results processed by the breakdown analysis script are put into this leaderboard.