simple-evals-mm
A reproducible, reliable evaluation framework for LLMs and VLMs on English and Japanese tasks.
Leaderboard
About simple-evals-mm
simple-evals-mm is the evaluation framework for LLMs and VLMs developed and used by the LLM-jp Multimodal WG, and this leaderboard is built with it. Beyond computing scores, it keeps evaluations reproducible and reliable, and helps you analyze and share the results.
- Any model, API or local. API models through OpenRouter, OpenAI, Gemini or Azure, and local models served with vLLM, SGLang or Hugging Face transformers. In-house models and inference backends plug in through presets, model args and serving patches.
- Text and vision, English and Japanese. 50+ benchmarks on one harness: hard English benchmarks such as MMMU-Pro, GPQA Diamond, MMLU-Redux, AIME 2026 and LiveCodeBench v6, and Japanese benchmarks such as MECHA-ja, JaWildText, HakushoBench, JamC-QA and JFinBench.
- Reproducible. A job pins the code, the data, the model files, the settings and the exact endpoint by hash, and keeps every input and output. It resumes only if none of them changed; otherwise a new job is started.
sem jobs comparechecks whether results from different runs, versions or people are comparable. Each row lists the jobs and the commands behind it. - Checked against model cards. Each model runs at its card's recommended settings, and
sem job paritycompares our scores with the reported ones, so an underestimated baseline is caught and the gap to a model in development is measured correctly. A gap is looked into: for Gemma 4 it was an SGLang kernel bug (CountBenchQA 70 → 97 with our fix). - Results you can see and share. The leaderboard covers frontier models such as GPT-6 Astra down to the main 30B and 9B open models. The Explore tab (a model output viewer) shows each question and model output; the same pages build from your own jobs for local error analysis, and jobs can be published for other researchers.
- Built for reasoning models. Every trial records its output tokens and whether it hit the output limit (the Cut off column), so the truncation ratio and errors such as overthinking, which matter when developing reasoning models, are easy to analyze.
- Harbor-compatible. Terms follow Harbor, and trajectories are stored in its ATIF format, so ATIF tools such as Harbor's viewer can read them.
- Documented. From installation to running, settings and analysis: Docs. Feedback and contributions are welcome.
Parity with model cards
Citation
If you use simple-evals-mm in your work, please cite it as:
BibTeX
@software{simple_evals_mm,
author = {{simple-evals-mm contributors}},
title = {{simple-evals-mm: A reproducible, reliable evaluation framework for English and Japanese multimodal tasks}},
year = {2026},
url = {https://github.com/llm-jp/simple-evals-mm}
}
