Introducing AURORA, the Multilingual AI Leaderboard that measures Frontier AI on non-English agentic tasks

LLM leaderboards show frontier models nearly neck-and-neck, making it look like general reasoning has been solved. But for AI teams shipping agents to users in Tokyo, Berlin, or Riyadh, there's a catch: those leaderboards almost entirely stop at English.

Nearly every published measure of frontier progress relies on English prompts or machine-translated test sets. That was fine when LLMs mostly summarized or translated text. Now agents write code and handle multi-turn customer support, and a failure outside English means a broken build, an abandoned cart, or a wrong operational decision.

The problem: why translated evals fail

Teams assume strong English performance carries over to other languages. Our data shows it doesn't. Machine translation introduces artifacts and strips out cultural context. Aggregate global scores mask a key fact: model rankings flip dramatically depending on the language and task.

Introducing AURORA: the multilingual AI leaderboard

To fix this industry gap, we are launching AURORA: LILT's Multilingual AI Leaderboard. AURORA stands for Agentic Understanding and Reasoning in Original-language, Real-world Assessments.

Rather than relying on translated tests, AURORA evaluates models using LILT's suite of localized, multilingual benchmarks, adapted directly from industry standards, to measure non-English performance on agentic, multimodal, and socio-cultural tasks. What sets these benchmarks apart is their creation: it features tasks that are designed and verified by native-language domain experts. These evaluations are specifically designed to reflect real-world enterprise use cases rooted in authentic language, regional context, and culture.

These LILT benchmarks include:

• Multilingual Terminal-Bench: Evaluates software development and coding capabilities across 10 languages, representing real-world challenges in software written for native-language users. Read the Terminal-Bench-LILT benchmark.
• Multilingual τ³-bench: Tests multi-turn agent interaction across various sectors, including banking, retail, telecom, and airline customer support.
• Multilingual MultiChallenge: Assesses contextual memory, self-coherence, and instruction-following within extended contexts.
• Multilingual GAIA-v2-LILT: Measures complex tool utilization and agentic reasoning capabilities. Read the GAIA-v2-LILT benchmark.

We recently completed evaluations against these benchmarks across both proprietary and open-weights state-of-the-art models. The sections below provide a comprehensive breakdown of our benchmark results and key findings.

Multilingual Terminal-Bench

Multilingual Terminal-Bench is our multilingual version of Terminal-Bench [1] and is intended to be a predictor of model performance on terminal-based tasks in 10 languages. The benchmark consists of 324 tasks, each designed in collaboration with a native speaker and contains a minimum of 30 tasks per language.

Key findings:

• Claude Opus 5.5 pulls out a big lead (~3 pp) over its already capable predecessor, Opus 5, at less than 60% of the price
• GPT-6 Sol edges ahead of GPT-5.6 Sol at less than half the cost
• Meta's latest model has comparatively low per-token price, but uses far more tokens and thus costs more per-task than more "expensive" models
• GLM 5.3 Flash (fp8) significantly outperforms GPT-5.4 at 8% of the price of the OpenAI model and is catching up to Opus 4.7

Opus 5.5 is the new leaderboard winner, showing consistent improvements over Opus 5 in task resolution for almost every language, with the biggest gains coming in Chinese and Korean, with a slight regression in German. While this model release brings a 20% reduction in token costs, usage analysis shows that this model also uses significantly fewer tokens (about 30% on average) to complete tasks resulting in an overall reduction of 45% from Opus 5.

GPT-6 Sol is the highest performing model from OpenAI, but only just. The last three models from OpenAI all score within 2 pp of each other, indicating a plateauing of multilingual coding capabilities. However, the pricing of this model (-50%) means it costs significantly less on coding tasks than before.

Google's Gemini 3.8 Flash ranks above GPT-6 Sol, by 2.5 pp. However, it requires more turns to get to the solution and ends up costing 7x that of GPT-6 Sol.

Meta's Muse Spark 1.3 trails slightly behind its predecessor on pass rate as it uses significantly more tokens and turns per-task and runs into time limits. Open-weights models are also catching up to the best proprietary models from ~6 months ago, with GLM 5.3 Flash significantly outperforming GPT-5.4 (+19.3 pp) at 8% of the cost of GPT-5.4, on average. Cohere's Command A+ scores 4.6%, with trajectory analysis showing that the model struggles with tool calls, input file resolution and script recognition.

Multilingual τ³-bench

Multilingual τ³-bench is our adaptation of τ²-Bench [2] into German, Korean, Turkish and Farsi.

Key findings:

• Opus 5.5 and GPT-6 Sol both bring performance improvements and a reduction in token price over their predecessors, posting strong results on this benchmark while reducing per-task cost.
• Opus 5.5 takes top spot, with a ~9 pp gap to GPT-6
• Gemini 3.8 Flash brings Opus 5 performance at 1/3rd the cost per-task

Opus 5.5 wins comprehensively across all languages, showing gains in every non-English language in almost every domain over its predecessor. It also costs less than half of Opus 5 to solve tasks on average; $0.28 for Opus 5.5 vs $0.59 for Opus 5. Both Opus models show a tendency to switch languages during the conversation, but this is reduced by 80% for Opus 5.5.

It's a similar story with GPT-6 Sol, which improves over its predecessor across the board with very small regressions. The improvements are enough to get it to second place for the leaderboard, but it still trails Opus 5.5 by a significant amount. GPT-6 Sol also reduces price per-task compared to its predecessor; $0.16 vs $0.26 for GPT-5.6 Sol.

⁺ Banking domain is evaluated on a 20-task subset.

Google's Gemini 3.8 Flash performs comparably to Opus 5, but costs less than a third of Anthropic's model.

Cohere has a much stronger showing for this benchmark with Command A+, though it scores below other models tested. The largest drops in performance, compared to English, are in Korean and Persian, at 18.2 and 19.2 pp respectively.

These results show that for long conversations in customer support scenarios, models still have some way to go to be equally helpful in all languages.

Multilingual MultiChallenge

Multilingual MultiChallenge is our translated and human-verified version of the MultiChallenge [3] benchmark, which measures the performance of LLMs on multi-turn natural conversations.

Key findings:

• Gemini takes top spot on every non-English language
• GPT-6 Sol regresses on nearly every language compared to GPT-5.6 Sol

Gemini 3.8 Flash leads this benchmark for all non-English languages, followed by Opus 5.5. This is especially impressive considering that Gemini costs less than 1/6th per-task. OpenAI scores comparatively lower in this benchmark, showing a regression on nearly every language, except Arabic (+0.1 pp). The 4.4 pp regression in English stands out, as frontier models continue to ace English benchmarks.

Kimi-K3 and Cohere's Command A+ represent the open-weights models. K3 is a stronger performer in all languages, beating Opus 4.6 in all non-English languages. Command A+ trails other models tested in all languages except Arabic, where it performed better than GPT-5.2.

Explore the full results

No single model leads across every language, task, and cost profile. The right choice depends on where your users are and what your agents need to do. The full AURORA results, including methodology, per-language scores, per-task costs, and domain-level breakdowns for each benchmark, are available at aurora.lilt.com.

We will keep updating the leaderboard as new models are released and new languages are added. Follow us on LinkedIn or X or contact us.

References

[1] Merrill, Mike A. et al. "Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces." ArXiv abs/2601.11868 (2026): n. pag.

[2] Barres, Victor, et al. "τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment." arXiv preprint arXiv:2506.07982 (2025).

[3] Deshpande, Kaustubh, et al. "MultiChallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier LLMs." Findings of the Association for Computational Linguistics: ACL 2025. 2025.

Frequently Asked Questions