Introducing Terminal-Bench-LILT: Multilingual Agentic Benchmark Grounded in Language, Region, and Culture

Frontier coding agents are often evaluated in English, but real software has to work across languages, scripts, locales, and cultural conventions. Terminal-Bench-LILT measures that gap with 300 coding tasks across ten languages, created by native speakers and grounded in real regional and cultural challenges. Even the strongest model, GPT-5.5, resolves only 63.1% of the benchmark, meaning it still fails more than a third of trials, and many tasks go unsolved by every one of the seven frontier models evaluated.

Most frontier coding agent evaluations run entirely in English. But that is not how these systems are used in practice. A developer may prompt a model in English, while the software it produces still has to process Arabic text, sort Japanese records, or handle Korean input correctly. Terminal-Bench-LILT measures whether agents can handle those real multilingual software challenges.

Why translating the prompt isn't enough

The usual approach is to translate an English task set and post-edit the prompts. That can test whether a model understands instructions in another language, but it does not capture the full challenge of building software for multilingual users. Hofman et al. (2025) translated SWE-Bench tasks into eleven languages and found that average non-English performance was only 1.3 percentage points below English. The harder problems often come from the software requirements themselves, such as wrapping Japanese terminal output by display width, applying the Czech alphabet correctly, or handling right-to-left text.

Benchmark Overview

Terminal-Bench-LILT consists of 300 realistic coding tasks across ten languages: Arabic, Czech, German, Spanish, Hindi, Japanese, Korean, Serbian, Turkish, and Chinese. The benchmark includes 30 tasks per language and groups them into three difficulty tiers: easy, medium, and hard.

The tasks can be run under Harbor in an isolated environment with deterministic verification, following the original Terminal-Bench (Merrill et al., 2026) task structure. Every task comes with an instruction in both the native language and English, a Docker environment with any native-language data assets, and a verifier with deterministic tests, plus an oracle solution for reference. Metadata captures the ISO 639-1 language code, challenge category and subcategory, difficulty, and infrastructure configuration, and a README explains the task's motivation, realism, and linguistic and technical challenges.

Examples of multilingual complexity

Two tasks show how multilingual coding challenges can arise from both software behavior and implicit linguistic knowledge.

Japanese terminal width (ja-terminal-width)

The image shows why character count is not the same as display width. A naive wrapper treats each Japanese character as one column, causing the text to overflow a 40-column terminal. A correct solution must count Japanese characters and emoji as two columns, ASCII as one, and also handle halfwidth katakana, variation selectors, and terminal color codes without altering the underlying text.

Czech alphabet frequency (cs-implicit-alphabet-frequency)

The task asks the agent to count each letter in a Czech text without defining the Czech alphabet. A correct solution must know all 42 letters, including accented characters and the ch digraph, which is treated as a single letter.

How the tasks are built

A distributed team of native-speaking experts created the tasks, with progress tracked as pull requests through review. Candidates were screened for computer science or engineering backgrounds and multilingual experience, then completed an LLM-breaking assessment. Fewer than 10% of candidates with matching profiles passed.

Each author received structured onboarding and a one-hour training session covering task architecture, evaluation framework, the benchmark taxonomy, quality standards, and the submission process.

Reviewers gave detailed feedback on each author's first task and remained available for ongoing guidance. Every task was tracked from submission through initial and final human review to merge, with the responsible author or reviewer notified whenever a task changed state or failed an automated check.

The four-layer quality control pipeline

Every task passes through four layers of quality control. Deterministic checks validate the task structure, Docker environment, metadata, and whether the oracle solution passes all tests. LLM judges then check semantic properties such as alignment between the native and English instructions, task categorization, and whether tested behavior can reasonably be inferred from the task materials.

Two rounds of human review follow. Reviewers assess realism, clarity, policy compliance, overlap with existing tasks, and the balance of the final benchmark. Review agents support this process by preparing execution logs and failure summaries so human reviewers can focus on judgment.

Difficulty is calibrated from performance across all seven evaluated models, using 30 trial outcomes per task. Tasks are classified as easy at 66.7% pass rate or above, medium from 33.3% to below 66.7%, and hard below 33.3%. The final set contains 133 easy, 86 medium, and 81 hard tasks.

What the benchmark covers

The tasks fall into five categories:

• Internationalization: 52 tasks covering locale-based templates such as plurals, number and date formats, and Korean counting conventions (like the man-based numbering system), along with right-to-left rendering, Arabic shaping, and script variants such as Simplified versus Traditional Chinese.
• Interaction with the environment: 25 tasks involving file encodings, system configuration, input methods, shell tools, and locale-correct sorting.
• Text conversion: 76 tasks covering normalization, mojibake repair, script conversion, tokenizer training, and other forms of language-specific data processing.
• Text handling: 18 tasks involving document search, stemming, Unicode confusion lists, authentication, and database migration.
• Cultural and Other: 129 tasks covering calendar and date conversion (lunar, Hijri, Umm al-Qura, holiday calculations), social-system rules and regulations, colloquial conventions, and other challenges rooted in a specific language, region, or culture.

What the frontier models achieve

Seven frontier models were evaluated with the fixed Terminus-2 agent: Claude Opus 4.7 and 4.8, GPT-5.4 and 5.5, Muse Spark 1.1 and Gemini 3.1 Pro and 3.5 Flash. Each task-model pair was run 5 times at the provider's default reasoning setting.

Native language vs. English instruction ablation

Changing the prompt to English did not solve the problem. Across German, Hindi, and Chinese, replacing the native-language instruction with an English translation changed pass rates by no more than seven percentage points. The difficult part is not understanding the prompt; it is correctly handling native-language data and locale-specific rules.

Key Takeaway

Strong performance on English coding benchmarks does not guarantee that an agent can handle multilingual software in production. Teams building for multilingual users should evaluate agents with tasks grounded in real language, regional, and cultural requirements, along with verifiers that test whether the software actually works. Multilingual coding is a distinct capability, and Terminal-Bench-LILT shows that even frontier models still have substantial room to improve.

Read the full technical report here.

Sample dataset here.

Interested in the full dataset? Contact us here.

About LILT

LILT is a multilingual applied research lab partnering with researchers to design custom evaluations, benchmarks, and RL environments that measure real model behavior across 200+ languages. LILT works alongside academics, industry leaders, and global nonprofits to improve representation of underrepresented languages in AI systems, building the open datasets, benchmarking standards, and community engagement playbooks that the next generation of multilingual infrastructure demands.