LILTBench Hackathon Is a Wrap: 85 Attempts to Break a Frontier Model Across 31 Languages

Six days, 85 submissions, 31 languages, and one stubborn question: where do frontier models still struggle when the task is written in a language other than English?

That's the question LILTBench was built to answer. Rather than ranking models by broad average performance, LILTBench ranks tasks, specifically those that get harder the moment they leave English behind. Each task pairs a native-language prompt with an English counterpart, runs both against frontier models, and measures the gap between them. The wider the gap, the more clearly the task exposes a multilingual weakness.

We launched LILTBench because most AI benchmarks still under-measure the multilingual reality of software, research, and enterprise work. Users don't just translate English prompts, they work with local scripts, encodings, calendars, sorting rules, numbering systems, and cultural context that English-only testing never touches. Our aim is to give researchers, builders, and model teams a clearer picture of where frontier systems actually fail, so the next generation of multilingual AI can close those gaps.

The hackathon was our first public test of that idea: invite contributors to design coding tasks that could break Claude Opus 4.6 in the languages they know best, then see what kinds of challenges surfaced.

By the numbers: 83 tasks, 31 languages

We received 85 submissions over roughly six days. Two of them didn’t clear CI/CD before the code freeze, which left 83 tasks that made it into the final evaluation. Those tasks spanned 31 languages, which is a lot more linguistic range than we expected to see in under a week.

Chinese led the submission pool, accounting for close to 30 percent of all submissions. Other major submission languages included Japanese, Arabic, German, and Korean, followed by a long tail: 16 languages showed up with exactly one submission each. So a few teams went deep on specific languages, while many others submitted tasks across a broader range using machine translation and general AI.

How LILTBench scoring worked

For scoring, each evaluated task was measured by the pass rate of Claude Opus 4.6 on its native-language instruction. The number of passes put each task into a difficulty bucket, and harder tasks were worth more points:

• Easy, 13 to 15 passes, 1 point
• Medium, 9 to 12 passes, 2 points
• Hard, 4 to 8 passes, 4 points
• Very Hard, 0 to 3 passes, 8 points

A team’s score was the sum of points across all evaluated tasks by that team. Quality counted for more than quantity, although teams benefited from bringing both.

What surprised us: difficulty was bimodal

Difficulty came out bimodal, and pretty dramatically so. Tasks tended to be either very easy for the model or very hard, with relatively little in between. Easy tasks made up 40 percent of the field and Very Hard tasks made up 46 percent. Medium and Hard combined were only about 15 percent.

That split tells you something about how hard it is to design a task that lands in the sweet spot. Aiming for “challenging but solvable” is a lot harder than it sounds. We spot-checked 0-out-of-15 tasks for runtime errors, connection errors, and signs that the model was not actually trying on the task.

Pass rates also varied widely by submission language. Among the higher-volume languages, Chinese tasks had the lowest native pass rate at about 22 percent, while Korean tasks reached roughly 74 percent. Several lower-volume languages, including Hindi, French, and Telugu, recorded 100 percent pass rates, although each was represented by only two or three tasks. These figures reflect the particular tasks submitted in each language, not the difficulty of the languages themselves.

Pass rates by language

How we ran quality control

Every task went through an internal LLM review before it counted. We were looking for cheating patterns, translation mismatches between the native and English markdown files, and verifier tests that a reasonable native speaker or model couldn’t actually infer from the instruction. We also went back and re-inspected everything submitted since the very first entry.

The verdict: nothing rose to the level of disqualification. Plenty of tasks could still be sharpened. Tests, oracles, and instructions all had room for improvement, but the submissions were strong across the board.

Hackathon winners

Thanks

To everyone who submitted, thank you. You gave us a clearer view of the multilingual coding tasks that can still trip up a frontier model. And to everyone who joined in the spirit of the hackathon without chasing the podium, we hope you walked away having learned something new.

About LILT

LILT specializes in identifying the multilingual failure modes that English-first evaluation misses, and building the data needed to fix them. LILT leverages vast domain expert network benchmarking and AI research expertise to help Frontier labs and enterprise AI teams understand where their models fall short for non-English users.

If you're building AI coding tools and want to evaluate or improve multilingual performance, get in touch with our team. We'd love to talk.