One of my first jobs before I progressed pursuit of a career in tech was translation, between French, Italian and English. The whole discipline is about what fails to survive the crossing: what a sentence stops meaning once you move it, and what you have to rebuild on the other side for it to mean the same thing again. That turns out to be the central problem in multilingual AI safety evaluation, which I did not expect.
One of the five questions I took into an advising call was about comparative advantage: where does my particular mix of skills do the most good? Fifteen years in technology, agentic systems in production now, and before that translation, search and real-time bidding, growth work across sixty-three countries, social enterprise in Guatemala, and five languages at working level or better. I asked whether that combination pointed anywhere I wasn’t seeing, or whether it was just texture around ordinary technical work.
I think I’ve found at least one answer, and it was sitting in the published work programme of a national safety institute.
What Singapore is working on
The Singapore AI Safety Institute (https://sgaisi.sg/about-us/), a partnership between the Digital Trust Centre and the country’s infocomm regulator, lists among its current work: multilingual AI evaluation methodologies, collaborative testing to improve evaluations across different languages, and AI agent safety testing examining whether agents can complete real-world tasks without leaking sensitive data. It participates in joint testing exercises with other national institutes.
Read that list against my own background and it’s uncomfortably close. Multilingual evaluation, agent safety, and international coordination between institutes, which is a translation problem in the professional as well as the literal sense.
Singapore is a sensible place for this to be happening. It’s small, multilingual by default, diplomatically well-positioned between the US and China, and has an obvious national interest in evaluation standards that work in more than one language.
Why the gap exists
Nearly all safety evaluation is built in English, by English speakers, on English benchmarks.
There are decent reasons, the models are strongest in English, the research community works in English, and the benchmarks that exist are the ones people build on. But what follows from them is not decent at all:
Safety properties don’t transfer across languages reliably. Guardrails tuned on English refusals behave differently in Kaqchikel. This isn’t speculative and it isn’t new. Deng et al. (https://arxiv.org/abs/2310.06474) measured it at ICLR 2024: low-resource languages were around three times more likely to surface harmful content. Combine a multilingual prompt with a malicious instruction and unsafe output ran at 80.92% on ChatGPT, 40.71% on GPT-4. What I’d expect, and what nobody has measured, is that it gets worse for agentic behaviours, where the failure isn’t a refusal but a decision.
Translation isn’t evaluation. Machine-translating an English benchmark gives you an English benchmark with noise, not a test of how the model behaves for someone reasoning natively in another language. The cultural context that makes a prompt harmful or a refusal appropriate doesn’t survive the translation, which is precisely what someone who spent three years as a professional translator would tell you.
That last point is the one where I have something specific to say, and it is why I opened with the translation years. Applying that instinct to evaluation design isn’t a stretch; it’s the same skill pointed at a new artefact.
The version of this I’d actually build
If I were designing a cheap test for whether this lane suits me, it would be roughly:
1. Take an existing agentic safety benchmark, a prompt-injection suite, say, since that’s the area I know best.
2. Produce a properly localised version in two or three languages I speak well, distinguishing carefully between translated and natively-constructed cases.
3. Measure attack success rate by language on a current open model, with the English set as the baseline.
4. Read a random sample of the transcripts by hand in each language, which is the step almost nobody can do at volume, and where a multilingual investigator has an actual, unfakeable advantage.
The hypothesis I’d expect to test: injection defences generalise worse across languages than the underlying capability does, so the safety gap between English and non-English deployment is larger than the capability gap. If true, that matters for every organisation deploying agents in multiple markets (which is most of them) and it’s the kind of finding that’s cheap to produce and hard to argue with.
So I spent the first hour of the test finding out what exists, which is what you do before claiming a gap.
Quite a lot exists. None of it is the experiment above. Deng et al. is single-turn: a harmful query, translated, answered or refused. Singapore’s institute co-leads joint testing across ten languages and five harm categories with the Japanese and Canadian institutes, and there is a Singapore-specific benchmark, RabakBench, plus published red-teaming of the country’s low-resource languages (https://arxiv.org/abs/2509.15260). All of it is text-first. All of it is turn-by-turn.
The unclaimed part is narrower than “nobody has noticed”, and better. Nobody has run this on agents: multi-turn, tool-using, where the injection arrives inside tool output rather than in the user’s message, and where the thing that goes wrong is an action rather than a sentence. That version needs someone who can read the transcripts natively, which is the whole point of the post.
The thing I keep getting wrong
I have consistently listed languages at the bottom of applications, in the section where people put hobbies. Five working languages was a fact about me, not a capability I was offering.
The reframe is that the field has an evaluation coverage problem that more English-speaking researchers cannot fix, and that reading transcripts natively in five languages is a research instrument rather than a biographical detail.
Whether the gap is big enough to matter at scale, I don’t know. It’s certainly big enough to run an experiment in, and running the experiment is how I’d find out.