This post is aimed at developers and people new to localisation who might be tempted to think an LLM’s output is perfect. At LanguageOps we have automated processes that check for:
- Variables,
$0,{name}are correctly formatted. - Product names and terms use the approved spelling.
- Number and date formats correct for target and match source.
- Links or markup tags still operational and correct.
- Forbidden terms nowhere in the text.
- Source capitalisation matches target.
- …And dozens of other checks we run (nearly 40 in total).
These are usually mechanical checks which require no LLM use and can be run quickly with a fixed set of rules, to save your reviewers falling into these repeated traps across languages and projects. Whole QA software products exist to prevent these leaking into translations - we’ve built them in to the editor for linguists or PMs to quickly run, filter and resolve.
Think of it as unit-testing your text.
What does it mean to unit-test a translation?
Where in software we’ll make several test cases to check against, we can do the same for prose or translations, when comparing to a source text. Do they match and have the minor details been adapted correctly?
Of course language output is not the same as code, but it is similarly constrained - code has to work correctly to pass the tests; language has to match the source AND be correctly amended to the target language.
Automated testing in this respect can stop spending as much human time on the same issues that repeat across all language work. This is a solved problem, and your language work should always include it as a baseline. But there is more…
Human judgement
These questions are harder to turn into a simple pass or fail:
- Does the text mean what the author intended?
- Is the tone right for this audience?
- Does the call to action sound natural?
- Is the wording appropriate in this cultural context?
- Is the message clear when read alongside the interface or product?
For this there is another known solution: LQA, or linguistic quality assurance. Typically we run these checks through an “MQM matrix”, which is a fancy jargon way of saying we test for accuracy, fluency, terminology and style, among others. At Language Ops we use a custom metrics set based on MQM, designed to also flag LLMisms (em-dashes, typical sentence structure such as “and honestly”, “and here’s the quiet part” etc.).
Our harness for the translation side should weed these out before this stage, but you can bring any bilingual or monolingual text into Language Ops to edit and improve, and build your brand style and terms.
With this LQA check, which is LLM driven, we can home-in on the segments in the text with outstanding linguistic issues and have either a proofreading/review stage or a final pass of the translation before going for further in-country review. With Language Ops you can create as many custom workflow stages as you’d like.
Keep the checks for the next job
The results of both of these checks - QA and LQA - are saved to translation memory. Key terms are saved to the termbase/glossary, and next time you run a similar or related translation, fewer errors will be flagged as the pre-translation stage will have included the fixes. This way your texts remain consistent and become easier to review each time.
You are free to export your QA and LQA results for linguists to review and take into account for future work, or rely on the translation memory and termbase features to automate the specific fixes and wording used for a team to share across languages and projects.
Unit-testing is crucial in code; linguists treat translations the same way
In both cases we want to provably ensure the small repeatable bugs haven’t slipped through, and that all meanings and semantic issues have been retained and adapted correctly for target audiences.
We have built this into Language Ops, which is a complete set of professional processes, resulting from years and 100s of millions of words of translation for global household names and all levels of quality request over decades. You can rely on a process like this much more than gathering and rotating through freelance linguists who may all have different ideas and priorities. With this system, everyone works with the same processes and consistency and quality is ensured at each turn.