Research
What retrieval gets wrong when your users write in Malay
Every retrieval benchmark you have read is English. Here is what actually breaks when the corpus is code-switched Bahasa Malaysia, and how to measure it before it embarrasses you.
A retrieval system that scores well on an English benchmark can fail badly on a Malaysian support corpus, and it will fail quietly. The answers stay fluent. They are simply drawn from the wrong documents.
The three failure modes we see repeatedly
Code-switching inside a single sentence. Real queries are not in one language. A customer writes half a sentence in Malay, drops in an English product name, and finishes with a colloquial abbreviation. Most embedding models handle each language competently in isolation and lose the thread when they interleave.
Morphology. Malay builds meaning with affixes: a root verb takes prefixes and suffixes that shift it substantially. Tokenisers trained mostly on English split these in ways that scatter what should be a single semantic unit across several tokens, and similarity scores degrade accordingly.
Register mismatch. Your documents are written in formal Malay. Your users write in the register they text in, including spellings that no style guide would endorse. The gap between the two is wider than the equivalent gap in English.
Measuring it before it hurts
The fix is not a clever model choice. It is an eval set that reflects the language your users actually type.
- Pull two hundred real queries from your support logs. Not paraphrases: the actual text, spelling errors included.
- Have a bilingual reviewer label the correct source document for each.
- Score your current pipeline against it. Expect the number to be worse than the vendor benchmark suggested.
- Only then start changing things.
What actually moved the number for us
On a recent engagement, in order of impact:
- Query expansion into both languages before retrieval. Cheap, and it recovered most of the code-switching loss.
- A reranker applied over a wider candidate set. Slower, but it caught the cases where the right document ranked eighth.
- Chunking on document structure rather than fixed token counts, so a policy clause stayed intact instead of being split across two chunks.
Changing the embedding model, which is where everyone starts, came fourth.
The general lesson
The benchmark is not your product. Any time you are working in a language or a register that the published evaluations do not cover, the honest move is to build a small, real, locally-labelled eval set first and treat every published number as a hypothesis.