English tokenizers split on whitespace, so 'document management system' becomes three searchable tokens for free. Thai writes ระบบจัดการเอกสาร with no separators, so a naive tokenizer stores the entire run as a single token. A user searching for เอกสาร (document) never matches, because that substring was never indexed as a word. The result looks like a broken search box even though every record is present.
We segment with a hybrid approach: a maximal-matching pass against a Thai dictionary handles known words, and a probabilistic model resolves ambiguous boundaries where multiple segmentations are valid. Domain dictionaries matter — government and legal corpora contain terminology that general dictionaries miss, so we extend the lexicon per deployment. Each resolved token is then indexed individually, restoring substring search.
Segmentation fixes recall, but ranking needs tuning too. We weight exact word matches above partial ones and fold in field boosts so a hit in a title outranks a hit deep in a scanned attachment. Combined with OCR text extraction, this turns a pile of Thai PDFs into a corpus that answers queries in milliseconds rather than returning empty results.