What we learned cleaning a Sindhi text corpus before training a tokenizer

More than 34 million people in Pakistan reported Sindhi as their first language in the 2023 census, and millions more speak it in India and abroad.^[1] Yet compared with English or Urdu, very little clean, openly available Sindhi text exists for training language models. When data is scarce, every document matters, and so does every document that should not be there.

This article shares what we learned while auditing a large Sindhi text corpus, cleaning it and training a tokenizer on the result. The figures below come from our own datasets, which are public on Hugging Face so anyone can check them.

The starting point

Our starting dataset was the Sindhi Corpus 505M, a collection of 742,379 documents gathered from the web.^[8] On paper it looked like a strong foundation for Sindhi language models. Before training anything on it, we audited it.

What the audit found

The audit found a large block of documents that were not Sindhi at all. We were not surprised. When Kreutzer and colleagues manually audited 205 language-specific corpora from five major public web datasets, they found that at least 15 contained no usable text, that many were mislabeled, and that a significant share of the lower-resource corpora had less than half of their sentences at acceptable quality.^[2] Web crawls collect pages by domain or keyword, and text in neighbouring languages, boilerplate and mixed-script content comes along with them.

The lesson is simple and easy to skip: never trust a corpus label without sampling it yourself. Read random documents, check script statistics and look at the most frequent tokens. Contamination that is obvious to a human reader can stay invisible in a dataset card for a long time.

How we cleaned it

We kept the cleaning pipeline deliberately simple and transparent:

1. Unicode normalization (NFKC). Unicode allows the same visible character to be stored in more than one way, and Arabic-script text is full of such cases. The Unicode Standard defines normalization forms that map these variants to one consistent representation.^[4] Normalizing first means every later step, including the tokenizer, sees the same form.
2. An Arabic-script ratio filter. For each document we measured the share of characters in Arabic script and kept documents at or above a 0.80 threshold. This removes Latin-script pages, code and heavily mixed content.

The cleaned corpus, sindhi-corpus-clean, contains 528,723 documents.^[9]

A script filter has an obvious limit worth stating honestly: Urdu is also written in Arabic script, so a script ratio alone cannot separate Sindhi from Urdu. Signals such as letters that exist in the Sindhi alphabet but not in Urdu are a natural next step for finer filtering.

A negative result worth sharing

We also built a version of the corpus filtered with off-the-shelf language identification. In our experiments, tokenizers trained on the language-ID-filtered data performed worse than the one trained on the script-filtered corpus, and we documented this as a negative finding.

This matches published evidence. Caswell and colleagues at Google found that language identification models which looked accurate on clean test sets performed very poorly on real web text: for many lower-resource languages, human-judged accuracy of the resulting web corpora was only around 5%.^[3] For a language like Sindhi, a simple, well-understood filter can be the better choice.

Training the tokenizer

On the cleaned data we trained a 32,000-token tokenizer with the Unigram language model method introduced by Kudo, which learns a vocabulary of subword pieces and their probabilities from the training text.^[5] We used 200,000 documents from the cleaned corpus for training.

We measured the result with fertility, which Rust and colleagues define as the average number of subwords a tokenizer produces per word.^[6] Lower fertility means words are split into fewer pieces. Their study found that tokenizers which split a language’s words into many pieces tend to hurt model performance in that language, which is why fertility is a useful first check. Our production tokenizer reached a fertility of 1.3387: on average, each Sindhi word becomes about 1.34 tokens.

Always test the round trip

One more lesson came from testing our earlier tokenizer releases with decode tests. In those checks, text that was encoded and then decoded did not always come back intact. A tokenizer can look fine in a demo and still corrupt text in edge cases, especially with combining marks and less common letters.

Before publishing a tokenizer, encode and decode a large, varied sample of real text and confirm that every document comes back exactly as it went in.

Takeaways for low-resource language work

• Audit every corpus by sampling it yourself, whatever its label says.
• Normalize Unicode before anything else.
• Prefer simple, explainable filters, and measure whether fancier ones actually help.
• Publish negative results. They save other researchers weeks of work.
• Test tokenizers with full encode and decode round trips before release.

Related work

This work is part of our wider effort to build open NLP infrastructure for Sindhi and related Indo-Aryan languages, including SindhiNLTK, an open-source Python library for Sindhi text processing.^[7] If you need a language dataset, a data-quality audit or an evaluation of an AI model for Sindhi, Urdu or another South Asian language, see our Language AI work or get in touch.

References

1. Sindhi language: speakers by country, citing the 2023 Census of Pakistan. Wikipedia. en.wikipedia.org/wiki/Sindhi_language
2. Kreutzer, J., Caswell, I., Wang, L., et al. (2022). Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets. Transactions of the Association for Computational Linguistics, 10. arXiv:2103.12028
3. Caswell, I., Breiner, T., van Esch, D., and Bapna, A. (2020). Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus. Proceedings of COLING 2020. aclanthology.org/2020.coling-main.579
4. The Unicode Consortium. Unicode Standard Annex #15: Unicode Normalization Forms. unicode.org/reports/tr15
5. Kudo, T. (2018). Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. Proceedings of ACL 2018. aclanthology.org/P18-1007
6. Rust, P., Pfeiffer, J., Vulić, I., Ruder, S., and Gurevych, I. (2021). How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. Proceedings of ACL-IJCNLP 2021. aclanthology.org/2021.acl-long.243
7. Meghwar, A. Sindhi models, datasets and tools. Hugging Face. huggingface.co/aakashMeghwar01
8. Sindhi Corpus 505M (dataset). huggingface.co/datasets/aakashMeghwar01/sindhi-corpus-505m
9. Sindhi Corpus Clean (dataset). huggingface.co/datasets/aakashMeghwar01/sindhi-corpus-clean