Low-resource Language Identification of English and Kokborok Code-Mixed Sentences

Summary

The objective of the given protocol is to correctly identify word-level languages within the English-Kokborok code-mixed text using an unsupervised model that combines character n-grams and frequency dictionaries.

Abstract

There is a growing need for a model for automatic word-level language detection due to the growing usage of multilingual text. To identify code-switched and code-mixed sentences in the Kokborok language and English at the word level, we propose an unsupervised model. Numerous studies on code-mixed text have been published, including for several Indian languages. However, to the best of our knowledge, our work represents a pioneering effort, being the first to identify languages for low-resource English-Kokborok language pairs. We have employed a method that merges data from a frequency dictionary with data from a character n-gram model. The Viterbi technique combines a character n-gram Markov model with a frequency lexicon to aid in precise language identification at the word level. The proposed method performed well, with a word-level accuracy of 93.15%, which is better than the BiLSTM-CRF model of 88.5% and the rule-based baseline of 85.3%. This demonstrates the effectiveness of combining lexicon and statistical methodology for handling low-resource languages without depending on massive annotated datasets.

Introduction

In this digital age, the rapid growth of code-mixing and code-switching multilingual communication on social media platforms is becoming central to global communication. The need for accurate identification of languages within textual content is paramount. After observation, we found that many Indian languages like Hindi, Bengali, and Telugu have already been explored in language identification, and other Indian low-resource languages such as Manipuri, Bodo, and Assamese have been partially addressed in language identification research. However, there is still a substantial research gap in developing models for Kokborok, a low-resource language that is both linguistically and structurally distinct.

Unlike Manipuri, Assamese, and Bodo low-resource languages, Kokborok has a high level of orthographic variation and is written in Roman or Bengali script, often interchangeably in communication. This creates a lot of ambiguity in the token boundaries, transliteration, and feature extraction at the character level, which presents distinct challenges and has rarely been considered in the literature. Moreover, Kokborok is an agglutinative language, and its morphology includes rich prefixes and suffixes (e.g., -o, -do, -no) and phonemic tones, making it more difficult to apply existing code-mixed language identification models developed with respect to Indo-Aryan languages.

These issues are further enhanced by social media. Kokborok speakers often tend to mix English words into the text, not only for code-mixing and code-switching but also to compensate for missing lexical items in the Kokborok sentence. This leads to non-standard orthographies and context-related use of words, which is a setback to rule-based or purely lexical systems.

In this study, these gaps are addressed by proposing an unsupervised model that combines frequency dictionaries and character n-gram probabilities with a Viterbi decoding framework. The method is particularly focused on the orthographic, morphological, and contextual complexities of the English-Kokborok pair, and thus it is one of the first computational attempts to systematically treat and detect the word-level language barriers in this low-resource language.

BACKGROUND OF KOKBOROK
The words "Kok," which signifies language, and "Borok," which implies human, combine to form the compound word "Kok-borok," which denotes human. The term "Kok-borok" refers to the language spoken by the Borok people of Tripura, as well as those in the adjacent countries of Bangladesh and Myanmar, and the inhabitants of Mizoram, Manipur, and Assam. The Kokborok language, spoken primarily in Tripura, is written in both Roman and Bengali scripts. The Roman script is favored over the Bengali script by the majority of Kokborok speakers. In actuality, the Sino-Tibetan linguistic subgroup known as Tibeto-Burman is where the Kokborok language originated. The Kokborok language and the Bodo language are subtly similar. The 36 varieties of the Kokborok language bear subtle similarities to both the Bodo and Kachari languages.

Koloma was known as the original Kokborok script. The sole state in India, Tripura, was controlled by 184 people before it was united into the Indian Union on October 15, 1949. The Koloma script found in the Rajratnakar book is the one that was used to write down the historical narratives. Later, in the fourteenth century, two Brahmins by the names of Sukreswar and Vaneswar translated the language into Sanskrit and then again into Bengali scripts. There are stem homophony, phonemic tone, verb morphology, word order, and verb derivational suffixes produced from the verbs in the Kokborok language.

As time goes on, Kokborok has encountered languages such as English, Arabic, Bengali, Persian, etc. Different kinds of intricate agglutinative structures can be found in the Kokborok language. People who speak Kokborok as a native language in Tripura are unable to easily access information because the language does not yet have a widely adopted script, but this is being worked on.

There are several dialects of Kokborok spoken in Tripura. In Tripura, various dialects are spoken, including Tripura/Tripuri, Debbarma, Reang, Jamatia, Uchai, Noatia, Lusai, Kukis, Chaimal, Halam, Santal, Lepcha, Chakma, Khasia, Orang, Mog, and Garos. According to the 2011 Census report, there are 1,011,294 Kokborok speakers in India and more than 400,000 in Bangladesh, a neighboring nation.

LITERATURE REVIEW
The authors describe their ongoing research on automatic language identification using a new dataset of Facebook postings with Hindi, Bengali, and English code mixed together. They experiment with various techniques, such as dictionary-based approaches and supervised classification, for language identification at the word level, highlighting the importance of considering contextual clues for accurate language identification in such environments.

The paper presents a dataset of annotated tweets for the purpose of identifying hate speech on social media platforms with a Hindi-English code blend that uses lexicon-based, character-level, and word-level techniques¹. An error-tolerant text categorization method based on N-grams is presented in the paper. First, the system is used to compute profiles using training set data, such as samples of newsgroup material or phrases, that reflect the various categories. After that, the system creates a profile for the specific document that has to be categorized. Lastly, the algorithm calculates the distance measure between each category profile and the profile of the document. The category whose profile is closest to the profile of the document is chosen by the system².

Several word embedding techniques, including Continuous Bag of Words and Skip-Gram models, were compared in order to create feature vectors. Adaboost, Random Forest, K-Nearest Neighbors, Gauss Naive Bayes, Support Vector Machine, and Logistic Regression all produced good cross-validation scores³. A summary of prior approaches is provided in Table 1.

Making use of lexicon-based classifiers, the study outperforms existing classifiers on three language pairings evaluated in the study: Spanish-English, Dutch-English, and German-Turkish with social media datasets⁴. The study's approach shows improved model resilience in terms of convergence and consistency in test performance, outperforming current techniques for sentiment analysis in code-mixed texts by 3.31 percent accuracy⁵.

In their seminal work, Dunning's statistical identification of language was first presented by the Computing Research Lab at New Mexico State University in Technical Report⁶. This paper analyzes different types of supervised learning using the Hidden Markov Model, Random Forest Classifier Method, Conditional Random Field, and Naive Bayes Classifier based on the language pair of English-Telugu code-mixed data⁷. The paper focuses on Indian languages including English, Bodo, and Assamese while discussing a novel approach for recognizing languages in code-mixed social media material. The researchers use Bidirectional Long Short-Term Memory in a deep learning model⁸.

In this paper, they have used a sizable dataset pertaining to the Gujarati language-in which the Gujarati language is mixed with English and Hindi. They employed a variety of machine learning classifiers. Among them, the Support Vector Classifier gave the best accuracy of 92 percent accuracy⁹.

A linguistic framework is used in the separate classification stage to differentiate between code-switching and borrowing of Dutch-English tweets. Using rule-based LID, Support Vector Machines, and Decision Tree Classifiers, they learn that the DTC (micro F1 = .95) and the rule-based LID system (micro F1 = .95) perform best¹⁰.

The performance of their model was evaluated using the code-switched aggressiveness detection TRAC-1 dataset, the Hinglish Offensive Tweet dataset, and the humour-classification dataset¹¹. To improve retention in adult lexical acquisition through reading, a probabilistic approach was proposed as an L2 methodology for producing code-mixed text¹². A dictionary module is incorporated and used to do tests on various supervised classifiers in order to control word-level code-mixing¹³.

They used a variety of metrics to analyze code-switching patterns and discovered that, for both Spanish-English and Hindi-English, neural network models performed worse than the Conditional Random Fields (CRF) model by a margin of 2.5 and 3.5 percentage points, respectively¹⁴.

Using a bilingual lexicon and an English text as input, the method builds a factor graph over lexical mentions to produce a code-switched text that optimizes a given "learnability" parameter. In this paper, they seek to better understand the concepts of the CanVEC toolbox based on the language pairs of Hindi-English code-switched data. They successfully achieved an F1-score of 87.99 percent accuracy^15,16.

The authors first examine code-mixing between Gujarati and English¹⁷, applying a three-stage pipeline to identify the language of each token, normalize the orthography, and transliterate segments back into their native scripts. They then shift focus to a Hindi-English corpus¹⁸, where they introduce novel sentiment-detection algorithms specifically tuned to bilingual sentence structures. To support controlled experimentation, they generate synthetic code-mixed text by training skip-gram models on monolingual data and blending the outputs¹⁹.

Next, a Konkani-English social-media dataset²⁰ is processed via a rule-based, word-level identification method, achieving robust performance without any manually annotated training data²¹. Expanding the scope, one study leverages a large transliterated corpus of English, Hindi, Bengali, and Assamese from platforms like Facebook²² to evaluate a range of word-level tagging techniques. Building on this, another framework²³ dynamically switches between languages to detect embedded or borrowed phrases with high precision. In the agricultural domain, Punjabi-English comments are classified using several models-among them, an n-gram classifier delivers the best accuracy²⁴. For Sinhala-English CMST, ensemble learners (Random Forest, SVM, Naive Bayes, Decision Tree, Logistic Regression) are compared, with Random Forest emerging as the top performer²⁵, while character-level embeddings paired with SVMs achieve strong results on Tamil-English and Malayalam-English pairs²⁶. Finally, the Crúbadán Project²⁷ is presented as a large-scale effort to build under-resourced language corpora by crawling the web, laying the groundwork for future code-mixing research.

Dataset	Key Results	Accuracy/F1-Score/Accuracy/Prcision/recall
Hindi–Bengali–English dataset taken from Facebook posts	Highlighted the Language Identification challenges in word-level  social media Text	89.30%
News articles, documents	Effective performed using N-grams for language identification	0.99%
Indian code-mixed datasets	Showing the good accuracy in word-level Language Identification	Not reported(NR)
Wikipedia-based datasets	Robust token-level code-switch detection	Not reported(NR)
Hindi–English tweets	Performanced Improved in sentiment classification of Hindi and English Code-Mixed	0.78%
Multilingual corpora	Established foundational Language identification framework	78.00%
Code_mixed English–Telugu  data	CRF achieved best performance	89.30%
English–Bodo code-mixed text	Achieved high word-level accuracy	92.00%
Code-Mixed Gujarati–Hindi  scripts	Sentence-level Language identification accuracy is ≈92%	92.00%
Dutch–English tweets	DTC and rule-based models achieved F1 ≈0.95	95.00%
Code-switched datasets	Competitive performance across models	Not reported(NR)
Synthetic Code-Mixed text	Improved lexical learning	Not reported(NR)
English–Kannada Code-Mixed text	Effective automatic LID	Not reported(NR)
TRAC-1, Hinglish datasets	The CRF  outperformed in Neural Network models	91.00%
Multilingual online data	Accurate word-level Lnaguage identification	88.00%
Hindi–English social media	High accuracy in Language identification of Hindi-English	0.93%
English–Gujarati Code -Mixed	Effective bilingual processing
Social media Code-Mixed datasets	Improved sentiment detection	0.90%
Code-Mixed Synthetic  data	Strong embedding representations	Not reported(NR)
Konkani–English Code-Mixed Social media Text	Robust performance without annotation	0.89%
Code-Mixed dataset from Twitter	Improved detection of switches	90.20%
Assamese–Bengali–Hindi–English	Accurate multi-language Language identification	91.00%
Social media Code-Mixed text	F1-score ≈0.95	0.95%
English–Punjabi Code-Mixed data	N-gram model performed best	0.88%
Sinhala–English Code-Mixed data	RF achieved best accuracy	0.92%
Facebook Code-Mixed comments	Improved word-level LID	0.93%
Crúbadán Corpus	Enabled low-resource language research	Not reported(NR)
Code-Mixed datasets	Good performance shown in the word-level Language Identification	0.94%

Table 1: Code-Mixed Language Literature summary.

Protocol

The code-mixed and code-switched text in English and Kokborok was manually collected from local conversations and publicly available posts on social media platforms. No personally identifiable or sensitive information was collected. All procedures followed institutional ethical guidelines.

3. Design and implementation

3.1 Algorithm
3.1.1 For each token in Sentence (S):
a. Using the frequency dictionary, the lexical probability P_lexi is calculated.
b. Based on N-gram model, the character-based probability P_char is calculated
c. weighted interpolation of both:
P_{combined =} λ_lexi P_lexi + _λchar P_char
3.1.2. The probability combination P_combined for each token has been store as emission probability.
3.1.3. Apply the Viterbi Algorithm:
a. Given initial with starting language probabilities.
b. For each token:
i. Calculate the transition P_count transition (same-language continuation).
and P_switch (likelihood of language switching).
ii. Select the language path and maximize the sequence probability.
3.1.4. Output the maximum probability language sequence L with highest overall probability.

3.2 Computational environment
The implementation of all experiments was done using Python 3.10 on the Google Colab platform. The system used a set of basic Python packages like NumPy, Pandas, and Matplotlib to process and visualize data, and scikit-learn (v1.3) to make statistical predictions and analyses. The frequency dictionary integration, character n-gram modeling, and Viterbi decoding were implemented with custom Python scripts. The model was trained and tested on a dataset of 10,000 sentences, with a 70/30 split for training and testing.

3.3 Language Identification
Once a sentence has been tokenized, the language identification module decides whether each token is in Kokborok or English. The full system architecture is illustrated in Figure 1. This language assignment is then used to determine which transcription modules are appropriate. By merging data from a character n-gram model and a frequency dictionary, the language of a token is determined.

Because of the significant orthographical differences between the two languages described above, a classifier that uses only information from the token itself, without any contextual features, already performs well. However, after the initial assignment, another classifier using the Viterbi algorithm is applied. This second classifier reduces misclassification of interlingual homographs and favors word groupings that belong to the same language.

Figure 1: Word-level patterns and penalizing language switches using the Viterbi algorithm. Please click here to view a larger version of this figure.

Figure 2: Method for code-mixed sentences language identification. Please click here to view a larger version of this figure.

We collected the data from the Crúbadán corpus²⁷ for the dictionary and the character model. Numerous languages, including many minority languages such as Kokborok, are present in this corpus.

Since the dataset in the Crúbadán corpus is insufficient for our purpose, we tested a new Kokborok Bible dataset. Compared to other languages, there are comparatively few available sources. Accordingly, the data for this paper's mixed sentences were gathered from Kokborok-speaking WhatsApp groups such as Platinum Jubilee, Population Group, Restoration International Group, Khumulwng, Depachara Baptist Society, United Tiprasa Forum (UTF), Depachara Youth Fellowship, Krishna Nagar (EU), Uchoi Christian Fellowship (UCF), Tipra Kwsrang Bodol, Tribal Engineering Society, National Institute of Technology Students Borok, Information Yarung, and Tripura Uchoi Youth Association (TUYA).

A total of 10,000 code-mixed sentences in Kokborok and English were gathered from these WhatsApp groups between August 2021 and December 2023. The sources and domain distribution of the dataset are listed in Table 2.

Name	Total count words
Kokborok Bible words	718883
English and Kokborok code-mixed sentences	68789

Table 2: Total Kokborok words.

Language	Total Tokens
Kokborok (trp)	711509
English (en)	1767141

Table 3: Character 2-Gram Model.

Sl. No	Total Code-Mixed sentences used
1	10,768

Table 4: Total mixed sentences trained.

The character model and the dictionary were taken from the Crúbadán corpus²⁷. This collection, which is accessible for a wide range of languages, including numerous minority languages, is created automatically by searching the internet for materials in the target language. The Kokborok corpus in particular is rather small because the approach was initially created for Kokborok.

While a Kokborok and English corpus is also available, it was purposefully not used due to a lack of datasets, particularly in terms of code-mixing and code-flipping. There are several English terms in the Kokborok corpus as well; some of these words (including no, do, and o) are more common than in the English corpus.

The dictionary and language model carry out the task of individual word independent labeling. Our task is comparable to that of King et al.²⁸.

The lexical (lex), character (ch), and word label probabilities are utilized to create a linear interpolation between the frequencies of the lexical (ch) and lexical (lex) components. The combined (comb) probability that results from this is shown in equation (1) below; the detailed interpolation parameters are presented in Table 3. The interpolation strategy is depicted in Figure 2.

P_combined = λ_lexi · P_lexi + λ_char · P_char   (1)

Where 0 ≤ λ_lexi, λchar ≤ 1; and λ_lexi + λchar = 1. Words with no lexical basis have an extremely low but nonzero lexical likelihood. Presumably, in the event that a word is absent from both lexicon languages, the character model is used for implementation instead of the lexical probability, even if λ_lexi = 1.

Our method for developing the character model is based on Dunning's character n-gram Markov chain (1994). This is accomplished by counting n-grams at the beginning and end of tokens, respectively, in order to estimate the initial and transition probabilities.

The performance comparison of the trained 2-gram and 4-gram models is shown below.    

   (2)

·

(3)

The initial and transition words of a probability can be multiplied to get the likelihood of a word using the language model (4).

P(〈w1w2w3〉) = P_initial(〈w1w2) · P_transition (c3|〈w1w2) · P_transition(〉|w1w2w3)   (4)

Kokborok language is very rich in prefixes and suffixes and is a morphologically agglutinative language where suffixes such as "o," "do," and "no" are appended to the base words. The 2-gram helps in capturing the local morphological transitions effectively at affix boundaries, while 4-grams can capture those Kokborok words that have longer orthographic dependencies found in root and compound words such as Phailaidido, Thanlainai, and Mwchangkha.

This applies to a large corpus, which increases the likelihood of several data sparsity issues. Additionally, the character combination may not appear in all cases, which might cause the probability to go to zero. Lambda smoothing is used to solve these kinds of issues by giving each n-gram containing unseen characters a small value with a nonzero probability. The value of lambda is set at 0.001.

   (5)

where N = represents the quantity of distinct observations in n-grams.

   (6)

where D= is the number of detected differences in n-grams. It is assumed that the probability for an unobserved n-gram is the same.

   (7)

where C indicates the character size.

The initial values of Pcount, λ_lexi, and λchar were empirically set by frequency analysis of the Kokborok and English corpora. First, we initialized λ_lexi to 0.5 and weighted both lexical and character-based probabilities, thus making both equally contributing in the first run. This level of initialization allowed us to identify the comparative role of the lexicon and the character n-gram model between datasets.

The transition parameter Pcount (continuing in the same language) was set to 0.6, based on the ratio of the monolingual sequences to the switch points in the manually marked dataset. These values provided a good starting point for convergence during tuning. Then, refinement of all parameters was carried out through a series of repeated trials to maximize the Matthews Correlation Coefficient (MCC), which is detailed in the Results section.

Contextual identification
The contextual model uses the probabilities that were already initialized (Pcount and Pswitch) during the independent identification phase and refines them using the Viterbi sequence decoding so that transition inconsistencies are reduced and language switching is detected optimally.

The language model's analytical output is context-dependent and is primarily obtained by combining the current token with the subsequent token and using both the previous and present tokens. Since there is a possibility that two words with the same frequency value will have similar meanings and cause misclassification, the context-dependent approach was used to find the similarity terms that are present in both languages.

Certain words, such as "no," "do," "o," "are," "da (day)," "go," "sa (say)," "rose," "ring," and many more, are similar in both languages. As a result, it can be quite difficult to identify the correct language when this kind of ambiguity arises, and occasionally the languages are incorrectly classified.

To address these language identification problems, we presented a set of language models using a discriminative Hidden Markov Model and supervised machine learning. The transition probability for both languages is assumed to be the same. For the same-language Pcount, the continuing probability is the primary parameter that needs to be declared. Through this, we can calculate the likelihood of switching the language.

P_switch = 1 − P_count   (8)

Contextual classification using the Viterbi path is shown in Figure 3. The Viterbi algorithm's objective is to determine which of the token sequences' languages is the best one according to Pcount and Pswitch.

The Viterbi algorithm is applied to contextual classification. Pcount and Pswitch are used to label edges, and nodes are labeled with the relative likelihood assigned by the first classifier.

Due to the higher likelihood of using the Kokborok language, the dashed path would be chosen if the first classifier were the only one used and no contextual information were available.

The impact of context-based adjustment is visualized in Figure 4. This demonstrates how, in the absence of contextual information, the second, third, and fifth words in the phrase "Next week o phaisidi do" would be incorrectly identified as English (en) (dashed path). Although the probabilities are slightly greater for English, the relative values are close.

Two language shifts will be penalized, and the sequence's probability will be lower than the Viterbi algorithm's optimal path if transition probabilities are also taken into account. Thus, only words that are substantially more likely to originate from the other language-rather than common terms appearing in both-can trigger language changes in brief sequences.

Matthews' Correlation Coefficient was selected as the assessment metric because, in comparison to other metrics such as precision, recall, and accuracy, it is significantly more appropriate for binary classification tasks, as it accounts for true positives and true negatives as well as false positives and false negatives.

   (9)

Results

In Table 4, we have shown the MCC scores and parameter tuning results for Ptrp that calculated the initial bias towards starting sequences with Kokborok words and calculated the Pcount, which is the probability of continuing in the same language.

λ_lex

Pcount = 0.66

Pcount = 0.70

Pcount = 0.74

Pcount = 0.79

Pcount = 0.82

Pcount = 0....

Discussion

While the suggested model demonstrates an accuracy of 93.15% at the word level, thorough analysis of errors reveals several recurring patterns that highlight the challenges of English-Kokborok code-mixed language identification.

Using borrowed words and ambiguous words
A significant portion of errors arises from the use of borrowed English words, which have become common in Kokborok usage. Words like no, do, o, go, r...

Materials

List of materials used in this article

Name	Company	Catalog Number	Comments
Raspberry Pi 4	Raspberry Pi Foundation	RPI4-MODBP-4GB	Used for low-resource language processing
MicroSD Card 64GB	SanDisk	SDSQUAR-064G-GN6MA	For OS and dataset storage
Power Supply 5V/3A	Official Raspberry Pi	SC0218	To power the Pi board
USB Microphone	Boya	BY-M1	For audio input in language data collection
Monitor (HDMI)	LG	22MK600M	Output display during testing
Keyboard & Mouse Combo	Logitech	MK270	Interface control
WiFi Dongle (if Pi 3)	TP-Link	TL-WN725N	Wireless data transfer (if no onboard WiFi)
Python IDE (Thonny)	Open Source	-	IDE for programming
Lexicon Dataset	Custom Built	-	Kokborok-English language pair
N-gram Dataset	Custom Built		English-Kokborok language pair
Code-Mixed and Code-Switched	Custom Built	10,000 sentences	Collected from Social Media text, Whatsapps grouops, NGO, BIBLE, etc