MixLID: Token-Level Language Identification in Mixed-Language Text

Abstract

MixLID identifies languages at the token level across 111 language-script combinations. Its training starts with monolingual examples and then incorporates mixed-language examples, including code-switched data and loanword-based augmentation. The model is evaluated on both monolingual language identification and token-level identification in mixed-language text.

BibTeX

@article{2026-mixlid-token-level-language-identification-mixed-language-text,
  title = {MixLID: Token-Level Language Identification in Mixed-Language Text},
  author = {Seongtae Hong and Youngjoon Jang and Seungyoon Lee and Jungseob Lee and Heuiseok Lim},
  journal = {arXiv preprint arXiv:2610.16601},
  year = {2026},
  month = {10},
  abstract = {MixLID identifies languages at the token level across 111 language-script combinations. Its training starts with monolingual examples and then incorporates mixed-language examples, including code-switched data and loanword-based augmentation. The model is evaluated on both monolingual language identification and token-level identification in mixed-language text.},
  url = {https://arxiv.org/abs/2610.16601},
  doi = {10.48550/arXiv.2610.16601}
}