Robust multilingual document identification

Abstract

We consider in this paper the problem of detection the language of document when no assumptions are made about a document: it can be of any size and contain zero, one or several languages. Language identification is considered a solved task, but actually, among others shortcomings, does not deal with the case of accurately the presence or absence of several languages in arbitrary documents. In order to tackle these problems, we propose an approach based on word dictionaries using Bayesian statistics and ad-hoc features. We show on two datasets that with sufficient statistics our approach is able to give very satisfying results in dealing with both unsolved tasks: detection of documents with no languages and identification of languages in multilingual documents.