Large-scale Optical Character Recognition of Pre-modern Chinese Texts

Abstract

Optical character recognition (OCR) the fully automated transcription of text appearing in a digitized image offers transformative opportunities for the scholarly study of written materials produced prior to the digital age. Digitization, in the sense of photographic reproduction, is now a well-established and efficiently performable mechanical process, and one with significant value in its own right for purposes of preservation as well as access to rare materials. As a result, hundreds of millions of pages of pre-modern Chinese works have already been digitized by libraries and academic institutions around the world a significant portion of this increasingly being made freely available online. To make use of this material eff iciently, transcriptions of the textual content of these images are needed. Given the enormous volume of image data in existence and its ongoing production as digitization continues this task is only feasible if it can be fully automated: performed by software without manual human intervention. Individually, reliable transcriptions produced by OCR offer enormous time savings to researchers, enabling efficient navigation of materials in ways not possible without digital transcription. In aggregate, however, these transcriptions facilitate entirely new ways of exploring historical materials for example, rapidly identifying material that one suspects might exist somewhere, without knowing in advance where that might actually be. It is also a prerequisite also to virtually any type of statistical analysis of these materials the potential utility of which continues to increase as a larger and larger proportion of the extant corpus is transcribed. This paper introduces a procedure for OCR of pre-modern Chinese written materials, both printed and handwritten, describing the complete process from digitized image through to automated transcription and manual correction of remaining errors, with particular attention to issues arising in this domain. The process described has been applied to over 25 million pages of pre-modern Chinese works, and the paper also introduces the Chinese Text Project (https://ctext.org) platform used to both make these results available to scholars as well as provide a distributed, crowdsourced mechanism for facilitating manual corrections at scale as well as further analysis of the materials.