Tidying up Old Papers: Article separation in Chinese students’ journals
This experiment is part of a larger research project that seeks to re-examine the impact of liumei xuesheng (Chinese students returned from America) on modern China and the building of Sino-American relations during a foundational century (mid 19th-mid 20th centuries), based on a new bunch of digitized sources and new methodologies (e.g., natural language processing, historical network analysis, topic modeling). Crucial to this project is the creation of a textbase of students’ journals running over forty-five years (1906-1949) to enable full-text search and complex queries in vast bilingual (English/Chinese) collections (Table 1). Such a textbase will further allow historians to build question-driven corpora and to conduct quantitative and qualitative text analysis across multiplie articles, issues, and collections of journals.
|Title (short)||Title||Publisher||Language||Period||Periodicity||Total number of pages|
|CSM||Chinese Students’ Monthly||Chinese Students’ Alliance||English||1906–1931||Monthly||19000|
|WCSJ||World Chinese Students’ Journal||World Chinese Students’ Federation||English||1906–1913||Monthly||2000|
|CSCJ||Chinese Students’ Christian Journal||Chinese Students’ Christian Association||English||1910-1921||Monthly||3000|
|LMXSJ||Liumei xuesheng jibao 留美學生季報 (Chinese Students’ Annual/Quarterly)||Chinese Students’ Alliance||Chinese||1914–1928||Quarterly/annual||In progress|
|HMJX||China and America 華美協進 (Huamei jixie)||China Institute of America||English||1948-1949||Semi-monthly||In progress|
Table 1. Major collections of liumei’s journals before 1949.
These journals are rich sources for documenting the liumei‘s discursives practices aimed at legitimizing their position in society, the social clubs, associations and activities they established to organize themselves as a self-conscious group, the local and global contexts in which they evolved in China and the United States, the political and social events that most concerned them, their literary writings and philosophical essays, their daily life (job announcements, classified advertisements), their sources of funding and their networks of patronages (commercial advertisements). Two of these journals have been the object of specific studies (Bieler, 2003; Chen, 2013), but in as much as scholars had to rely on print materials and manual methods of selection, they were not in a position to take fully advantage of the extent and depth of the journals’ contents.
From print to full text
These journals have benefited from the worldwide, massive digitization of historical press materials which began two decades ago with Google/HathiTrust. Digitization opens new opportunities for research but at the same time it raises important challenges. The scanning of journals is usually done at the page level, which raises well-known problems for analyzing documents with complex layout such as journals and newspapers. Their pages often include multiple columns containing both textual and visual contents. In the current state of the art, it is therefore impossible to make refined queries at the level of single articles or to filter (out) specific items such as advertisements, tables, illustrations captions. To enable the identification of unique articles with clearly defined boundaries beneath the page level, and to refine the classification of documents types, it is necessary to manually annotate samples of documents to create a ground truth. This ground truth can then be used to train a model to automatically detect and classify text contents in entire collections of (non-annotated) journals.
In this preliminary experiment, we used the Chinese Students’ Monthly (CSM) as a proof of concept to create our first ground truth. The CSM has been entirely digitized by Brill. Part of the collection is also available for free on Internet Archive. The CSM provides an ideal test case because it presents an intermediate structure between newspapers (which include multiple columns and advertisements) and scientific journals (which include specific features such as tables, figures and footnotes). Furthermore, the layout of the journal evolved significantly during its period of existence, which allows us to cover a wide range of possible layouts. We identified three main periods corresponding to three major types of layout, with an increasing degree of complexity:
- Early years (1906-WWI): single column, mostly text;
- 1920s: single column, textual and visual contents;
- Late years (early 1930s): two columns, text and visual contents.
|Layout characteristics||Sample issue||Number of pages|
|Case 1 (1906)||Single column, mostly text||Vol.2, no.1 (December 1906)||22|
|Case 2 (1919)||Single column, textual and visual||Vol.15, no.1 (November 1919)||104|
|Case 3 (1931)||Two column, textual and visual||Vol.26, no.6 (April 1931)||26|
- Paragraph: the most basic unit of text.
- Article: the most basic semantic unit (from a thematic or authorial point of view). An article can be composed of one or multiple paragraphs. In the latter case, we draw boxes for each paragraph and then we draw a larger overlapping “Article” box to comprehend all the paragraph contain in the article.
- Paragraph bottom: in case when the paragraph is interrupted at the end of the page.
- Article continued: in case when the article runs over two or more pages. In case the article runs over non-continous pages, we also tag “Continued p.” with a specific box.
- Heading : main title (first level) usually title of major section.
- Sub-heading : second level of title (sub-section or title of article)
- Sub-subheading : in case when there are more than two levels of titles.
- Author : named author of an article
- Caption (of illustration)
- Tabular (tables, lists)
- Other : metadata and other contents we choose to discard in substantial text analysis but which we are still interested to retrieve for metadata analysis (e.g., information on editorial board, subscription rates…)
Case 1 (1906): Single column, mostly texts
Case 2 (1919): Single column, textual and visual contents
Case 3 (1931): Two columns, textual and visual contents
The tables below give a statistical overview of this preliminary round of annotations:
|Time (ms) per Annotation||16392||4816||5558||26766|
|Time annotating (hour)||1h05||1h11||0h24||2h40|
|Annotations per category||1906||1919||1931|
|Annotated Images Per Category||1906||1919||1931|
We can already make several observations from these statistics. First, we notice important discrepancies between categories and samples. Some categories are clearly over represented (paragraphs, articles), compared to others that are of very limited use (sub-subheading). The latter will probably be discarded during post processing or will require additional annotations to be properly recognized. Furthermore, some categories are specific to certain samples (advertisements in 1919 and 1931, caption in 1919, footnotes in 1931). They will also require additional annotations to be properly recognized in unknown sets of documents.
We can also gain interesting insights on the structure and content of the journal itself, and how it evolved over time. In 1931, for example, it is obvious that the journal contained more extensive, lengthy articles running over multiple pages. Advertisements clearly dominated in 1919, when readers were confronted with an average of 3 advertisements per page (up to 8 per page).
In future posts, we will discuss the results of the post-processing phase, the validity of the method for the automatic separation of articles in the rest of the collection, and its transferability to other collections.
This project has received the support of the Chiang Ching-kuo Foundation for International Scholarly Exchange. It is part of a collaboration between the European Research Council (ERC) Advanced Grant “ENP-China” project (Elites, Networks, and Power in modern China) led by Christian Henriot (Aix-Marseille University) and the NewsEye project team led by Antoine Doucet at La Rochelle University.
Barman, Raphaël, Maud Ehrmann, Simon Clematide, Sofia Ares Oliveira, and Frédéric Kaplan. “Combining Visual and Textual Features for Semantic Segmentation of Historical Newspapers.” Journal of Data Mining & Digital Humanities, no. HistoInformatics (January 19, 2021): 6107. https://doi.org/10.46298/jdmdh.6107.
Bieler, Stacey. “Patriots” or “Traitors”?: A History of American-Educated Chinese Students. New York: Routledge, 2003.
Chen, Xiao. “‘Liumei Xusheng Jibao’ Jiqi Kejiao Sixiang de Chuanbo (The Chinese Students’ Quarterly and the Dissemination of Ideas Related to Science and Education).” M.A. Thesis, Xibei daxue, 2013.
Markewich, Logan, Hao Zhang, Yubin Xing, Navid Lambert-Shirzad, Zhexin Jiang, Roy Ka-Wei Lee, Zhi Li, and Seok-Bum Ko. “Segmentation for Document Layout Analysis: Not Dead Yet.” International Journal on Document Analysis and Recognition 25, no. 2 (June 1, 2022): 67–77. https://doi.org/10.1007/s10032-021-00391-3.
Meier, Benjamin, Thilo Stadelmann, Jan Stampfli, Marek Arnold, and Mark Cieliebak. “Fully Convolutional Neural Networks for Newspaper Article Segmentation,” 414–19, 2017. https://doi.org/10.1109/ICDAR.2017.75.
Michael, Johannes, Max Weidemann, Bastian Laasch, and Roger Labahn. “ICPR 2020 Competition on Text Block Segmentation on a NewsEye Dataset.” In Lecture Notes in Computer Science, (LNCS, Volume 12668). Springer, 2021. https://doi.org/10.5281/zenodo.4555751.
Zhu, Wenzhen, Negin Sokhandan, Guang Yang, Sujitha Martin, and Suchitra Sathyanarayana. “DocBed: A Multi-Stage OCR Solution for Documents with Complex Layouts.” arXiv, February 3, 2022. https://doi.org/10.48550/arXiv.2202.01414.