Introduction

Text encoding research has mainly focused on textual information, leaving the structural representation of image–text relationships in illustrated fiction underexplored. Recent studies on Japanese pre-modern classics have applied TEI and IIIF to digital archives and image–text integration projects involving picture scrolls and maps. However, these studies have primarily focused on named entities and searchability rather than semantic or interpretive relationships between text and image (Nakamura et al. 2021). This study proposes a method for structurally encoding these relationships in kibyōshi, a genre of early modern Japanese illustrated fiction published mainly between 1775 and 1818.

Kibyōshi as a Case Study

Kibyōshi is particularly suitable for this encoding model because it frequently employs close associations between text and image while remaining comparatively less abstract than other illustrated literary traditions, such as Victorian novels. Whereas Victorian illustrations often produce ambiguity or tension with the text, image–text relationships in kibyōshi are generally synergetic, reinforcing scenes or conveying information concisely. Their interpretation depends heavily on the audience's shared cultural knowledge of Edo-period customs, legends, and theatre, requiring participation in a “double reading” process between image and text. At the same time, these visual and textual expressions often refer to concrete figures or events and involve relatively limited interpretive variation among readers. This makes kibyōshi an appropriate corpus for demonstrating a structural annotation method for image–text relationships(John 2016; Nakamura 1982).

Technical Framework: Integrating TEI and IIIF

To capture the multifaceted relationships between visual and linguistic layers, our approach integrates two major standards: IIIF (International Image Interoperability Framework) (Institute for Digital Humanities 2024) and the TEI (Text Encoding Initiative) Guidelines　(Institute for Digital Humanities 2022).

• IIIF: We utilize the IIIF Presentation API to annotate web-accessible source images, allowing for the referencing of specific regions within illustrations.
• TEI: We employ TEI to structurally encode the text and provide a consistent framework for linking textual passages to image annotations.

Our workflow utilizes the Oxygen XML Editor for both TEI-compliant markup and image annotation via its Image Map Editor function. This process defines polygonal <zone> tags (containing coordinate information) within the <facsimile> element of the TEI file (Figures 1 and 2). These zones are then referenced by @facs attributes in the text to link spoken dialogue or character mentions directly to their corresponding visual representations (Figures 3 and 4).

Figure 1: Mechanism for describing IIIF images using the <facsimile> element.

Figure 2: The interface of the Image Map Editor in Oxygen XML Editor.

Figure 3: Linking text to specific image regions using the @facs attribute.

Figure 4: Synchronized display of dialogue and speakers.

(https://teikem.kemco.keio.ac.jp/dihuco/teikem_viewer/?file=kibyoshi/tsunedunegusa_20260331.xml)

Annotation Structure and Visualization

We have developed specific TEI encoding methods to capture a variety of annotation types representing image-text relationships:

• Text-only Annotation: <seg> elements within the text are linked to explanatory <note> elements via the @corresp attribute (Figures 5 and 6).

Figure 5: Example of TEI markup for textual annotation.

Figure 6: The resulting display of the textual annotation in the interface.

(https://teikem.kemco.keio.ac.jp/dihuco/teikem_viewer/?file=daihino_senroppon_20260512_DH2026.xml)

• Image-only Annotation: <note> elements with incorporating the @target attribute are used to annotate specific regions within illustrations (Figures 7 and 8).

Figure 7: Example of TEI markup for image annotation.up.

Figure 8: The resulting display of the image annotation in the interface.

(https://teikem.kemco.keio.ac.jp/dihuco/teikem_viewer/?file=daihino_senroppon_20260512_DH2026.xml)

• Dual Annotation (Text and Image): Text segments (<seg>) are linked directly to image regions (<zone>) via the @facs attribute, while shared annotations are connected using a combination of @corresp and @target (Figures 9 and 10).

Figure 9: Example of integrated annotations for both text and images.

Figure 10: Synchronized display of textual notes and image

(https://teikem.kemco.keio.ac.jp/dihuco/teikem_viewer/?file=daihino_senroppon_20260512_DH2026.xml)

• External Image/Data Reference: External IIIF-compliant visual materials, such as ukiyo-e prints, are integrated through <figure> and <graphic> elements (Figures 11 and 12).

Figure 11: Example of TEI markup for referencing external images and data.

Figure 12: The resulting display of integrated external image and data references.

(https://teikem.kemco.keio.ac.jp/dihuco/teikem_viewer/?file=kibyoshi/tsuunohitokoe_20260331.xml)

The resulting structured data is displayed using the TEI Classical Text Viewer (“TEI Kotenseki Byūwa”), which visually synchronizes the text and image. The viewer enables users to highlight corresponding annotations or images by hovering over the text or image regions (Figures 4, 6, 8, and 12).

Conclusion and Future Work

This method provides a robust framework for digitally preserving and interpreting the complex expressive structures of illustrated fiction. While the current annotations reflect modern scholarly interpretations, the TEI-based structure inherently supports the parallel encoding of multiple interpretive layers.

Future work will focus on developing methods to visualize this interpretive plurality within reading environments and exploring semi-automated annotation approaches. These technical refinements are particularly aimed at addressing the limitations observed in existing general-purpose segmentation models, such as SAM (Kirillov et al. 2023), when applied to historical Japanese materials.