Eye tracking for analysing reception of machine translation and sign language (EyeSign)

Recent advancements in the field of natural language processing (NLP) make it clear that language-related AI will play a fundamental role in shaping the digital transformation of society. AI-enhanced machine translation has already offered invaluable solutions for media accessibility.

This study addresses comprehension, subjective perception and emotional reception of human translated vs. machine-translated texts when reading texts or understanding texts in sign language. It is based on an eye-tracking data analysis involving typical readers and deaf and hard of hearing participants.

The eye-tracking data will provide insights into the cognitive processes during language comprehension of both populations allowing us to analyze aspects such as text understanding, subjective perception and emotional reception of the texts. The proposed project will have a direct impact on advancing eye-tracking research in Lithuania. Additionally, the research into machine translation will bring insights regarding the use of machine translation and AI technologies for enhanced inclusiveness of the deaf and heard of hearing.

AI-driven natural language processing (NLP) is transforming digital communication and improving media accessibility. EyeSign explores comprehension, subjective perception and emotional reception of human- vs. machine-translated texts, based on eye-tracking data involving typical and deaf and hard of hearing participants.

The eye-tracking data will shed light on how typical and deaf and hard-of-hearing readers process language, enabling analysis of comprehension, perception and emotional reception of texts. EyeSign will contribute to the development of eye-tracking research in Lithuania and offer insights into how machine translation and AI technologies can support greater inclusion for the deaf and heard-of-hearing.

The idea of the project, grounded in ​​scientific research, originates from a wish to support media accessibility for the deaf and hard-of-hearing people in Lithuania—a field that remains largely unexplored at the national level. Drawing on previous studies (Kasperė et al., 2023), EyeSign hypothesises that cognitive processing differs when individuals engage with human- vs. machine-translated texts. Since gaze behaviour reflects cognitive processes in both reading (Rayner, 1998) and sign language perception (Agrawal et al., 2021), eye-tracking serves as a valuable method for psycholinguistic research.

In Lithuania, no scientific studies have yet explored the possibilities of applying machine translation to enhance media accessibility and inclusion for the deaf and hard-of-hearing people. It remains important to investigate how the integration of AI-based technologies and machine translation would affect the deaf and hard-of-hearing community, and whether such applications would be acceptable and appropriate to the target audiences.

The aim of this study is to analyse the comprehension, subjective perception, and emotional reception of human-translated and machine-translated texts in order to test the potential of AI-based sign language translation solutions to increase media accessibility for the inclusion of the deaf and hard of hearing community.

Research objectives:

• to analyze the regularities of text comprehension, subjective perception and emotional reception when deaf and hard of hearing and hearing participants read human and machine translated text;
• to determine the correlations between text comprehension, subjective perception, emotional reception and gaze tracking parameters (fixation duration, number of regressions, gaze duration, saccade length and duration) when deaf and hard of hearing and hearing participants read human and machine translated text;
• to determine the regularities of text comprehension, subjective perception and emotional reception when deaf and hard of hearing participants interpret texts from human translation and machine translation into Lithuanian in sign language;
• to perform a comparative analysis of text comprehension, subjective perception and emotional reception data of deaf and hard of hearing and hearing participants.

The level of text comprehension can be measured by asking participants carefully prepared text comprehension questions after they have read the text (Berzak et al., 2020). The text comprehension questionnaire will be given to the subjects after they have read the texts. As in eye-tracking studies conducted by other researchers (Björnsdóttir et al., 2023; Jäger et al., 2021), this will allow us to compare and look for correlations between the results of the text comprehension questions and the eye-tracking data, for example, whether longer fixation times always lead to better comprehension and whether there are differences in the comprehension of human and machine-translated texts.

Subjective perception. This study will not only ask questions about text comprehension, but also aims to determine the subjective perception of readers and sign language users about the translation method, e.g. whether human and machine translated texts are equally acceptable to readers and sign language users. In this study, we will assess the subjective perception of participants on a Likert scale (Andry, 2019).

Emotional reception. This study also analyzes the emotional reception of texts, i.e. whether participants react positively (happy) or negatively (sad) to certain types of text presentation and whether the translation method affects emotional reception. It also aims to verify the results of other studies on whether it is possible to identify different emotional reactions to text based on eye tracking data (Ballenghein et al., 2019; Van den Hoven et al., 2016). Therefore, this study will also have an impact on testing eye tracking as a methodological approach.

The experiments conducted in this study will use texts for reading and video recordings in which sign language interpreters will translate (interpret) texts translated by a human translator and machine-translated texts into Lithuanian sign language. For this purpose, Lithuanian sign language interpreters will be hired, working at the Lithuanian Sign Language Translation Center in the Kaunas territorial branch, whose purpose is to provide sign language translation services and improve information accessibility for the deaf. This scientific study would help to delve deeper into the importance of sign language translation, technological opportunities and challenges, as well as various aspects of perception that may affect people using this service.

Several different stimuli will be used in the gaze tracking experiment:
1. texts translated into Lithuanian by a human translator (the MultiplEYE text library is used, i.e. human translations of original fiction, administrative and popular science texts from various languages ​​into Lithuanian, accumulated and validated by scientists for gaze tracking experiments);
2. texts translated into Lithuanian using machine translation (the same original texts are used, only machine translated, a new text library of machine translated texts is created for gaze tracking experiments);
3. texts interpreted (presented in video format) in Lithuanian sign language (human translator and machine translated texts are used, which are interpreted (translated) into Lithuanian sign language by a professional sign language interpreter; a set of video recordings in sign language is created);

The study involves 2 groups of subjects:
1. Typical readers, healthy hearing adults, native Lithuanian speakers;
2. Lithuanian sign language readers, native Lithuanian speakers, including deaf and hard of hearing people.

Technology used in EyeSign:

Eyelink 1000 Plus package is used to implement eye tracking experiments, which allows for binocular (both eyes) gaze fixation. The KTU laboratory ensures conditions for high-quality instruction of subjects and experiment performance (adjustable intensity lighting, variable height tables).

Several eye tracking indicators are used to analyze eye tracking data and assess their relationship with text comprehension questions: fixation duration and number, regression number, gaze duration, saccade length and duration.

The scientific results of EyeSign and their publication are awaited:

Dataset – a set of eye-tracking data collected during experiments in healthy and deaf and hard-of-hearing subjects reading human-translated and machine-translated text. The dataset is stored in the Lithuanian HSM Data Archive (LIDA).

Scientific article – in 2025 it is planned to prepare a scientific article in English. Tentative title – “Cognitive disparities in comprehension of human translated vs. machine-translated texts”. It is planned to submit it to the scientific journal “Cognitive Linguistics” (WoS IF 1.7) or another scientific journal indexed in the Web of Science database with a similar rating.

Scientific article – in 2026 it is planned to prepare a scientific article in English. Tentative title – “An eye-tracking study on typical readers and sign language users”. It is planned to submit it to the scientific journal “Translation & Interpreting: The International Journal of Translation and Interpreting Research” (Scopus Q1, WoS Emerging sources citation index) or another scientific journal indexed in the Web of Science database with a similar rating.

Conference presentations at the international scientific conference “International Cognitive Linguistics Conference” in 2025. Conference paper at the international symposium Symposium on Eye Tracking Research and Applications 2026. Conference paper at the international conference “Media for all” 2027.

Scientific monograph – by the end of the project, it is planned to prepare a manuscript of a scientific monograph and submit it for review. The volume of the scientific monograph – at least 8 author’s arcs. The monograph will be prepared in English. Preliminary title – “Exploring the possibilities of machine translation in sign language communication”.

Expected impact of the scientific results of the project and their presentation to the general public:

EyeSign has a direct impact on the development of eye tracking research in Lithuania. In addition, the topic of machine translation affects the general public, which is important to inform about the correct application of machine translation and the use of modern artificial intelligence technologies. The results of this study will increase the general public’s understanding of the quality of human and machine translated texts and Lithuanian sign language. This will help to make strategic decisions regarding the appropriate application of machine translation in various media and contexts, including official events, celebrations, the world of entertainment, education and communication. Researchers and technology developers in the fields of philology, education, computer science and artificial intelligence will be able to use the results obtained during this project and develop solutions suitable and acceptable for the deaf and hard of hearing communities in interdisciplinary teams of scientists and business representatives.

The project partners, the Lithuanian Society of the Deaf, will help to involve deaf and hard of hearing people in the implementation of the experiments carried out in the project, the deaf and hard of hearing will be provided with opportunities to share their opinions and participate in scientific research, and relations between the hearing and deaf and hard of hearing communities will be promoted. Members of the deaf community and their associations will be invited to dissemination events at KTU.

Project no.: S-MIP-24-32

Project funding: Projects funded by the Research Council of Lithuania (RCL), Projects carried out by researchers’ teams

Project general results: Based on the research results obtained during the project, an eye tracking dataset, 2 research papers, 3 conference papers and a scientific monograph are being prepared.

Period of project implementation: 2024-09-02 – 2027-06-30

Project coordinator: Kaunas University of Technology

Project Head: Prof. Dr. Ramunė Kasperė

Project Duration in years: 2024 – 2027

Department:
Academic Centre of Social Sciences, Arts and Humanities, Faculty of Social Sciences, Arts and Humanities

Science areas: Humanities and Social Sciences

Scientific fields: Philology and Communication and Information