Interaction between Phonological and Semantic Processes in Visual Word Recognition using Electrophysiology

Summary

We present a protocol to explore the relative activation sequence of phonology and semantics in visual word recognition. The results show that consistent with interactive accounts, semantic and phonological representations may be processed interactively, and higher-level linguistic representations may affect early processing.

Abstract

Controversies have always existed in research related to reading abilities; on whether printed words are perceived in a feedforward manner based on orthographic information after which, other representations, such as phonology and semantics are activated, or whether these are fully interactive and high-level semantic information affects early processing. An interference paradigm was implemented in the presented protocol of phonological and semantic judgment tasks that utilized the same precede-target pairs to explore the relative order of phonological and semantic activation. The high- and low-frequency target words were preceded with three conditions: semantically related, phonological-related (homophones), or unrelated. The results showed that the induced P200 component of low-frequency word pairs was significantly greater than high-frequency words in both the semantic and phonological tasks. In addition, both the homophones in the semantic task and the semantically related pairs in the phonological task caused reduction in N400 when compared to the the control condition, word frequency-independently. It is worth noting that for the low-frequency pairs in the phonological judgment task, the P200 released by the semantically related word pairs was significantly larger than that in the control condition. Overall, semantic processing in phonological tasks and phonological processing in semantic tasks were found in both high- and low-frequency words, suggesting that the interaction between semantics and phonology may operate in a task-independent manner. However, the specific time this interaction occurred may have been affected by the task and frequency.

Introduction

The critical issue in any word recognition model is understanding the role of phonology in the process of semantic access¹. For alphabetic languages, many studies consistently view phonology as playing an important role in semantic access, including English^2,3,4, Hebrew⁵, French⁶, and Spanish⁷. In other words, written word recognition involves not only orthographic but also phonological and semantic processing. This observation in the interactive connectionist model is explained by extensions activated throughout the network, where orthography is associated with phonological and semantic representations through weighted connections⁸. This proliferation of activation provides the core mechanism for the visual word recognition model, which assumes that phonological and semantic representations are automatically activated in response to orthographic input⁹.

However, current empirical evidence supporting the hypothesis of interactive automation remains controversial. Some studies claim that the activation of phonological and semantic representations can be adjusted or prevented by task demands or attention, which implies a certain top-down influence on the high-level processes involved in word perception^10,11. However, the aforementioned description has been questioned by many findings that report phonological and semantic effects in visual word recognition even though these representations are completely irrelevant to the task or cannot be directly accessed¹², thereby supporting the view that semantics and phonology may be accessed automatically and forcibly during the reading process¹³. Therefore, there is uncertainty on whether the phonological and semantic activation in visual word recognition depends on the specific task or whether it occurs forcibly and automatically in a task-independent manner.

The answer to aforementioned question is difficult for Chinese readers. Compared to English, Chinese is a logographic script whose characters represent morphemes instead of phonemes¹⁴. At present, the role of phonology for semantic access to Chinese words remains controversial. Some studies have claimed that phonology plays an important role in semantic access to Chinese words^15,16,17. Others, however, have held the opposite view^18,19. After evaluating the aforementioned research for Chinese phonological processing, we found that the experimental paradigm and specific research methods differ. On the whole, it was mainly divided into two paradigms: word priming^15,18,19 and violation paradigm in the sentence^17,20,21. The target word is usually embedded at the end of the sentence in the violation paradigm²². In terms of language mechanism, a short two-word phrase is a more manageable unit than a complete sentence that is difficult to process²³. In addition, variables that are difficult to control in a sentence, such as syntax, context, or other factors, may lead to different conclusions²⁴. The word priming paradigm is a method commonly used to explore word recognition models, whether in alphabetic languages or Chinese. The task of this paradigm is to judge whether the target word preceded by the primes is a real word or pseudoword; that is, this paradigm usually contains only one lexical task. However, a single lexical decision task may not be the best choice to solve the problem of whether the activation of phonology and semantics depends on the task. Therefore, two different tasks may be more suitable for exploring this question.

Therefore, this research aimed to explore the role of phonology in Chinese word recognition and simultaneously attempt to determine whether the activation of phonology and semantics is in task independent. Our research includes two tasks using the interference paradigm: semantic judgment and phonological judgment. To the best of our knowledge, this is the first event-related potential (ERP) study of Chinese two-character compound recognition using this interference paradigm, and this method rarely appears in studies of alphabetic languages. Specifically, in the semantic judgment task, participants must judge whether the target word and its precedent are semantically related, while in the phonological task, they must judge whether the paired words have the same pronunciation.

The former is a semantic matching task that does not require a priori phonological processing, and the latter is a phonological judgment task that does not require a priori semantic processing. Therefore, we compared homophone pairs and unrelated control groups in the semantic judgment task to reveal whether and how phonology affects semantic processing. Similarly, we compared semantically related word pairs with unrelated control conditions in the phonological judgment task to reveal whether and how semantics affect phonological processing. In addition, the aforementioned problem was verified in high- and low-frequency words. Thus, this complementary semantic and phonological judgment task can not only reveal the importance of phonological processing in Chinese word recognition but also reveal whether and how phonology and semantics interact.

If the phonology and semantic processes are early, automatic, and interactive, the effect of phonological and semantic activation should be observed in the response time of the two tasks. For ERP, phonological and semantic processes trigger two different electrophysiological markers^2,7. In addition, their time courses and their spatial distributions should be different. An early positive component (P200) should reflect phonological processing, and the typical semantic processing marker N400 should also be identified^20,21. We assumed that both the phonologically related pairs in the semantic task and the semantic-related pairs in the phonological task would cause a significant decrease in N400, which would have indicated that phonological processing may lead to some degree of activation at lexical-semantic levels. In addition, we monitored whether the P200, which characterizes phonological processing, appeared in the semantic judgment task or the phonological judgment task. In the phonological judgment task, semantic-related conditions trigger the P200, which can be seen as evidence of the early influence of semantics on phonological processing.

Protocol

The protocol used for this study was approved by the Institutional Review Board of Tsinghua University.

1. Stimuli construction and presentation

1. Stimuli construction
   1. Stimuli preparation: Prepare target words containing approximately 140 Chinese two-character compounds, of which low- and high-frequency words account for half. Precede each target by three analogues: a phonologically identical word (homonymous word), a word with related meaning, and an irrelevant control word.
      1. Ensure that high-frequency targets are always preceded by high-frequency antecedents and low-frequency targets are always preceded by low-frequency precedents, whether in related conditions or unrelated control groups. In addition, ensure the precede-target pairs are similar in the number of strokes and frequency.
         NOTE: All the words for this study were selected from the Modern Chinese Frequency Dictionary (Xiandai Hanyu Pinlu Cidian). The frequency of low-frequency words was less than eight times per million, and the frequency of high-frequency words exceeded 800 times per million.
   2. Stimuli assessment: Recruit a separate group of approximately 30 students to evaluate the degree of semantic relevance between word pairs on a seven-point scale, wherein 1 reflects the lowest correlation and 7 reflects the highest correlation.
   3. Final stimulus determination: Delete inappropriate word pairs, such as word pairs with lower scores in semantically related conditions and word pairs with higher scores in homophone and irrelevant condition scores.
      1. Calculate the respective average scores of high- and low-frequency semantic-related word pairs and ensure no significant difference is present between the two.
      2. In addition, ensure that the scores of homophone pairs and unrelated pairs are not significantly different at both high and low frequency. Finally, determine the final experimental stimulus material (see Table Materials).
         ​NOTE: For the semantic-related values of the semantic-related pairs in this experiment, the final average values were 5.62 and 5.73 for high-frequency and low-frequency pairs, respectively, and there was no significant difference between the two (p > .1). In addition, the semantic relatedness between the homophonic and unrelated pairs was not significantly different (p > .1).
2. Stimuli presentation
   1. Build a program to show the task to the subjects and fill in the aforementioned materials (the program can be written in E-prime or other programming languages).
   2. Ensure that each core structure of the program starts with a screen displaying a " + " sign that lasts for 300 ms, directly after which the preceding word must appear for 140 ms, with no interval between the two.
   3. After that, set a blank screen lasting for 360 ms, and then set the target word, which will appear for 500 ms. Finally, set a question mark (?) that will continue to be displayed until the participant has decided on the word pair just shown and pressed the button as quickly and accurately as possible.
   4. Tell the participants in advance that they need to judge whether the word pair is semantically related in the semantic judgment task and whether the phonology is the same in the phonological judgment task.
   5. Practice session setup: Set up two practice groups to include semantic judgment and phonological judgment tasks, respectively, with no less than 10-word pairs for each task. Inform the participants that they can repeat the exercises to ensure that the accuracy in the practice session is more than 70%.
   6. Formal experiment setup: Divide the whole experiment into 6 blocks, with the semantic judgment task and the phonological judgment task each accounting for half.
      1. Make sure that there are no repeated target words in each block, and that the number of priming types in each block is the same. In addition, set up a few filler trials to reduce the response deviation caused by the unequal number of tests requiring positive or negative reactions.
      2. Randomize the order of items in each block and counterbalance the block order among the subjects.
         ​NOTE: The entire experiment can also be divided into eight or ten or more blocks according to the number of experimental materials to be prepared, which minimizes the repetition of target words in each block.

2. Experiment preparation and electrophysiological recording

1. Recruit right-handed native Chinese speakers with a normal vision that may have been previously corrected.
   1. Exclude participants with any neurological or psychiatric diseases.
   2. Ensure there is a balanced number of female and male participants in the desired age range (18-28 years old).
   3. Ensure that the participants do not have any history of perming or dyeing their hair for the past two months.
   4. Inform the participants that they will need to have sufficient sleep and rest time before the experiment²⁵.
   5. When participating in the experiment, please make sure the participants are in a healthy state at the time of conducting the experiment.
2. When a participant arrives at the lab, introduce the experimental equipment, tasks, and time costs. Explain the requirements (such as not being sleepy, moving, and blinking) to help them understand the entire process and eliminate unnecessary worries.
3. If the participant has no other questions about the experiment, ask them to fill in the Edinburgh Handedness Query Form, which is used to confirm that all participants have the same right-handed habits.
4. Provide the informed consent form to the participants and ask them to read and sign carefully. If participants have questions about the content of the consent form, provide them with the necessary explanations.
5. Instruct the participant to properly clean their scalp and dry their hair in the laboratory. While waiting for participants, please prepare all experimental materials.
   NOTE: The electroencephalogram (EEG) signal is amplified using an amplifier system with a bandpass of 0.01 to 100 Hz and continuously sample at 500 Hz.
6. Invite the participants to sit comfortably on a chair in the chamber where the experiment will be conducted. Instructed them not to move the chair.
7. Use cotton swabs and facial scrubs to clean the skin under the participant's left eye (for the vertical electro-oculographic electrode), near the outer canthus of the right eye (for the horizontal electro-oculographic), and around the right and left mastoid bones (for Tp9 and Tp10, which will be used as new offline references).
   NOTE: The distribution of electrodes may vary depending on the caps used.
8. Place the elastic cap on the participant's head and make sure that the Cz electrode is at the center of the top of the head. Fix the electrode cap strap under the chin with care to ensure that it is not too tight or too loose.
9. Make sure that the cap and amplifier are connected to the recording system. Next, switch the recording software to the impedance monitoring interface.
10. Ensure that the impedance of all electrodes does not exceed 5 kΩ or 10 kΩ, starting with the reference (Ref) and ground (Gnd) electrodes.
11. Pass the syringe filled with conductive gel through the small hole of an electrode to the scalp and then push the plunger to inject a small amount of conductive gel into the scalp while being careful not to cause an overflow. At the same time, monitor the display system that displays the impedance in real-time until the impedance drops to the threshold.
12. After the Ref and Gnd electrodes are prepared, reduce the impedance of the other electrodes in the same way. Treat the impedance reduction of ocular electricity carefully.
    1. Tape the small holes on one side of the two electro-oculographic electrodes to prevent the injected conductive gel from leaking. Fix them to the bottom of the left eye and the outer canthus of the right eye with tape.
13. After all the electrodes are prepared, instruct the participants to be ready for the experiment. Instruct the participants to relax and avoid excessive eye blinking and body movement during the experiment.
14. Present the stimulus via the stimulus demonstration program and let the participants practice in the practice section.
    NOTE: After the practice session, participants can ask questions if they have any doubts or questions about how to proceed.
15. Start the formal experiment and record the EEG information. Monitor the recording system during recording. If an electrode is loose or the resistance exceeds the threshold, refill the electrode when the participant is resting.
    NOTE: Participants can rest for 4-10 min after each block.
16. After the experiment is completed, save the EEG signal and turn off the equipment, such as the recording system and amplifier. Then take off the participant's cap and instruct the participant to wash off the conductive gel from the hair and skin. Finally, reward the participants and thank them for their cooperation.

3. EEG preprocessing

1. Utilize semi-automatic ocular correction with independent component analysis.
2. Compute the ERPs from 100 ms to 600 ms after the onset of the target word (100 ms pre-target baseline).
3. Set the EEG bandpass-filtered offline from 0.05 to 30 Hz (zero phase shift mode, 24 dB/oct).
4. Discard epochs exceeding ±80 µV by artifact rejection and eliminate the trials of erroneous responses.

Representative Results

This protocol was used in a recent study to investigate the role of phonology in Chinese two-character compound recognition and to infer the word recognition model²⁶. All stimuli used in this study were fully disclosed²⁶. Three time windows were selected on the basis of global field power (GFP): at 100-150 ms, 160-280 ms, and 300- 500 ms for N1, P200, and N400 components, respectively²⁶. The average amplitudes of the above two time windows were analyzed by repeated measures analysis of variance (ANOVA), and frequency (low and high), relationship type (phonologically or semantically related, or unrelated), and lateral areas (left and right hemispheres × front, middle, and posterior regions = six areas in total) or midline electrodes (Fz, Cz, Pz), which were the three within-participant factors involved. More detailed results and graphs can be found in Wang et al. (2021)²⁶.

ERP results for the semantic judgment task
The 100-150 ms Period (N1)
For the midline electrodes, ANOVA yielded a main effect of Frequency [F(1, 23) = 9.451, P = .005, ƞ²_p = .291], indicating that high-frequency pairs elicited a significantly more negative waveform than low-frequency condition. ). A similar significant main effect of Frequency was also observed at the lateral sites. In addition, for high-frequency pairs, a significant main effect of Relation Type was also observed [F(1, 23) = 8.826, P = .007, ƞ²_p = .277], showing that unrelated pairs elicited a significantly more negative waveform than homophone condition. A similar significant main effect of Relation Type for high-frequency pairs was also observed at the left hemisphere.

The 160-280 ms Period (P200)
For the midline electrodes, ANOVA yielded a main effect of Frequency [F(1, 23) = 5.546, P = .027, ƞ²_p = .194], indicating that low-frequency pairs elicited a significantly more positive waveform than high-frequency condition. No other significant effects or interactions were observed in the midline electrodes. Moreover, the main effect of Frequency was also found at lateral sites.

The 300-500 ms Period (N400)
In the 300-500 ms time window, ANOVA yielded a significant main effect of Relation Type [F(1, 23) = 27.783, P < .001, ƞ²_p = .547] in the midline electrodes, showing that a target primed by homophones elicited significantly less negative amplitude than did the unrelated condition (see Figure 1). A similar significant main effect of Relation Type was observed at lateral sites.

Figure 1: Grand mean event-related potentials in response to target words, from representative electrodes (Fz, Cz, Pz), for homophonic and control pairs in the semantic task. This figure, taken from Wang et al. (2021)²⁶, shows that in the semantic judgment task, homophone pairs released a smaller N400 components than irrelevant conditions regardless of high and low frequencies. Please click here to view a larger version of this figure.

ERP Results for the Homophone Judgment Task
The 100-150 ms Period (N1)
No significant effect or interaction was found in the midline electrodes or lateral sites.

The 160-280 ms Period (P200)
No significant main effect for Relation Type or Frequency (ps > .1) was observed at the anterior frontal electrodes. However, a significant interactive effect between Frequency and Relation Type was found [F(1, 23) = 7.951, P = .010, ƞ²_p = .257]. Further analysis found that the influence of Relationship type was only significant under low-frequency conditions at the anterior frontal electrodes (FPz: P =.055; FP1: P =.027; FP2: P =.004; AF3: P =.060; AF4: P =.021; AF8: P =.009), indicating that in the P200 time window, the ERP signal was significantly more positive under semantically related conditions than under unrelated conditions (see Figure 2).

In addition, the analyses showed that the effect of Frequency was significant in two regions (left central: F(1, 23) = 4.506, P = .045, ƞ²_p = .164 and left posterior: F(1, 23) = 10.470, P = .004, ƞ²_p = .313).

Figure 2: Grand mean event-related potentials in response to target words from six anterior frontal electrodes (Fpz, Fp1, Fp2, AF3, AF4, AF8) for the semantically related and control pairs of low-frequency in the homophone task. This figure, taken from Wang et al. (2021)²⁶, shows that in the phonological judgment task, low-frequency semantic related words released a more positive P200 component than unrelated words. Please click here to view a larger version of this figure.

The 300-500 ms Period (N400)
In the N400 time window, a significant main effect of Relation Type was found [F(1, 23) = 9.082, P = .006, ƞ²_p = .283] in the midline electrodes, indicating that the target primed by semantically-related words released a significantly less negative amplitude than unrelated primes (see Figure 3). In addition, a significant main effect of Relation Type was observed at the lateral sites.

Figure 3: Grand mean event-related potentials in response to target words, from representative electrodes (Fz, Cz, Pz), for the semantically related and control pairs in the homophone task. This figure, taken from Wang et al. (2021)²⁶, shows that in the phonological judgment task, semantically related pairs released a smaller N400 components than irrelevant conditions regardless of high and low frequencies. Please click here to view a larger version of this figure.

Discussion

Experimental results and significance:
The purpose of this protocol was to infer the following: 1) whether the word recognition model is a feedforward model or an interactive model and 2) the interaction between the phonological and semantic patterns in Chinese two-character compound recognition of high and low frequency under different tasks. An interference paradigm of phonological and semantic matching task using the ERP technique was adopted. The ERP responses preceded by homophones and unrelated words to targets were compared in semantic judgment tasks that did not require a priori phonological processing to reveal when and whether phonology affects semantic processing. Similarly, for phonological judgment tasks that did not require a priori semantic processing, the ERP responses of target words triggered by semantic and unrelated words were compared to reveal whether and semantics interferes with phonological processing. Next, according to the latency of the relevant ERP components, the relative time course of semantic and phonological processing was compared, and the influence of word frequency on such processing patterns was checked.

Our results showed that the target primed by semantically related precedents in the phonological task and the targets primed by homophones in the semantic judgment task both triggered a significantly less negative N400 component than did the unrelated precedes, regardless of word frequency. Hence, for high-frequency words, the data suggested that semantic activation occurs earlier or at least no later than phonological processing during the recognition of Chinese two-character compounds in different tasks. In addition, the induced P200 component of low-frequency word pairs were significantly more positive than that of high-frequency word pairs in both semantic and phonological tasks. Other studies have also concluded that early ERP components may be sensitive to word frequency^27,28. The earliest N1 and the P200 effects can also be, at least partly, due to the semantic processing of the previous word. However, for low-frequency words in the phonological judgment task, it was found that the semantically related pairs released a significantly larger P200 than the control condition. In contrast, it was found that P200 triggered by low-frequency semantically related word pairs was significantly more positive than low-frequency control conditions in phonological judgment tasks. This result does not seem to be difficult to explain for low-frequency words, as phonological activation is expected in phonological judgment tasks, but the obvious P200 component was triggered by semantic precedents, which again strengthened the hypothesis that semantic processing may not occur later than phonological processing.

The above interaction between semantics and phonology confirmed the interaction model of word recognition that proposed that the system may be fully interactive, with low-level information flowing from bottom to top to the entire lexical information and high-level information flowing from top to bottom to form early visual word processing¹. In addition, the P200 caused by frequency effects in the two tasks also confirmed the speculation that higher-level linguistic information may already exert its influence during early processing. Note that the interactions found in the two tasks, regardless of the high- and low-frequency conditions, supported the automatic and possibly mandatory access to phonology and meaning during reading. However, the specific time at which this interaction occurred may have been affected by the task and frequency. For example, for low-frequency words, it was found that the interaction occurred in the P200 time window in the phonological task while in the N400 time window in the semantic judgment task. Nevertheless, for high-frequency words, the interaction was observed in the N400 time window for both semantic or phonological tasks. In conclusion, the current findings suggest the automatic interaction of semantics and phonology in a task-independent manner while considering that the interactive time and mode may be affected by tasks, frequency, etc.

Effectiveness of the method
In general, this interference paradigm can more comprehensively explore the interaction modes of phonological and semantic processing. Our experiments included phonological matching tasks that do not require priori semantic processing and semantic matching tasks that do not require priori phonological processing. In this way, the influence of semantics on phonological processing or the influence of phonology on semantic processing can be observed more clearly. Furthermore, as the phonology or semantics of the precedents and the target word need to be compared in both tasks, the phonology or semantics are forcibly activated in the two tasks. Therefore, if any interference effect occurs, it will be more obvious. The common method of exploring word recognition is a lexical decision task that includes priming conditions. Specifically, it only needs to judge whether the target word is a real word or a pseudoword. First, the semantic activation of lexical decision tasks may not be strong enough, and second, a single judgment task cannot explore the interaction mode under different tasks. Therefore, the interference paradigm of the two tasks may be more suitable for exploring the word recognition model. For the two distinct tasks of the interference paradigm, one needs to strongly activate semantic processing, and the other needs to strongly activate phonological processing, which is more conducive to exploring whether the interaction between phonology and semantics is task-independent and how to interact under different tasks.

Future applications of the technique
The present protocol was the first to use the interference paradigm to explore the semantic access to Chinese two character compound of high- and low-frequency. Currently, the two-task interference paradigm rarely appears in the studies of word recognition in alphabetic languages. Therefore, this method may provide a new opportunity for different languages characterizing by different relations between orthography, phonology, and semantics to explore word recognition models.

Materials

Name	Company	Catalog Number	Comments
BrainAmp DC amplifier system (Brain Products GmbH)	Brain Products, Gilching, Germany		BrainAmp S/N AMP13061964DC Input 5.6DC=150mA Operation 7mA Standby
Easycap (Brain Products GmbH)	Brain Products, Gilching, Germany		62 Ag/AgCl electrodes with a configuration of the international 10–20 system of electrode