Journal of Electronic Science and Technology  2019, Vol.17 Issue (3): 260-277   DOI: 10.11989/JEST.1674-862X.80520017   PDF    
http://dx.doi.org/10.11989/JEST.1674-862X.80520017
51K

Article

Xue-Yan Li, Hui-Li Wang, Pertti Saariluoma, Guang-Hui Zhang, Yong-Jie Zhu, Chi Zhang, Feng-Yu Cong, Tapani Ristaniemi
Processing Mechanism of Chinese Verbal Jokes: Evidence from ERP and Neural Oscillations
Journal of Electronic Science and Technology, 2019, 17(3): 260-277
http://dx.doi.org/10.11989/JEST.1674-862X.80520017

Article History

Manuscript received May. 20, 2018
revised December. 10, 2018
Processing Mechanism of Chinese Verbal Jokes: Evidence from ERP and Neural Oscillations
Xue-Yan Li, Hui-Li Wang , Pertti Saariluoma, Guang-Hui Zhang, Yong-Jie Zhu, Chi Zhang, Feng-Yu Cong, Tapani Ristaniemi    
X.-Y. Li, H.-L. Wang, G.-H. Zhang, and C. Zhang are with the School of Foreign Languages, Dalian University of Technology, Dalian 116024 (e-mail: lixueyan@dlut.edu.cn; huiliw@dlut.edu.cn);
P. Saariluoma, Y.-J. Zhu, and T. Ristaniemi are with University of Jyväskylä, Jyväskylä FI-40014 (e-mail: pertti.saariluoma@jyu.fi);
F.-Y. Cong is with the Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology, Dalian 116024
Manuscript received 2018-05-20; revised 2018-12-10
Xue-Yan Li was born in 1979. She received the B.S. degree in English from Changchun University of Technology, Changchun in 2002 and the M.S. degree in English education from Maquarie University, Sydney in 2007. She received the Ph.D. degree from the Faculty of Information Technology, University of Jyväskylä, Jyväskylä in 2018. Now she is working with the School of Foreign Languages, Dalian University of Technology, Dalian, as an associate professor. Her research interests include neurolinguistics and technology design.
Hui-Li Wang was born in 1968. She received the B.S. degree in English from Shanghai International Studies University, Shanghai in 1989 and the M.S. degree in education from Dalian University of Technology in 2004. She received the Ph.D. degree in engineering from Dalian University of Technology in 2008. She is currently working with the Institute for Language and Cognition, School of Foreign Languages, Dalian University of Technology, as a professor. Her research interests include neurolinguistics, cognitive linguistics, and second language acquisition.
Pertti Saariluoma received both M.A. and Ph.D. degrees in psychology from University of Turku, Turku in 1978 and 1984, respectively. From 1981 to 1982 and in 1996, he was a visiting researcher with University of Oxford, Oxford. In 1989, he was a visiting researcher with University of Cambridge, Cambridge. In 2008, 2009, and 2011, he was a visiting researcher with University of Granada, Granada. He has introduced a number of independent scientific paradigms. He is currently working with the Faculty of Information Technology, University of Jyväskylä. His research interests include psychology and cognitive science and technology design.
Guang-Hui Zhang was born in 1989. He received his B.S. degree in telecommunications engineering from Dalian Polytechnic University, Dalian in 2015, and the M.S. degree in signal processing from Dalian University of Technology in 2018. He is currently pursuing his Ph.D. degree with the School of Foreign Languages, Dalian University of Technology. His research interests include signal processing in electroencephalography, principal component analysis, independent component analysis, and time-frequency analysis.
Yong-Jie Zhu was born in 1987. He received the B.S. and M.S. degrees in biomedical engineering from Dalian University of Technology in 2013 and 2016, respectively. Now he is pursuing the Ph.D. degree with University of Jyväskylä. His research interests include signal processing in electroencephalography, principal component analysis, independent component analysis, and time-frequency analysis.
Chi Zhang was born in 1987. He received his B.S., M.S., and Ph.D. degrees from Northeastern University, Shenyang in 2010, 2012, and 2016, respectively. He is currently working with the School of Foreign Language, Dalian University of Technology, as a lecturer. His research interests include biomedical signal processing, brain-computer interface, and cognitive science.
Feng-Yu Cong received the B.S. degree in power and thermal dynamic engineering and the Ph.D. degree in mechanical design and theory from Shanghai Jiao Tong University, Shanghai in 2002 and 2007, respectively. He also received the Ph.D. degree in mathematical information technology from University of Jyväskylä in 2010. Since March 2007, he has been working with the Department of Mathematical Information Technology, University of Jyväskylä, as a postdoctoral researcher from March 2007 to August 2008, a junior lecturer (faculty position) from September 2008 to June 2011, and a tenure-track faculty position from July 2011 to December 2013. In May 2012, he was conferred the title of docent (adjunct professor, tenured academic title, ranking between lecturer and full professor) in signal processing at the Department of Mathematical Information Technology, University of Jyväskylä. Since December 2013, he has been a professor with the Department of Biomedical Engineering, Faculty of Electronic Information and Electrical Engineering, Dalian University of Technology. He is also an IEEE Senior Member (from 2013), editorial board member of Journal of Neuroscience Methods (from 2013), and program committee member of LVA/ICA2012&2015, MLSP2013-2015. His research interests include signal processing, cognitive neuroscience, brain science, and machine learning.
Tapani Ristaniemi received his M.S. degree in mathematics and Ph.D. degree in signal processing for communications from University of Jyväskylä in 1995 and 2000, respectively. He is currently working with University of Jyväskylä, as a professor. His research interests include signal processing for wireless communications, brain signal processing, radio resource management, and optimization for wireless networks
Abstract: The cognitive processing mechanism of humor refers to how the system of neural circuitry and pathways in the brain deals with the incongruity in a humorous manner. The past research has revealed different stages and corresponding functional brain activities involved in humor-processing in terms of time and space dimensions, highlighting the effects of the time windows of about 400 ms, 600 ms, and 900 ms. However, much less is known about humor processing in light of the frequency dimension. A total of 36 Chinese participants were recruited in this experiment, with Chinese jokes, nonjokes, and nonsensical sentences used as the stimuli. The experimental results showed that there were significant differences among conditions in the P200 effect, which signified that the incongruity detection had already been integrated and perceived at about 200 ms, prior to the semantic integration at about 400 ms. This pre-processing is specific to Chinese verbal jokes due to the simultaneous involvement of both orthographic and phonologic parts in processing Chinese characters. The analysis on the frequency dimension indicated that beta’s power particularly reflected the characteristics of different stages in Chinese verbal humor processing. Jokes’ and nonsensical sentences’ relative power changes on the beta band ranked significantly higher than that of nonjokes at about 200 ms, which suggested the existence of more difficulties in meaning construction in pre-processing the incongruities. This indicated a continuity between the analysis of event related potential (ERP) components and neural oscillations and revealed the key role of the beta frequency band in Chinese verbal joke processing.
Key words: Beta band    humor processing    P200 effect   
1. Introduction

Humor is ubiquitous, happening in all individuals in all stages over a lifespan. It plays a very critical role in various social contexts, exerting great influences on societal and cultural development. Humor, rooted in its social component, presented in its cognitive component, passes laughter effects by its affective component. Different components embedded in humor decide the complexity of humor, causing the development of a great number of relevant theories. The incongruity theory[1], one of the cognitive theories of humor, highlighting the incongruity detection and resolution, has been the most frequently quoted theory in this field due to its application to the research on the cognitive processing mechanism of humor.

The cognitive processing mechanism of humor refers to how the system of neural circuitry and pathways in the brain deals with the incongruities in a humorous manner, which involves a complicated high-level cognitive process, rendering it worthwhile to be investigated. Advanced brain-imaging techniques have provided researchers with more possibilities to reveal more objective evidence on the mechanism of humor processing. Functional magnetic resonance imaging (fMRI) is employed to show specific brain regions activated during different stages of humor processing due to its excellent space resolution. The cognitive stage (including the incongruity detection and resolution) involves the bilateral activation in the brain, with inferior frontal gyrus, superior frontal gyrus, and middle temporal gyrus showing a stronger activation in verbal humor[2],[3]; the affective stage (mirth) is closely related with mesolimbic reward regions, with the highlight of amygdala, hippocampus, and insular cortex[3]-[5]. The event related potential (ERP) based technique, another frequently-used brain-imaging method, with its excellent temporal resolution, is used to determine different stages in humor processing. Different ERP components corresponding to different stages were extracted in a number of relevant studies. The N400 component has been found to be correlated with the incongruity detection[6]-[10]; the P600 component is related to the incongruity resolution[7],[8],[10],[11], and the component of late positive potentials (LPPs) is indicated to be connected with the mirth[7],[8],[10].

The P200 component, peaking between 150 ms and 275 ms, triggered by the visual stimuli, is larger for the expected endings than for the unexpected endings[12]. In language processing, P200 could be modulated by the contextual information, such as sentence-level constraints or congruity associated with the target word[13]. It is part of the cognitive matching system that compares sensory input with the stored memory[14]. All these previous findings about P200 present its great correlations with humor processing, because in essence, humor processing is just to find the way out to the unexpected endings, the incongruities or unmatching system. However, by far, very limited evidence has indicated the roles of P200 in humor processing research. Furthermore, a number of studies indicate there are significant differences in processing mechanisms between the alphabetic and pictograph languages. For example, the component of P200 was suggested to be closely related with the processing of Chinese characters instead of the processing of the alphabetic language[15], due to its inclusion of both orthographic and phonologic parts at the same time. Therefore, the P200 effect is worth further investigation in verbal humor-processing studies for its specific roles in both words and perceptions.

Neural oscillations can also be reflected on the frequency dimension apart from the time and space dimensions. Some frequency bands, such as alpha, delta, theta, beta, and gamma, are often detected and studied in some specific brain regions in cognitive processing. Though it is currently popular in language processing studies due to its meaningful applications to artificial intelligence and clinical studies, the frequency dimension analysis has been rarely used to investigate the verbal humor processing.

2. Method and Materials 2.1 Participants

A total of 42 right-handed adults (21 male and 21 female) with normal, or corrected-to-normal vision, aged from 19 years to 28 years (mean age: 23.75 years), from Dalian University of Technology and Liaoning Normal University, were recruited as paid volunteers to take part in the current experiment. No participant had neurological or psychiatric diseases and all of them were native Chinese speakers. Their average schooling years ranged from 14 years to 18 years (mean schooling: 16.30 years). All participants had completed the written informed consent, agreeing with following the directions during the experiment, and had been informed of the instructions of all procedures prior to the experiment. The current experiment was approved by the Research Ethics Committee of Liaoning Normal University. The participants were asked to minimize their movements and eye-blinks during the experiment. In the phase of data pre-processing, 6 participants were removed due to the invalidity of their data, leaving 36 participants’ data being analyzed.

Since each punchline matched up with three set-up conditions to avoid the reviewing effects caused by repeatedly reading the same punchline, each participant can only read one punchline for one time during the experiment. Thus, 36 participants were randomly grouped into 12 groups, with three participants in one group finishing one set of the stimuli, and each group was regarded as one subject.

2.2 Materials

Prior to the experiment, 90 question-answer type Chinese jokes (homophonic jokes) were selected from the Internet and magazines. After the pretest among 150 people (different from the participants in the experiment), top 60 jokes ranked as funny were subsequently sorted out, together with 60 nonjokes (normal statements of facts from the Internet and newspapers), and 60 produced nonsensical sentences (totally irrelevant questions and answers), were used as the stimuli in the experiment, 180 experimental sentences in total. The setup sentences instead of punchlines of jokes were controlled so that the neural activities were not triggered by different punchlines[10]. Every set of stimuli consisted of three different conditions: Jokes, nonjokes, and nonsensical sentences, sharing the same punchline. All stimulus sets were randomly divided into three blocks and each participant can only see one block for one stimulus type, so the same punchline would not be seen twice in the experiment. Nonjokes were all related to semantic memories from daily life and nonsensical sentences were all kept at a very low semantic level. Each setup sentence was limited to about 15 Chinese characters and each punchline was limited to 2 to 4 Chinese characters. Fig. 1 shows one of the examples of a set of stimuli, which consists of three conditions: Joke, nonjoke, and nonsensical sentence. The setup for the joke is that the sheep stops breathing, and guess an idiom. The answer or the punchline is “扬眉吐气”. Because “扬 (raise)” has the same pronunciation with the Chinese character “羊 (sheep)” and “眉 (eyebrow)” has the same pronunciation with the Chinese character “没 (stop)”. The puchline to the setup for the joke is “羊没吐气 (the sheep stops breathing)”, which carries the same pronunciation with the idiom “扬眉吐气 (raise eyebrows and feel elated after unburdening oneself of resentment)”. For the nonjoke, the setup is that raise eyebrows and feel elated after unburdening oneself of resentment, and guess an idiom. The answer or the punchline of “扬眉吐气” is just the correct answer to the setup question, which is a normal statement. The setup of the nonsensical sentence has no semantic relations with the punchline “扬眉吐气”. More examples see the appendix.

Fig. 1 Example of a set of stimuli.
2.3 Procedure

E-Prime 2.0 was used to design the procedure in this experiment. Participants were instructed about the experimental procedures prior to the experiment. Each of them was seated in a quiet room and a screen was placed approximately at a 100 cm distance. Three trials, similar to the formal experiment trials, were firstly presented to the participants to help them become familiar with the procedure. The flow of the stimulus presentation in each trial went as follows: Firstly, the participant saw a fixation point (+) (the duration was 1000 ms) in the center of the screen; then the setup of the joke/nonjoke/nonsensical sentence, in the question type, appeared; after the setup, the participant pressed any button to go on with the trial by seeing another fixation (the duration was also 1000 ms) and then the punchline, in the answer type, appeared in the center of the screen; after 3000 ms, the participant was required to make three ratings of R1, R2, and R3, respectively, on the degrees of surprise (not surprised at all/not surprised/surprised/very surprised), comprehensibility (not comprehensible at all/not comprehensible/comprehensible/very comprehensible), and funniness (not funny at all/not funny/funny/very funny) on a four-point Likert scale.

2.4 Data Acquisition

Electroencephalograph (EEG) was used to collect the data of brain electrical activities in this experiment for its high temporal resolution over other brain imaging methods to better illustrate the mechanism of humor processing from the perspectives of different dimensions. During the experiment, brain electrical activities were recorded from 64 scalp sites by electrodes fixed in the electrode cap with the references on the left and right mastoids, using the Active Two system (BioSemi, the Netherlands). All interelectrodes impedance was maintained below 5 kΩ. ERP waveforms were time-locked to the onset of the punchline. Trials, contaminated with artifacts, such as the excessive vertical or horizontal electro-oculographic potentials, excessive muscle activity, bursts of electromyographic activity, or peak-to-peak deflection exceeding ±100 mV, were excluded from the averaging. The averaged ERP epoch was 2000 ms, including a 200 ms pre-solution baseline. For the first-time analysis, all epochs were band-pass filtered in the range of 0.1 Hz to 50.0 Hz using digital zero-phrase shift filtering. After artifact correction, 60 valid epochs per condition were obtained for every participant.

3. Experimental Results 3.1 Behavioral Results

Behavioral data were analyzed on the basis of 36 participants instead of 12 subjects to guarantee more accurate behavioral results. Among the ratings, jokes’ funniness scores were the highest compared with nonjokes’ and nonsensical sentences’; nonjokes’ comprehensibility ratings were higher than jokes’, and jokes’ comprehensibility ratings were higher than nonsensical sentences’; nonsensical sentences’ surprise ratings were rated the highest followed by jokes’ surprise ratings; nonjoke’s surprise ratings were rated the lowest (see Fig. 2). The paired samples test showed that there were significant differences in surprise ratings between jokes and nonjokes (Mean=1.001, Standard error mean=0.0695, t=14.401, and p=0.000), between nonjokes and nonsensical sentences (Mean=–1.557, Standard error mean=0.086, t=–18.025, and p=0.000), and between jokes and nonsensical sentences (Mean=–0.556, Standard error mean=0.602, t=–9.236, and p=0.000); there are significant differences in comprehensibility ratings between jokes and nonjokes (Mean=–0.308, Standard error mean=0.524, t=–5.882, and p=0.000), between jokes and nonsensical sentences (Mean=1.343, Standard error mean=0.081, t=16.557, and p=0.000), and between nonjokes and nonsensical sentences (Mean=1.651, Standard error mean=0.082, t=20.068, and p=0.000); there are significant differences in funniness ratings between jokes and nonjokes (Mean=0.968, Standard error mean=0.074, t=13.062, and p=0.000), between nonjokes and nonsensical sentences (Mean=–0.506, Standard error mean=0.935, t=–5.410, and p=0.000), and between jokes and nonsensical sentences (Mean=0.463, Standard error mean=0.104,t=4.451, and p=0.000).

Fig. 2 Comparison among ratings of jokes, nonjokes, and nonsensical sentences.
3.2 ERP Results

EEG data were processed and analyzed offline using MATLAB 2013. The component amplitudes were analyzed with one-way ANOVAs using the factors task conditions (jokes, nonjokes, and nonsensical sentences) and electrode sites (Cz, Fz, and Pz). And the 0.05 level of significance was adopted throughout all ERP analyses. All stimuli elicited typical ERP components of visual words, such as P1 and N1, which were elicited during reading, probably reflecting the visual feature extraction necessary to the processing of visual information in the memory[16]. By using the wavelet analysis, the effects were significant on the time windows of 180 ms to 240 ms, 320 ms to 450 ms, and 600 ms to 900 ms. By using the conventional analysis, the effects were significant on the time window of 900 ms to 1500 ms.

3.2.1 180 ms to 240 ms

The statistical analysis of the 180 ms to 240 ms time window showed there were main effects between stimulus conditions and electrode sites Fz and Cz (F(1, 11)=4.37, p=0.0252, and η2=1.5432). Nonjokes elicited more positive waveforms than nonsensical sentences and jokes. Moreover, the observed brainwaves stimulated at Fz and Cz for the main effect of midline site were larger than that at Pz in this time window (see Fig. 3). Maximal deflections in positivity were mainly located around the frontal and central regions of the scalp for all three types of stimuli, but for jokes, the Broca’s area was activated more than that for nonjokes and nonsensical sentences (see Fig. 4).

Fig. 3 Brain waveforms on the electrodes (Cz, Pz, and Fz) on the time windows of 180 ms to 240 ms, 320 ms to 450 ms, and 600 ms to 900 ms: (a) wavelet-Fz, (b) wavelet-Cz, and (c) wavelet-Pz.

Fig. 4 Topographic maps of different stimuli (jokes, nonjokes, and nonsensical sentences) on the different time windows: (a) 180 ms to 240 ms, (b) 320 ms to 450 ms, and (c) 600 ms to 900 ms.
3.2.2 320 ms to 450 ms

The statistical analysis of the 320 ms to 450 ms time window showed there were main effects between stimulus conditions and electrode sites Cz (F(1, 11)=3.68, p=0.0417, and η2=2.005) and Pz (F(1, 11)=5.04, p=0.0157, and η2=0.823). Both jokes and nonsensical sentences elicited more negative waveforms than nonjokes. Moreover, the observed brainwaves at Fz and Cz on the main effect of midline site were larger than that at that at Pz in this time window (see Fig. 3). Maximal deflections were mainly located around the frontal and occipital regions of the scalp for all three types of stimuli, but for both jokes and nonsensical sentences, there were more activations in negativity in the temporal pole and right frontal lobe than that in the occipital region (see Fig. 4).

3.2.3 600 ms to 900 ms

The statistical analysis of the 600 ms to 900 ms time window showed there were main effects of stimulus conditions and electrode sites Fz (F(1, 11)=3.97, p=0.0337, and η2=1.65) and Cz (F(1, 11)=4.07, p=0.0313, and η2=0.952). Nonjokes elicited more positive waveforms than jokes and nonsensical sentences. What is more, the observed brainwaves elicited at Fz and Cz on the main effect of midline site were larger than that at Pz (see Fig. 3). For both jokes and nonsensical sentences, maximal deflections in negativity were mainly located around the right frontal lobe, or more specifically, close to the middle frontal gyrus for both jokes and nonsensical sentences, but for jokes, the Broca’s area was distinctively activated (see Fig. 4).

3.2.4 900 ms to 1500 ms

The statistical analysis of the 900 ms to 1500 ms time window showed there were main effects of stimulus conditions and electrode sites Fz (F(1, 11)=4.07, p=0.0313, and η2=3.06144). Jokes elicited more positive waveforms than nonsensical sentences and nonjokes. What is more, the observed LPPs stimulated at Fz and Cz for the main effect of midline site were larger than that at Pz (see Fig. 5). Maximal deflections were mainly located around the frontal and parietal regions of the scalp for all three types of stimuli, but for jokes, right hemisphere was activated more than left hemisphere.

Fig. 5 Brain waveforms on the electrodes (Cz, Pz, and Fz) on the time window of 900 ms to 1500 ms: (a) conventional-Fz, (b) conventional-Cz, and (c) conventional-Pz.
3.3 Neural Oscillations Results

Fieldtrip toolbox[17] is the main analysis tool used in this study and high-pass filter is used to eliminate the slow drifts. For a proper analysis of oscillatory dynamics, different analytic tools had been used in this experiment, including the wavelet based time-frequency analysis (for quantifying amplitude changes) and event related coherence analysis (for quantifying changes in phase coherence between electrodes), being performed in the domain of the language comprehension. For each trial, the time horizon was determined from –4000 ms to 6000 ms to get rid of the border artifacts in the power spectrum[18]. The data were analyzed in a 10 ms time step from –500 ms to 1000 ms, and in a 1 Hz step from 4 Hz to 100 Hz, using Morlet wavelets of seven cycles each[19]. Power in the stimulus interval was transformed to the percentage of change relative to the baseline. Four time windows (about 200 ms, 400 ms, 600 ms, and 800 ms) close to ERP components analyzed in the previous part were examined to explore whether the oscillations in different conditions of stimuli in different time windows were qualitatively different and whether there was a continuity between the analysis of ERP components and neural oscillations. All dependent variables were analyzed by multivariate ANOVAs. Effects with a significance difference of p<0.05 were reported.

At about 200 ms, jokes’ beta power ranked higher than nonsensical sentences’ and nonjokes’, with differences being significant among three stimuli (p<0.05). All three stimuli’s power decreased to their lowest points at about 400 ms, with jokes’ beta power being the highest and nonsensical sentences’ being the lowest, with differences being significant among three stimuli (p<0.05). From the view of the electrodes activated by joke-nonjoke in the beta range, the beta band was significantly activated (p<0.05) in the electrodes of C4, CP4, P4, PO4, and P7 at about 200 ms; FT7 and C4 at about 400 ms; P3 at about 600 ms; FC1, FZ, FT8, T8, and P8 at about 800 ms (seeFig. 6 (a)). From 400 ms, all three stimuli’s beta power increased, with both jokes’ and nonjokes’ being higher than nonsensical sentences. At about 600 ms, jokes’ and nonjokes’ beta power were at almost equally height, being higher than nonsensical sentences. Then, all three stimuli’s beta power continued to increase and similar situations were kept up to about 800 ms (see Fig. 6 (b)).

Fig. 6 Beta’s electrode area of joke-nonjoke and its oscillations in the different time windows: (a) electrodes with significant differences of joke-nonjoke in the beta band and (b) brain oscillations of the beta band at the time windows of about 200 ms, 400 ms, 600 ms, and 800 ms.
4. Discussion 4.1 Behavioral Results

The results of different ratings indicated that the stimuli selected in this study highlight the features of jokes, nonjokes, and nonsensical sentences, respectively. Jokes had the highest ratings in funniness due to their affective stage, mirth, which was aroused after the incongruity resolution. Though the incongruity resolution was not achieved in nonsensical sentences, the absurd feelings caused by low semantic meanings between the setups and the punchlines would also arouse slight emotional changes. That was why nonsensical sentences’ funniness ratings were higher than nonjokes’ and lower than jokes’. The setups and punchlines of nonjokes were the materials from common sense, greatly related to semantic memories, so their comprehensibility ratings were the highest. In comparison, jokes’ comprehensibility ratings were relatively lower than nonjokes’, because jokes’ comprehension needs the conversion of fixed mindset and not all participants can resolve the incongruities. Nonsensical sentences’ comprehensibility ranked the lowest due to their unresolved incongruities between the setups and the punchlines, and that was why it ranked the highest in surprise ratings. Then the paired comparisons were also made between two conditions in each set of the stimuli to further test the validity of the material, and the results showed the significant differences in every pair, providing later neural activity analysis with a solid premise.

4.2 P200 Effect

The visual P200 is a positive going electrical potential of the ERPs measured at the human scalp which peaks at about 200 ms varying between about 150 ms to 275 ms induced by visual input stimuli[15]. It is relevant to cognitive processes[20], such as the working memory[21], semantic processing[22], and cognitive matching system, showing significant task-related amplitude differences between congruent and incongruent targets[14] and reflecting the match of visual input with expectations[23]. In this experiment, all three stimulus-conditions elicited apparent amplitudes at about 200 ms, which was caused by the P200 effect for their visual words processing. However, the degrees of the elicitation among three conditions in electrodes of Fz and Cz were significantly different (see Fig. 4). Nonjokes stimulated the most positive P200, followed by nonsensical sentences and jokes. This result was consistent with previous research: The P200 amplitude was larger for the expected endings than that for the unexpected ones in the sentence reading task[24]; a reduced P200 amplitude to incongruent trials relative to congruent trials suggested a reduced information gain from incongruent stimuli[25]. In this experiment, nonjokes’ setups and punchlines were normal questions and answers, congruent with each other and in line with readers’ expectations, so the P200 amplitudes of nonjokes were larger, and nonsensical sentences and jokes elicited relatively reduced P200 amplitudes for their incongruence between the setups and punchlines. Besides, at 200 ms, the incongruity in jokes and nonsensical sentences had not been resolved, leading to a reduced information gain from these two conditions compared with nonjokes, so the amplitudes of nonsensical sentences and jokes in P200 were smaller than that of nonjokes.

Furthermore, Chinese characters are ideographs or pictographs with very distinctive hieroglyphic features, with both orthographic and phonologic parts in a Chinese character. A number of Chinese language researchers found that P200 was an indicator of early processing of Chinese characters and it was associated with both orthographical processing and phonological processing, simultaneously[26],[27]. The radical interference effect on P200 components suggested that phonological processing does occur at a sub-lexical stage[28]. By contrast, in research on the lexical processing of the alphabetic language, P200 was not confirmed to feature the definite relation between orthographic processing and phonologic processing, decided by the alphabetic writing system in western languages. The obvious P200 effect in this experiment indicated the simultaneous processing of both orthographic and phonologic parts of Chinese characters. More importantly, the significant differences in the P200 effect among different conditions signified that the incongruity detection in jokes and nonsensical sentences was already integrated and perceived at about 200 ms after the stimulus onset. In other words, prior to the semantic integration at about 400 ms (N400 component), there was specific semantic integration pre-processing in comprehending Chinese verbal jokes. In terms of topography, most of the frontal areas were activated in joke processing, with Broca’s area being activated more than nonjokes and nonsensical sentences, because the stimuli used in this experiment were homophonic jokes, one type of phonological jokes, which involved the activation of a left hemisphere network focusing on the speech production regions[29].

4.3 N400 Effect, Protracted P600-Like Positivity, and LPPs

The N400 effect functionally reflects semantic integration, linguistic/non-linguistic context and conceptual binding with the long-term memory in an active comprehension process[30]. Earlier in the 1980s, N400 was reported by Kutas and Hillyard[31] when participants responded to semantically anomalous words in a sentence context, wherein more predictable words elicited smaller N400s than less predictable words. As to the processing mechanism of verbal jokes, Coulson and Kutas[32] found that N400 was more negative for jokes than that for nonjokes in high-constraint sentences. Du et al.[7] reported that funny items elicited a more negative ERP deflection (N350 to N400) than unfunny items did, reflecting the registration of surprise in joke comprehension, and similar results can also be found in Feng et al.’s[8] research. In this experiment, compared with nonjokes, during 320 ms to 450 ms, jokes and nonsensical sentences elicited more extensive negative neural activities (N400), as a result of the incongruity and unpredictability between their setups and punchlines. Besides, the more negative deflections of jokes and nonsensical sentences in the N400 component also reflected more brain responses for the incongruity detection, indicating more efforts made in the semantic integration and more conflicts with semantic expectations.

The P600 effect, in language processing, was associated with syntactic anomalies[22], but it was also demonstrated to reveal a combination of syntactic and semantic processing[33]. Coulson and Kutas[32] indicated that, in joke processing, both good and poor jokes comprehenders showed sustained greater negativity in 500 ms to 900 ms for joke than non-joke endings, indexing the frame-shifting between the setups and punchlines needed to re-establish the coherence. This frame-shifting reflected the operations of a semantic re-analysis process that reorganized existing information into a new frame or schema retrieved from the long-term memory[6]. In this experiment, both jokes and nonsensical sentences evoked greater negativity (in about 600 ms to 800 ms) than nonjokes, with nonjokes’ eliciting positive deflections at about 700 ms and jokes/nonsensical sentences’ eliciting slow-rising positivity at about 900 ms, reflecting greater attention resources being employed in semantic re-analysis in frame-shifting between the setups and punchlines in jokes and nonsensical sentences. The slow-rising late positive potentials were probably the protracted P600-like positivity caused by the combination of syntactic and semantic processing in the P600 effect.

Additionally, the difficulty in understanding jokes was associated with the right hemisphere damage[34], especially damages to the right frontal lobe[35]. In this experiment, in terms of topography in 600 ms to 900 ms, for both jokes and nonsensical sentences, maximal deflections in negativity were mainly located around the right frontal lobe, mainly around the middle frontal gyrus, which indicated the involvement of frame-shifting. But for jokes, Broca’s area was distinctively activated, reflecting the resolution between the setups and punchlines in phonologic jokes[29]. By contrast, Broca’s area was not activated in nonsensical sentences, suggesting the irresolution between the setups and punchlines.

LPP is a scalp-recorded ERP concerning facilitated attention to emotional stimuli, which is smaller following neutral interpretations compared with emotional interpretations[36]. LPP was reduced when participants made non-affective compared with affective judgments about emotional stimuli[37]. In verbal humor processing, funny stimuli, compared with unfunny ones, elicited more positive ERPs (1250 ms to 1400 ms), related to the affective appreciation stage in joke processing[7]; jokes elicited the most positive slow-wave activities between 900 ms to 1500 ms, suggesting the emotional processing[8] involved in it. In this experiment, jokes elicited more positive waves in 900 ms to 1500 ms than nonsensical sentences because of jokes’ emotional arousal or mirth. Though nonsensical sentences did not lead to mirth, they may cause emotions like ridicule, due to their totally irrelevant semantic relations between the setups and punchlines. For nonjokes, the normal statements of facts, without emotion arousal, activated relatively smaller waves in this time window.

4.4 Function of Beta Oscillations in Processing Chinese Verbal Jokes

In the recent humor-related processing mechanism research, most studies focused on the neural activities relevant to time courses and brain regions, but the investigations of event-related oscillatory activities in frequency bands did not receive enough attention. Previous studies found that frequencies in the beta range (13 Hz to 30 Hz) turned out to be particularly important with respect to cognitive and linguistic manipulations during language processing[38]. Some studies showed sentence-level syntactic binding also stimulated neuronal synchronization in the beta frequency band[35]. The beta activity involved higher-order linguistic functions, such as the discrimination of word categories and the retrieval of action semantics as well as the semantic memory, and syntactic binding processes, in line with the meaning construction in sentence processing[38]. In this experiment, at 200 ms, the beta’s power evoked by three conditions showed significant differences, with jokes’ beta power ranking higher than nonsensical sentences’, and nonsensical sentences’ ranking higher than nonjokes’, which reflected that the incongruity detection was already integrated at about 200 ms and evoked the activities in the beta band. This result supported the existence of the preprocess of the incongruity detection in retrieving the semantic memory at about 200 ms from the angle of oscillatory activities in processing Chinese verbal humor. Higher power of the beta band of jokes and nonsensical sentences implied more requirements for updating the system to detect the unexpected stimuli.

From 200 ms, beta power for all three conditions of stimuli declined and reached the lowest point at 400 ms, with jokes’ and nonsensical sentences’ beta power decreasing more sharply than nonjokes’. According to the previous study, the beta power decrease predicted the probability of new processing demands, or to be specific, beta power would decrease if the current cognitive state was interrupted by novel and/or unexpected stimuli[39]. For jokes and nonsensical sentences, their setups and punchlines were unexpected and novel compared with nonjokes’. Therefore, the degree of beta power decreased more for jokes (Power decrease=1.02–0.92=0.10) and nonsensical sentences (Power decrease=0.98–0.87=0.11) than that for nonjokes (Power decrease=0.94–0.89=0.05), which indicated the activities of the beta band were closely associated with the incongruity detection in processing Chinese verbal jokes.

At the time point of 400 ms, there was an obvious turning point for all three stimulus-conditions, with beta power going up. Jokes’ and nonjokes’ beta power kept going up and converged to a similar extent at about 600 ms, but nonsensical sentences’ beta power ranked much lower, though it also kept increasing, which implied the involvement of the beta band in the process of syntactic binding and the more successful incongruity resolution occurring at about 600 ms to jokes and nonjokes than that to nonsensical sentences. This result supported the previous analysis on the incongruity resolution concerning the P600 effect.

To investigate the difference between the joke and nonjoke in the beta band distributed in space, the activated electrodes of joke-nonjoke were analyzed. At the time window of about 200 ms, the electrodes activated of joke-nonjoke (C4, CP4, P4, PO4, and P7) corresponded to the nearest brain regions of BA4 (primary motor cortex), BA40 (Wernicke’s area), BA39 (the angular gyrus), BA19 (associative visual cortex), and BA37 (fusiform gyrus). In sequence, these brain regions were closely correlated to the functions of action word processing, written words comprehension, visual information processing, and word cognition, which suggested that at 200 ms, beta oscillations already played an important part in the processing of Chinese verbal jokes, reflecting the pre-processing of the incongruity detection.

At the time window of 400 ms, the electrodes activated in the beta band by joke-nonjoke (FT7 and C4) were the nearest to BA44/45 (Broca’s area) and BA4 (primary motor cortex). The activation in the beta band related to Broca’s area in the beta band suggested the key role that the beta band played in the incongruity detection in processing Chinese verbal jokes, due to the critical functions of Broca’s area in joke-processing. The activation in the beta band was the nearest to primary motor cortex because the decrease of beta power was stronger for verb processing around 400 ms to 600 ms[34], occurring in the premotor cortex even without having an explicit motor task, revealing the retrieval and integration of action semantics for a certain word[40].

At 600 ms, the electrode activated by the beta band of joke-nonjoke was P3, corresponding to BA39 (the angular gyrus), which was related to both the mediation of the memory retrieval and the contradiction between what was expected from the retrieval and what was unusual[41]. The difference of joke-nonjoke reflected the contradiction between the setups and punchlines, so the activation of the beta band in this area at 600 ms indicated the close correlation between the beta band activity and the incongruity resolution in processing Chinese verbal jokes.

At 800 ms, the electrodes activated by the beta band in the regions of difference between jokes and nonjokes were Fz, FC1, FT8, T8, and P8, corresponding respectively to BA8 (frontal eye fields), BA6 (premotor cortex), BA11 (superior frontal gyrus), BA21 (middle temporal gyrus), and BA37 (fusiform gyrus). Among these activated brain regions, the superior frontal gyrus was identified to produce laughter consistently accompanied by a sensation of merriment or mirth[42], which was a significant brain region involved to reveal the affective stage of humor processing. In addition, the activations of middle temporal gyrus and fusiform gyrus were also related to emotion processing due to their adjacencies to the limbic lobe. Thus, we may conclude that beta oscillations may contribute to the mirth evoking in processing Chinese verbal jokes.

5. Conclusions

A great number of studies have highlighted the features of verbal humor processing at the time windows of about 400 ms, 600 ms, and 800 ms to illustrate the corresponding stage division: Incongruity detection, resolution, and mirth, but there were very few studies to investigate the features of P200. Previous studies revealed that the P200 component was closely related with the incongruity or the unmatching system in sentence understanding, so this experiment put emphasis on analyzing the time window of about 200 ms to try to reveal the correlations between the P200 effect and Chinese verbal humor processing. The results showed that three different conditions (jokes, nonjokes, and nonsensical sentences) in this experiment started to present significant differences from this time point, which signified that the incongruity detection in jokes and nonsensical sentences was already integrated and perceived at about 200 ms after the stimulus onset. In other words, prior to the N400 effect, there was already a specific semantic integration pre-processing in comprehending Chinese verbal jokes. The oscillation analysis results, the beta band in particular, also supported this finding. The significant differences among three different stimuli in the beta band started to occur at about 200 ms, and at 400 ms, like ERP components results, all three stimuli activated their own peak values with significant differences, which indicated the critical functional role of the beta band in the incongruity detection in humor processing. In addition, at 600 ms, the power of the beta band of jokes and nonjokes increased to similar heights, much higher than that of nonsensical sentences, implying the more incongruity resolution of both jokes and nonjokes than that of nonsensical sentences. Besides, the brain regions of differences between jokes and nonjokes also revealed the key role of the beta band in different stages in processing Chinese verbal jokes.

Although this experiment validated the processing model of Chinese verbal jokes from the aspects of both ERP components and oscillation activities, several limitations should be considered for further improvement in the future studies. All jokes were screened by another group of participants different from the subjects in the current experiment beforehand, but we cannot exclude the possibility of individual differences in processing jokes. For the material selection, it was hard to completely eliminate the semantic association for nonsensical sentences, but we tried to keep it low. The future research should include different stimuli in different languages to differentiate the characteristics of humor-processing mechanisms between Chinese characters and alphabetic words. Furthermore, other types of verbal jokes, besides phonologic jokes, should also be examined to further explore the processing mechanism of Chinese verbal jokes. Additionally, the degree of plausibility towards jokes could also affect the brain activities in the current experiment. Meanwhile, different life phases should also be included in the variables in the future study to elaborate on the developmental characteristics in humor-processing mechanisms.

Appendix

More examples of experimental materials are shown as Table A.1: Each set of stimuli consists of three conditions: Joke, nonjoke, and nonsensical sentence and all these three conditions match a same punchline.

References
[1] W. H. Martineau, J. H. Goldstein, and P. E. McGhee, The Psychology of Humor: Theoretical Perspectives and Empirical Issues, New York: Academic Press, 1972.
[2] Y.-C. Chan, T.-L. Chou, H.-C. Chen,et al., " Segregating the comprehension and elaboration processing of verbal jokes: An fMRI study,” Neuroimage, vol. 61, pp. 899-906, Jul. 2012.
[3] M. Shibata, Y. Terasawa, and S. Umeda, " Integration of cognitive and affective networks in humor comprehension,” Neuropsychologia, vol. 65, pp. 137-145, Dec. 2014.
[4] D. W. Campbell, M. G. Wallace, M. Modirrousta,et al., " The neural basis of humour comprehension and humour appreciation: The roles of the temporoparietal junction and superior frontal gyrus,” Neuropsychologia, vol. 79, pp. 10-20, Dec. 2015.
[5] Y.-C. Chan, T.-L. Chou, H.-C. Chen,et al., " Towards a neural circuit model of verbal humor processing: An fMRI study of the neural substrates of incongruity detection and resolution,” Neuroimage, vol. 66, pp. 169-176, Feb. 2013.
[6] S. Coulson and R. F. Williams, " Hemispheric asymmetries and joke comprehension,” Neuropsychologia, vol. 43, no. 1, pp. 128-141, 2005.
[7] X. Du, Y. Qin, S. Tu,et al., " Differentiation of stages in joke comprehension: Evidence from an ERP study,” Intl. Journal of Psychology, vol. 48, no. 2, pp. 149-157, 2013.
[8] Y.-J. Feng, Y.-C. Chan, and H.-C. Chen, " Specialization of neural mechanisms underlying the three-stage model in humor processing: An ERP study,” Journal of Neuolinguistics, vol. 32, pp. 59-70, Nov. 2014.
[9] B. Mayerhofer and A. Schacht, " From incoherence to mirth: Neuro-cognitive processing of garden-path jokes,” Frontiers in Psychology, vol. 6, pp. 550:1-19, May 2015.
[10] L.-C. Ku, Y.-J. Feng, Y.-C. Chan,et al., " A re-visit of three-stage humor processing with readers' surprise, comprehension, and funniness ratings: An ERP study,” Journal of Neurolinguistics, vol. 42, pp. 49-62, May 2017.
[11] M. Shibata, Y. Terasawa, T. Osumi,et al., " Time course and localization of brain activity in humor comprehension: An ERP/sLORETA study,” Brain Research, vol. 1657, pp. 215-222, Feb. 2017.
[12] K. D. Federmeier, " Thinking ahead: The role and roots of prediction in language comprehension,” Psychophysiology, vol. 44, no. 4, pp. 491-505, 2007.
[13] S. Coulson and D. Brang, " Sentence context affects the brain response to masked words,” Brain and Language, vol. 113, no. 3, pp. 149-155, 2010.
[14] R. Freunberger, W. Klimesch, M. Doppelmayr,et al., " Visual P2 component is related to theta phase-locking,” Neuroscience Letters, vol. 426, no. 3, pp. 181-186, 2007.
[15] M. Xie, Q. Yang, and Q. Wang, " The P200 component in lexical processing,” Advances in Psychology, vol. 6, no. 2, pp. 114-120, 2016.
[16] M. Kutas, " Views on how the electrical activity that the brain generates reflects the functions of different language structures,” Psychophysiology, vol. 34, no. 4, pp. 383-398, 2007.
[17] R. Oostenveld, P. Fries, E. Maris, et al., " FieldTrip: Open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data,” Computational Intelligence and Neuroscience, 2011, DOI: 10.1155/2011/156869
[18] S. Regel, L. Meyer, and T. C. Gunter, " Distinguishing neurocognitive processes reflected by P600 effects: Evidence from ERPs and neural oscillations,” PloS One, vol. 9, no. 5, pp. e96840:1-11, 2014.
[19] J. P. Lachaux, E. Rodriguez, J. Martinerie,et al., " Measuring phase synchrony in brain signals,” Human Brain Mapping, vol. 8, no. 4, pp. 194-208, 1999.
[20] B. E. McDonough, C. A. Warren, and N. S. Don, " Event-related potentials in a guessing task: The gleam in the eye effect,” Intl. Journal of Neuroscience, vol. 65, no. 1-4, pp. 209-219, 1992.
[21] C. D. Lefebvre, Y. Marchand, G. A. Eskes,et al., " Assessment of working memory abilities using an event-related brain potential (ERP)-compatible digit span backward task,” Clinical Neurophysiology, vol. 116, no. 7, pp. 1665-1680, 2005.
[22] K. D. Federmeier and M. Kutas, " Picture the difference: Electrophysiological investigations of picture processing in the two cerebral hemispheres,” Neuropsychologia, vol. 40, no. 7, pp. 730-747, 2002.
[23] E. W. Wlotko and K. D. Federmeier, " Finding the right word: Hemispheric asymmetries in the use of sentence context information,” Neuropsychologia, vol. 45, no. 13, pp. 3001-3014, 2007.
[24] E. M. Moreno and I. C. Rivera, " Setbacks, pleasant surprises and the simply unexpected: Brainwave responses in a language comprehension task,” Social Cognitive and Affective Neuroscience, vol. 9, no. 7, pp. 991-999, 2013.
[25] A. Zinchenko, P. Kanske, C. Obermeier,et al., " Emotion and goal-directed behavior: ERP evidence on cognitive and emotional conflict,” Social cognitive and Affective Neuroscience, vol. 10, no. 11, pp. 1577-1587, 2015.
[26] A. Barnea and Z. Breznitz, " Phonological and orthographic processing of Hebrew words: Electrophysiological aspects,” The Journal of Genetic Psychology, vol. 159, no. 4, pp. 492-504, 1998.
[27] B. Chen, W. Liu, L. Wang,et al., " The timing of graphic, phonological and semantic activation of high and low frequency Chinese characters: An ERP study,” Progress in Natural Science, vol. 17, no. B07, pp. 62-70, 2007.
[28] L. Zhou, M. C. M. Fong, J. W. Minett,et al., " Pre-lexical phonological processing in reading Chinese characters: An ERP study,” Journal of Neurolinguistics, vol. 30, pp. 14-26, Jul. 2014.
[29] V. Goel and R. J. Dolan, " The functional anatomy of humor: Segregating cognitive and affective components,” Nature Neuroscience, vol. 4, no. 3, pp. 237-238, 2001.
[30] M. Kutas and K. D. Federmeier, " Thirty years and counting: Finding meaning in the N400 component of the event-related brain potential (ERP),” Annual Review of Psychology, vol. 62, no. 1, pp. 621-647, 2011.
[31] M. Kutas and S. A. Hillyard, " Brain potentials during reading reflect word expectancy and semantic association,” Nature, vol. 307, no. 5947, pp. 161-163, 1984.
[32] S. Coulson and M. Kutas, " Getting it: Human event-related brain response to jokes in good and poor comprehenders,” Neuroscience Letters, vol. 316, no. 2, pp. 71-74, 2001.
[33] B. G. Lopez and J. Vaid, " Psycholinguistic approaches to humor,” in The Routledge Handbook of Language and Humor, S. Attardo, Ed. Oxon: Routledge, 2017.
[34] A. M. Bihrle, H. H. Brownell, J. A. Powelson,et al., " Comprehension of humorous and nonhumorous materials by left and right brain-damaged patients,” Brain and Cognition, vol. 5, no. 4, pp. 399-411, 1986.
[35] P. Shammi and D. T. Stuss, " Humour appreciation: A role of the right frontal lobe,” Brain, vol. 122, no. 4, pp. 657-666, 1999.
[36] T. A. Dennis and G. Hajcak, " The late positive potential: A neurophysiological marker for emotion regulation in children,” Journal of Child Psychology and Psychiatry, vol. 50, no. 11, pp. 1373-1383, 2009.
[37] G. Hajcak, J. S. Moser, and R. F. Simons, " Attending to affect: Appraisal strategies modulate the electrocortical response to arousing pictures,” Emotion, vol. 6, no. 3, pp. 517-522, 2006.
[38] S. Weiss and H. M. Mueller, " Too many betas do not spoil the broth: The role of beta brain oscillations in language processing,” Frontiers in Psychology, vol. 3, pp. 201:1-15, Jun. 2012.
[39] L. A. Hald, M. C. M. Bastiaansen, and P. Hagoort, " EEG theta and gamma responses to semantic violations in online sentence processing,” Brain and Language, vol. 96, no. 1, pp. 90-105, 2006.
[40] M. van Elk, H. T. van Schie, R. A. Zwaan,et al., " The functional role of motor activation in language processing: Motor cortical oscillations support lexical-semantic retrieval,” Neuroimage, vol. 50, no. 2, pp. 665-677, 2010.
[41] H. J. Park, J. J. Kim, S. K. Lee,et al., " Corpus callosal connection mapping using cortical gray matter parcellation and DT-MRI,” Human Brain Mapping, vol. 29, no. 5, pp. 503-516, 2006.
[42] I. Fried, C. L. Wilson, K. A. MacDonald, et al., " Electric current stimulates laughter,” Nature, vol. 391, no. 6668, pp. 650:1-2, 1998.