Modality switching and negation: ERP evidence for modality-specific simulations during negation processing

The Perceptual Symbol System Theory of cognition (Barsalou, 1999) suggests that modality-specific simulations underlie representation of concepts. This is evidenced by processing costs for switching modalities. That is, participants are slower to verify a property in the auditory modality (e.g., BLENDER-loud) after verifying a property in a different modality (e.g., CRANBERRIES-tart) and faster when verifying a property in the same modality (e.g., LEAVES-rustling). The modality switch cost has also been shown to lead to a modulation of the N400 event-related potential (ERP) (Collins, Pecher, Zeelenberg & Coulson, 2011 using a property verification task; Hald, Marshall, Janssen & Garnham, 2011 using a sentence verification task).

In a separate line of research, ERP studies have indicated that without a discourse context, negated sentences are more difficult to process than affirmative sentences, leading to a different N400 pattern for negative sentences than for affirmative sentences (e.g., Fischler, Bloom, Childers, Roucos, & Perry, 1983.) Unlike affirmative sentences, sentences containing negation show a larger N400 for correct, semantically coherent single sentences (i.e., factually true sentences) than for semantically incorrect sentences (i.e., false sentences). However, Nieuwland and Kuperberg (2008,) found that the pragmatics of the sentence can change this N400 pattern to one closer to affirmative sentences.

The goal of the current study was to explore whether the processing of negation could be aided by modality matching information. Other evidence suggests that comprehenders create a simulation of negative sentences (Kaup, Yaxley, Madden, Zwaan, & Lüdtke, 2007), but it is unclear whether modality matching information could modulate the processing cost of negation. Essentially, can modality matching information as reported by Collins et al., (2011) and Hald et al., (2011) lead to a modulation of the N400 for true negated sentences similar to that seen when discourse pragmatics supports negation? Furthermore, we were interested in whether the modality switch effect would lead to a similar pattern in the ERPs as that found with affirmative sentences (Collins et al., 2011 & Hald et al., 2011).

Using a within-subjects design we used 160 pairs of experimental pairs which were either of the same or of a different modality. All experimental items were either visual or tactile modality and were drawn from existing sets of materials (Pecher, et al., 2003; Van Dantzig, et al., 2008). For example, a different modality pair was “A light bulb is very hot” followed by “Rice isn’t black” versus a same visual modality pair “A giraffe is spotted” followed by “Rice isn’t black”. We predicted that the underlined word is where a modulation in the N400 may be seen. Additionally, we explored veracity by making half of the experimental target sentences false (“Rice isn’t white”). Participants were asked to judge whether each sentence was typically true or false.

Our initial results indicate a modulation in the ERP based on both modality and veracity. For true (“Rice isn’t black”) and false target statements (“Rice isn’t white”) different modality pairs elicit a larger frontal-central N400 like effect compared to same modality pairs (Figure 1), replicating Hald et al., (2011). When comparing true versus false, the different modality pairs elicit a large posterior N400 for true compared to false sentences (replicating the effect for negated sentences, i.e., Fischler et al., 1983). However for same modality pairs there is a reduction in the N400 to true statements; no difference is seen between true and false statements in the ERP (Figure 2).

These results replicate and extend previous ERP findings using the modality switch paradigm. The evidence suggests that not only do modality-specific simulations occur but they can even aid the processing of negation.