Multisensory processing, affect and multimodal manipulation: A cognitive-semiotic empirical study of travel documentaries

Multisensory processing represents the mirror image of multimodal meaning-making, in that interpreting multimodal discourse predominantly requires multisensory processing, even when different modes rely on the same sensory channels (Khateb et al., 2002), for example images and text in a book (Gibbons, 2012, p. 40). Remley (2017) makes a similar point when discussing the neuroscience of multimodal persuasive messages, when he asserts that “[t]he term ‘multisensory integration’ is the biological equivalent of the term ‘multimodal’ in rhetoric” (p. 9). An understanding of multisensory processing can therefore be (and presumably is) exploited at the stage of text-production as a resource for manipulative multimodal discourses, with all the ideological consequences that entails.

The concept of manipulation has been a matter of discussion in critical discourse studies (CDS) and pragmatics for more than a decade. Agreement on how to define and analyse the latter has yet to be reached, although most scholars seem to agree that Relevance Theory (Sperber and Wilson, 1995) can provide a useful entry point thanks to its theorisation of variable contexts and individual cognitive environments (de Saussure, 2005; Maillat, 2013; Maillat and Oswald, 2009; Oswald, 2014). Moreover, the concept of epistemic vigilance (Sperber et al., 2010) has been used to investigate the cognitive barriers that need to be bypassed in order for manipulation to work (Hart, 2013; Mazzarella, 2015). Finally, Sorlin (2017: 133) recently highlighted the need to focus not only on the cognitive aspects influencing manipulation, but also on “the psychological aspect of manipulation that often consists in exploiting the target's weaknesses”, thus pointing towards the dimension of affect as a further explanatory force.

This paper begins with an overview of the concepts of manipulation and epistemic vigilance, before discussing insights from the field of multisensory processing in the neurosciences. Then, drawing on some principles from Relevance Theory (Sperber and Wilson, 1995) and looking at some data from travel documentary programmes and their viewers, examples are offered of how manipulation is attempted and achieved through this specific multimodal genre in individual case studies. The focus of the analysis will be on bottom-up (i.e. text-driven) processes and the interpretation/reaction of an audience. The research draws on a novel methodological approach (Castaldi, 2021) that integrates Audience Research (e.g., Schrøder et al., 2003) and Social Semiotics (e.g. Kress and van Leeuwen, 1996, 2001; van Leeuwen, 1999; Machin and Mayr, 2012) in order to analyse media interactions in their individuality.

Results suggest that the affective dimension, predominantly attended to through sonic and visual modes, plays a key role for multimodal manipulation to successfully occur.