Panel 6 | Multimodal discourse

This presentation is part of Social Semiotics, whose challenge, as discourse analysts, is to pay attention to neglected meanings, and focus on a commitment to social justice (Bezemer & Kress, 2016). This look at the creation of meanings highlights in the foreground the power that is present in different semiotic practices through the promotion or invalidation in Chile of certain signs and creators of meaning in relation to two recent events: the social outbreak of 2019 and the pandemic that began in 2020. In turn, both occurrences have been framed in a deep governmental and communication crisis of the current Chilean Administration. Due to the pandemic, different On-Line artistic activities have proliferated in Chile, as in the case of the Chilean choir Ensamble Manifiesto. This group was born from the demands of the October 2019 Social Outbreak. In the wake of the virus pandemic, they reinvented the format of choral concerts as a new way of expressing their social demands through music giving way to On-Line Encounters, called Music and Resistance. This presentation aims to characterize the critical multimodal discursive construction of meanings around social demands in the previously mentioned On-Line sessions. In accordance with the stated objective, three questions guide the research in which this presentation is framed: 1. What semiotic resources are used by the participants of these On-Line Encounters Music and Resistance in their discursive construction? 2. What evaluative prosodies are formed in these discourses? 3. What strategies and macro-strategies are typical of the participants of the cited artistic event? The analytical approach is carried out from a theoretical-methodological perspective, which is framed in the Social Semiotic, and integrates the contributions of the Appraisal framework and Critical Multimodal Discourse Studies. The methodology is qualitative with a descriptive and interpretive scope. The corpus is constituted by multimodal texts from four meetings, in which musicians from Chile, Peru, Brazil and Mexico participated. The results show: (a) varied semiotic combinations of the different participating musicians, who create multimodal assemblages that include poetic texts, diverse genres (eg fugue); musical styles (eg bossa nova); and images linked to the crossing of competing memories and counter-memories related to human rights; (b) legitimation strategies: defense of indigenous culture, valuation of the territory and different spaces of memory; defense of free education; (c) delegitimization strategies: condemnation of violence in the different participating countries, as well as in countries alluded to in the songs, such as the Czech Republic, Soweto and Vietnam; (d) identification of macro-strategies to transform the status quo and overcome the situation of communicational and governmental crisis of the current Adminstration.

References

Bezemer, J. & Kress, G. (2016). Multimodality, Learning and Communication. A Social Semiotic Frame. New York: Routledge.

Kress, G. & van Leeuwen, T. (2021). Reading images: A grammar of visual design. London: Routledge.

Oteíza, T. & Pinuer, C. (2019). El sistema de valoración como herramienta teórico-metodológica para el estudio social e ideológico del discurso. Logos Revista de Lingüística, Filosofía y Literatura, 29 (2): 207-229.

This presentation introduces some preliminary results from an ongoing research project exploring how text-image relations can be computationally mapped and understood by applying a multimodal framework of analysis (Baldry 2004; Bateman 2014). It does so by using the Hateful Memes Challenge as a case study (Kiela et al. 2021).

In the last decade, a new generation of Artificial Intelligence (AI) models has emerged that accomplishes complex generative and analytical tasks with a sophistication previously achieved only by the human mind. Yet while technological improvements of neural-based machine-learning techniques have led to AI models for the automatic detection, analysis and interpretation of complex digital textualities, research still grapples with many unanswered questions, mostly and crucially regarding the issue of whether and how text-image combinations can be automatically processed, as machines can interpret texts in largely successful ways - but unimodally.

The picture is further complicated in those cases where the verbal contradicts the visual and vice versa, with the intention of generating humorous messages along a spectrum ranging from harmless good fun to the blatantly hateful (Sindoni 2018).

Usually instantiated by a captioned image but with a high degree of variability in form and function, memes are an example of digital textuality, whose success (and virality) derives mostly from non-obvious, ambiguous, and unexpected text-image combinations (Yus 2019) and whose cleverness and jack-in-the-box style surprise require pragmatic skills and cultural knowledge if they are to be correctly interpreted. They are thus exceptionally hard for machines to decode. Trivial as they may seem, memes’ virality affects whole communities and societies in powerful ways, so much so that they may become harmful and dangerous when they disseminate hatred or trigger divisiveness. Automatic detection of harmful contents is all the more vital when hateful or divisive memes go viral, but their multimodal intricacies can scarcely be resolved by means of traditional unimodal classifiers.

With the aim of presenting examples of problematic text-image combinations, we will use the open access dataset produced as part of the Hateful Memes Challenge developed by the Facebook AI research group. This dataset includes over 10,000 multimodal materials that combine text and images in ways specifically designed to trick machines (Kiela et al. 2020).

Our concluding remarks point to future steps in our research and suggest how a multimodal approach to semiosis in digital communication underpinned by pragmatics might well assist the training of machines in detecting problematic image-text combinations in digital textualities and genres other than memes.

References

Baldry, A. P. 2004. “Phase and Transition, Type and Instance: Patterns in Media Texts as Seen Through a Multimodal Concordancer”. In: O’Halloran, Kay L. (ed.). Multimodal Discourse Analysis: Systemic Functional Perspectives. London and New York: Continuum, pp. 83–108.

Bateman, J. A. 2014. Text and Image. A Critical Introduction to the Visual/Verbal Divide. London & New York: Routledge.

Kiela, D., et al. 2021. The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes. https://arxiv.org/abs/2005.04790

Sindoni, M.G. 2018. Direct hate speech vs. Indirect fear speech. A multimodal critical discourse analysis of the Sun’s editorial “1 in 5 Brit Muslims’ sympathy for jihadis”. Lingue e Linguaggi 28: 267-292.

Yus F. (2019) Multimodality in memes: A cyberpragmatic approach. In: Bou-Franch P., Garcés-Conejos Blitvich P. (eds) Analyzing Digital Discourse. Palgrave Macmillan.

Over the last three decades, the search for love, sex, and eroticism has increasingly become mediated by technology, with online dating as one of the most common and fastest growing means for people to meet (Anderson et al., 2020). With the emergence of mobile devices, online dating has become a “relationshopping” (Heino et al., 2010) experience that requires digital literacies, self-branding (Marwick, 2013), impression management (Tomlinson, 2013) and influencer logics (Abidin, 2021) to curate a desirable online self that will stand out in the attention economy. And while digital dating and hookup platforms have been shown to facilitate connections for many, especially within the LGBTQ+ community (Anderson et al., 2020), online dating has unique challenges for those who do not conform to normative notions of gender, sexuality, and emotional bonding. In this paper, I argue that dating apps are discursively constructed as spaces for queer community building as well as hostile spaces where fetishization, harassment, exclusion, and potential violence are regularly negotiated. To explore intersections of discourse, design, and normativity, I employed an ethnographic approach to conducting interviews and observing participants engaged in online dating practices (Mackenzie, 2018; Varis, 2016). To triangulate interview data, I overlaid multimodal critical discourse analysis (MCDA) (Machin et al., 2016; O’Halloran et al., 2019) with a queer linguistics lens (Motschenbacher, 2019) to analyze dating profiles and platform interfaces to examine the interplay between semiotic modes of meaning making about sexual normativities in online dating. This research is situated within the growing body of work that explores online dating through the lens of language (Mortensen, 2015).

References

Abidin, C. (2021). Mapping Internet Celebrity on TikTok: Exploring Attention Economies and Visibility Labours. Cultural Science Journal, 12(1), 77–103. https://doi.org/10.5334/csci.140

Anderson, M., Vogels, E. A., & Turner, E. (2020). The virtues and downsides of online dating. Pew Research Center, 6.

Callis, April Scarlette. 2014. Bisexual, pansexual, queer: Non-binary identities and the sexual borderlands. Sexualities 17(1–2): 63–80.

Heino, R. D., Ellison, N. B., & Gibbs, J. L. (2010). Relationshopping: Investigating the market metaphor in online dating. Journal of Social and Personal Relationships, 27(4), 427–447.

Machin, D., Caldas-Coulthard, C. R., & Milani, T. M. (2016). Doing critical multimodality in research on gender, language and discourse. Gender and Language, 10(3), 301–308. https://doi.org/10.1558/genl.v10i3.32037

Mackenzie, J. (2018). Language, Gender and Parenthood Online: Negotiating Motherhood in Mumsnet Talk. Routledge.

Marwick, A. E. (2013). Status Update: Celebrity, Publicity, and Branding in the Social Media Age (First Edition). Yale University Press.

Mortensen, K. K. (2015). Informed consent in the field of language and sexuality: The case of online dating research. Journal of Language and Sexuality, 4(1), 1–29.

Motschenbacher, H. (2019). Language and sexual normativity. In The Oxford Handbook of Language and Sexuality. Oxford University Press. www.oxfordhandbooks.com

O’Halloran, K., Wignell, P., & Tan, S. (2019). ‘Doing critical discourse studies with multimodality: From metafunctions to materiality’ by Per Ledin and David Machin. Critical Discourse Studies, 16(5), 514–521. https://doi.org/10.1080/17405904.2018.1556173

Tomlinson, E. C. (2013). The Role of Invention in Digital Dating Site Profile Composition. Computers and Composition, 30(2), 115–128. https://doi.org/10.1016/j.compcom.2013.04.003

Varis, P. (2016). Digital ethnography. In A. Georgakopolou & T. Spilioti (Eds.), The Routledge handbook of language and digital communication (pp. 55–68). Routledge.

Until very recently, experts on political science talked about the “Spanish exceptionalism” since Spain, together with Portugal, seemed to be impervious to the wave of far-right populism rising (or consolidating) in many European countries. This political scenario changed drastically when Vox- a party founded in 2013 by some disappointed members of the conservative PP (People’s Party) – achieved a historical victory in the Andalusian elections in 2018. That result was just the beginning of an unprecedented success for any far-right party in the recent history of Spain. In just six years, Vox won 52 seats in the Spanish Parliament.

In times of digital political communication, political parties are highly aware of mainstream and new media’s key role in their electoral success. Social media are excellent channels for parties to disseminate their political programme, recruit new followers and reinforce ideological bonds with their supporters. As such, populism and social media seem to be closely interrelated, and populist politicians have used digital platforms and applications particularly successfully to interact directly with the people through a more personal and informal language (Kreis, 2017).

Vox embodies the main principles of far-right populism (Mudde 2019, Wodak 2015) considered in this paper as a rhetorical style (Laclau 2005, Moffitt 2016) in which the spectacularization of politics and their performance on social media is essential. The party led by Santiago Abascal has a remarkable presence in social media, being the political force with a higher number of followers on Instagram (643,000), the social media application which has continued growing in the last two years (2020-2021), specially attractive for the millennial generation.

This paper analyses the multimodal material (photos, videos, and reels) posted on the Instagram account of “Vox jóvenes” (Vox junior members), the younger organisation of the party, from 2018 until December 2021. The goal of this paper is to give answers to the following questions: what are the main visual narrative axes of “Vox jóvenes”? Do they have a distinctive style or, on the contrary, are just a younger version of Vox’s far-right populism? The theoretical framework used to analyse images in this chapter combined two main models: multimodal critical discourse analysis (Kress and van Leeuwen, 1996, 2001; van Leeuwen, 2021) and visual framing (Rodriguez and Dimitrova, 2011). The corpus study will unveil “Vox jóvenes” mostly echo the party’s style and tropes but also, they display a solid collective identity primarily through their involvement in social activities.

References

IAB Spain & Eulogia. Estudio de redes sociales 2020, https://iabspain.es/estudio/estudio-redes-sociales-2020/ (accessed November 28, 2021)

Kress, G. and van Leeuwen, T. (1996). Reading Images: A Grammar of Visual Design. London: Routledge.

Kress, G. and van Leeuwen, T. (2001). Multimodal Discourse. The modes and media of contemporary communication. London: Bloomsbury.

Kreis, R. (2017): The "Tweet" politics of President Trump, Journal of Language and Politics, 16(4): 607-618.

Laclau, E. (2005). On Populist Reason. New York: Verso.

Moffitt, B. (2016). The Global Rise of Populism. Performance, Political Style, and Representation. Stanford: Stanford University Press.

Mudde, C. (2019). The Far Right Today. Cambridge: Polity Press.

Rodríguez, Lulu & Dimitrova, Daniela V. 2011. “The levels of visual framing”. Journal of Visual Literacy. 30(1): 48-65. doi: 10.1080/23796529.2011.11674684.

Van Leeuwen, T. (2021): The semiotics of movement and mobility, Multimodality & Society 1(1): 97-118.

Wodak, R. (2015). The Politics of Fear. What Right-wing Populist Discourses Means. London: Sage.

This contribution takes seriously the study of neo-Gramscian political discourse theory (PDT) labels such as non-violent counterhegemony and non-morbid symptoms (Acaroglu, 2020). We explore corporate materiality as metapolitical (De Cleen et al., 2021; Zienkowski, 2019), through a focus on the organizational discourse (Marttila, 2019) of Antonino De Masi, an Italian entrepreneur currently living under police protection after denouncing the ‘ndrangheta, Italy’s most dominant global mafia. Mafia territories are characterized by sociomaterial conditions of physical and symbolic intimidation, with the ‘ndrangheta as a criminal economic actor. It is active inside and outside Italy, trough financial extortion, illegal economic activities and competition, e.g. in agriculture, waste management and the construction sector. The ‘ndrangheta distorts entrepreneurial normality, and destabilizes personal and socio-economic networks and ecosystems.

Recently De Masi promoted the project al Sud (de Masi, n.d.), i.e. the transformation of his family-owned electromechanical company into a public company. It counters normalized negative hegemonic views (Krzyżanowski, 2020) of mafia dominance, labeled as deprivation, fatalism or opportunistic management.

• The company develops innovative safe and sustainable products for seismic risk mitigation and automated harvesting mechanics.

• It protects the excluded entrepreneur through a network of committed workers, trade unions, employer organizations and educational resources, against omertà and colluded credit providers.

• It leads the local community towards transregional or international collaborations.

We describe two sociomaterial dimensions (Jerne, 2018):

(1) the network of shareholders and socio-economic actors built around De Masi’s isolation;

(2) transregional product innovation and internationalization as a response to inaccessible local markets

The corpus consists of

(a) a 1h fieldwork interview conducted with De Masi (in 2019),

(b) corporate documentation,

(c) 3 corporate YouTube videos, about new products and launch events.

The results are threefold:

(1) antimafia expertise and discourse in Italian business networks is made explicit;

(2) a techno-economic response to the ‘ndrangheta is uncovered, aside legal repression or civil activism.;

(3) a corporate layer of description is added to traditional PDT genres of politics, policy and media.

References

Acaroglu, O. (2020). Rethinking Marxist Approaches to Transition: A Theory of Temporal Dislocation. Brill. http://brill.com/view/title/58376

De Cleen, B., Goyvaerts, J., Carpentier, N., Glynos, J., & Stavrakakis, Y. (2021). Moving discourse theory forward: A five-track proposal for future research. Journal of Language and Politics, 20(1), 22–46. https://doi.org/10.1075/jlp.20076.dec

de Masi, A. (n.d.). DE MASI AL SUD. La mission di Antonino De Masi. De Masi. Retrieved October 6, 2021, from https://www.demasi.it/demasi-al-sud

Jerne, C. (2018). The syntax of social movements: Jam, boxes and other anti-mafia assemblages. Social Movement Studies, 17(3), 282–298.

Krzyżanowski, M. (2020). Normalization and the discursive construction of “new” norms and “new” normality: Discourse in the paradoxes of populism and neoliberalism. Social Semiotics, 30(4), 431–448. https://doi.org/10.1080/10350330.2020.1766193

Marttila, T. (Ed.). (2019). Discourse, Culture and Organization: Inquiries into Relational Structures of Power. Springer International Publishing. https://doi.org/10.1007/978-3-319-94123-3

Zienkowski, J. (2019). Politics and the political in critical discourse studies: State of the art and a call for an intensified focus on the metapolitical dimension of discursive practice. Critical Discourse Studies, 16(2), 131–148. https://doi.org/10.1080/17405904.2018.1535988

In recent years, the world has witnessed many murderous manifestations of violent attacks by ISIS’s terror agents. While they tried to build authority through territorial advances, one of their most strategic focus points was exploiting cyberspace resources to broadcast their messages (Bloom & Daymon, 2018; Stern & Berger, 2015). Therefore, countering ISIS militarily may halt them for a fleeting moment, but what should be sought is the factors that enabled their longevity and ascent. Thus, to avert the resurgence of ISIS or similar terror organisations which may choose to emulate them in the future, it is crucial to carefully examine their propaganda releases in which lies their ideological underpinnings, hence lingering threats. To this end, linguistics and, in particular, critical (multimodal) discourse analysis provide efficient tools for unravelling the terrorists’ propaganda. However, it has remained largely unexplored within terrorism studies.

This study set out to examine the contents of all fifteen issues of ISIS’s communique, Dabiq, whose circulation can be far reaching because of language and its accessibility to millions of English speakers across the e-world (Gambhir, 2014). It incorporated a Multimodal Critical Discourse Analysis implementing Socio-semantic inventory proposed by Van Leeuwen (1996) on the textual and visual representation of social actors. It involves certain analytical categories from the Social Actor Network, namely Genericization, Specification and Functionalization; and the Visual Social Actor Network, namely Specific or Generic, Individual or Group categorization strategies. The focus is in turn on the ways multimodal resources are utilised by ISIS to shape the perception of unity and diversity among their members and conversely ingrain the theme of sameness among their perceived enemies.

The results showed that the in-group participants are deliberately specified, hence foregrounded, and at the same time, the diversity among the self-members is highlighted as a distinct attribute of their movement. ISIS attempted to exhibit more inclusion without drawing any racial, geographical, cultural, etc., lines among their participants to enhance its status and gain its desired ideological and socio-political purposes. This technique is a propaganda tool against (some parts of) the West with reported racism and discrimination with the hope that certain susceptible individuals would get recruited easier in search for justice and equity. Something that ISIS claims to be the cornerstone of their State.

On the other hand, their enemies’ sameness is textually identified through generic references and addressing them collectively regardless of their potential dissimilarities and visually through categorizing them through religiously, culturally, or nationally distinguishable properties. It acts as a legitimisation tool for their aggression against their perceived enemies by which striking against one enemy justifies striking against the others. The combination of such strategic (non)linguistic framing tasks serve as a propaganda tool for ISIS to create boundaries, arouse prejudice and indoctrinate susceptible individuals with misinformation and ultimately pave the way for mobilising individuals and the call for terrorism.

References

Bloom, M., & Daymon, C. (2018). Assessing the Future Threat: ISIS's Virtual Caliphate. Orbis, 62(3), 372-388.

Gambhir, H. K. (2014). Dabiq: The strategic messaging of the Islamic State. Institute for the Study of War, 15(4).

Stern, J., & Berger, J. M. (2015). ISIS: The State of Terror. London: William Collins.

Van Leeuwen, T. (1996). The representation of social actors. Texts and practices: Readings in critical discourse analysis, 1, 32-70.

This project examines the multimodal means by which characters in DC comics are represented within The DC Comics Encyclopedia. Specifically, I examine how marginalized cultures are depicted and whether or not (or to the extent to which) these depictions render them as other. My analysis is informed by the conception of Othering (Fabian, 1990), visual grammar from Kress and Van Leeuwen’s (2021) theory of modality, and Martin and Rose’s (2003) Appraisal Theory. I selected The DC Comics Encyclopedia because it is a comprehensive guide to characters of the DC universe, which provides illustrations and text descriptions. I focus on three characters: Blue Beetle (Jamie Reyes), Killer Croc, and The Question (Renee Montoya). I chose carriers based on their representation of race, gender, and sexual orientation. Brief attention is given to Blue Beetle (Ted Kord) and The Question as they are contrasting examples of companion characters Jamie Reyes and Renee Montoya. I analyze both the visual representation of each character and the accompanying linguistic descriptions. I argue that the popular media DC Comics, present Othering illustrations that are largely reinforced by the companion linguistic descriptions. In my findings, I present that Jamie Reyes, a Hispanic man, is othered both by his obscuring illustrations and linguistic tags. Moreover, his legacy partner, Ted Kord, heightens the Othering through his non-abstracted presentation and unique appraisal tags. I additionally find that Renee Montoya, a Hispanic lesbian woman, is othered in her obscured appearance and dark color scheme, which are further othered by The Question’s bright depiction. Finally, Killer Croc, a Black man, presents significant instances of Othering in his non-human appearance and linguistic description that renders him as violent, monstrous, and unintelligent. Thus, through identifying how human like features are warped or obscured in the illustrations of marginalized populations, as compared to their nonmarginalized counterparts, I assert that Othering is still in practice within this sample of DC Comics.

References

Dougall, A. & Ridout, C. (2021). The DC Comic Encyclopedia. Penguin Random House.

Fabian, J. (1990). Presence and representation: The other and anthropological writing. Critical Inquiry, 16(4), 753–772. https://doi.org/10.1086/448558

Kress, G., & Van Leeuwen, T. (2021). Reading images: The Grammar of Visual Design. Routledge.

Martin, J.R. & Rose, D. (2003). Appraisal: negotiating attitude, Worship with Discourse (pp.22-65). London: Continuum.

Beginning with Garzone’s (2017:218) notion that “the identity value of food becomes salient in contexts where a certain system or tradition comes into contact with other systems,” this presentation incorporates Critical Discourse Analysis (Fairclough 1992, 1999, inter alia), Multimodal Discourse Analysis (Royce & Bowcher 2007, O’Halloran et al. 2011, inter alia), and a Cultural Approaches to Discourse (Shi-xu 2005, 2009, 2012, 2016, inter alia) framework, in an examination of the cultural competence of information systems related to the issues of food security in the Hispanic communities of Chicago. Specifically, this presentation analyzes websites and apps for the following food-related services: Greater Chicago Food Depository [Banco de Alimentos de Chicago], Pilsen Food Pantry [Despensa de Comida Pilsen], Misfits Market, Too Good to Go, and Top Box Foods.

The food distribution websites equate food with dignity and hope (1), as well as culture (2).

(1) “The Food Depository is part of a united community effort working to bring food, dignity and hope to our Cook County neighbors. We act as the hub for a network of more than 700 food pantries, soup kitchens, shelters and other programs” (https://www.chicagosfoodbank.org/about/, accessed 11/9/2021).

(2) The mission of the Pilsen Food Pantry is, in part, “to address health and social outcomes through the direct distribution of high quality and culturally-appropriate foods” (https://www.figueroawufamilyfoundation.com/pilsen-food-pantry, accessed 11/9/2021).

However, these websites neither explain how food provides dignity nor do they offer examples of what constitutes “culturally-appropriate foods” apart from stating that “culturally-sensitive perishable (fruit, vegetables, eggs, cheese, meat, chicken, fish, etc [sic]), [and] non-perishable foods (grains, beans, peanut butter, canned items)” are available. Moreover, the language of these websites and apps, with the exception of the Greater Chicago Food Depository, is largely English. This presentation thus calls into question the cultural competence of these food security information systems, especially given the fact that Misfit Markets, Too Good to Go, and Top Box Foods offer no specific (socio)linguistic accommodations to food insecure Latinxs on their websites or apps.

Fairclough, Norman. 1992. Discourse and social change. Cambridge: Polity Press.

Fairclough, Norman. 1999. Global capitalism and critical awareness of language. Language Awareness 8:2, 71-83.

Garzone, Giuliana. 2017. Food, culture, language and translation. Journal of Multicultural Discourses 12:3, 214-221.

O’Halloran, Kay L., Sabine Tan, Bradley A. Smith, and Alexey Podlasov. 2011. Multimodal analysis within an interactive software environment: Critical discourse perspectives. Critical Discourse Studies 8:2, 109-125.

Royce, Terry D. and Wendy L. Bowcher, eds. 2007. New directions in the analysis of multimodal discourse. Mahwah, NJ: Lawrence Erlbaum.

Shi-xu. 2005. A cultural approach to discourse. New York: Palgrave Macmillan.

Shi-xu. 2009. Reconstructing Eastern paradigms of discourse studies. Journal of Multicultural Discourses 4 (1): 29-48.

Shi-xu. 2012. A multicultural approach to discourse studies. The Routledge handbook of discourse analysis, ed. by James Paul Gee and Michael Handford. London: Routledge, 642-653.

Shi-xu. 2016. Cultural discourse studies through the Journal of Multicultural Discourses: 10 years on. Journal of Multicultural Discourses 11:1, 1-8.