A Multimodal Language Faculty: A Cognitive Framework for Human Communication

Language has traditionally been held as an "amodal" system that flows into different forms like speech, writing, or signing; however, communication is multimodal by nature. We pair speech with gestures, use emoji with text, and combine writing with drawings and images in places from doodles to comics to advertising. Yet, the linguistic and cognitive theories maintaining the traditional amodal notion of language cannot account for the richness of this multimodal communication. What is needed is a new, multimodal paradigm of language.

This book presents a model of a multimodal language faculty which heralds a re-organization of the structures of language and their guiding assumptions. It shows that the primary human expressive behaviors - speaking, signing, drawing-may seem distinct, but actually decompose into similar cognitive building blocks, which coalesce into a multifaceted multimodal communicative system. The result is an account of human cognition where all communication - whether speech, gesture, graphics, and their multimodal interactions - arises as emergent activation states out of a singular cognitive architecture. The architecture put forward provides a 'grand unified theory' of language and communication, accounting for the insights of traditional linguistic formalizations, conceptual semantics, gesture studies, Peircean semiotics, visual language theory, and both unimodal and multimodal lexicons, all integrated into a single model. This overall approach directly confronts the traditional notions of language that have stood for at least two centuries with a new paradigm of acknowledging the multimodal nature of human communication, forcing us to reimagine what language is and how it works.