multimodal methods for analyzing communication and learning with digital technology
TRANSCRIPT
Summary● Multimodal Social Semiotics & Meaning Making
● Multimodal Social Semiotics: some definitions
● Multimodal Social Semiotics & Research Methods : Interaction analysis
● Multimodal Social Semiotics & Research Methods : video analysis & transcripts
● Applications:○ Learner / User generated content○ Embodiment & Technology
■ Embodiment & Technology: a case study
Social Semiotics Semiotics:
● Study of signs and signification processes● Interpretative semiotics (double formation of meaning / Semiosis:
processes of signification and interpretation (Eco))
Differences between traditional semiotics and social semiotics● Arbitrariness (signifier and signified):
- traditional: arbitrary rules that can not be modified (De Saussure)- social: the "codes" of language and communication are formed by social processes (= social dimension of meaning) :→ rules made by people that can be modified (through institutional or symbolic power)→ motivated sign: instead of fixed and unchanging "codes", signs are considered to be resources that people use and adapt to make meaning accordingly to the interests of the producer/interpreter and the characteristics of the objects
MEANING
MultimodalityMultimodality & Social Semiotics:People use differents modes to make meaning. Going being linguistics approach → framework for the analysis of visual, aural, embodied and spatial aspects of the interaction and the environment
● Each mode has different affordances that are the product of the social and historical use. The form should be particularly apt for conveying a certain meaning
● People orchestrate meaning through their selection and configuration of modes
meaning potential of an artifact
social & cultural environment
resources, interests and knowledge of the person
Multimodal Social Semiotics: some definitionsMode: a set of resources for making meaning and organizing principles, a channel of communication
Semiotic resources (Halliday) : a resource for making meanings → action, material and artefacts that we use to communicate (difference with sign). They have semiotic potential and semiotic affordances. Ej.: written text, image, walking
Multimodal Social Semiotics: some definitionsSemiotic potential: each semiotic resource has semiotic potentials (rather than a specific meaning) that are culturally and historically shaped
Affordance: material, social and cultural (not just perception!)
Semiotic affordance: each semiotic resource has semiotic potentials that are new meanings that have not been recognized yet and are latent in it
Modal affordance: potentialities and constraints of different modes; what is possible to express and communicate easily with the resources of a mode and what is less straightforward or even impossible. They are historically and socio-culturally shaped.
Multimodal Social Semiotics: some definitionsMotivated sign: agency of the sign maker, draws the attention on the interests and intentions that motivate a person’s choice of a given semiotic resources
multimodal ensemble: bring together different mode and look at how different mode bring different meaning
simultaneity: all the semiotic resources mutually elaborate each other. Each one by itself is incomplete. together they create a whole that is bigger than its part
coherence & cohesion: how different modes work together to produce coherence and make a complete “text” (are the used modes coherents between them or they contradict?) / cohesion: devices to construct coherence
Van Leeuwen, Theo. Introducing social semiotics. Psychology Press, 2005.Jewitt, C (2012) Multimodal methods for researching digital technologies. In: The Sage Handbook of Digital Technology Research. Sage, London.
Design: theoretical perspective
Full Body Interaction Environment as Multimodal Ensemble● what modes are present?
- which are their semiotic resources? semiotic potentials and modal affordances? what are the potentials and constraints of Full Body Interaction? What is possible to communicate and express easily with this mode? What is less straightforward or even impossible? (inventories, socio-cultural references)● How the different modes relate together?
Multimodal Ensemble
Semiotic Resources
Semiotic potentials
Modal affordances
Applications: Interaction analysis
HISTORY:
‘30/’60: Interaction Sociology (Goffmann, Garfinkel) → bottom- up approach to study society
‘60: Conversation Analysis
‘70: Conversation Analysis & Embodiment (Goodwin)
‘90: Workplaces studies (Goodwin, Mondada)
- role of body and action in achieving collaboration
- embodiment in conversations
CONCEPTS:- Adjacency pair (invitation to the other to respond) & paired action (adjacency but through action)
- Reading other bodies (Hindmarsh and Pilnick, 2002): people read each other bodies; you see somebody doing something and if you know the context you will know what to do next and what is expected from you (ej. study in anesthesia: they don’t speak so much but they coordinate each other through action & body)
Multimodal Social Semiotics & Research Methods : video analysis & transcriptsFrom video to data:- at the end of the day do a little description of each video to describe what is going on
- vignette, summary
- coding categories
- multimodal transcription - visualization
Multimodal transcription :- Define a methodological framework
- Define the purpose and focus of the transcript (ej. timescale, modes)
- Design the transcript
Multimodal Transcription: the quality of transcript is not how much it is a representation of the reality but the point to which it allows research your topic
- the modes selected for transcription depend on the focus of the research
Multimodal Social Semiotics & Research Methods : video analysis & transcripts
Bezemer, J. (2012). How to transcribe multimodal interaction? In Texts, Images and Interaction: A Reader in Multimodality. Mouton de Gruyter.Mavers, D. (2012). Transcribing video. NCRM working paper.
Applications: Learner / User generated content→ Motivated Sign / Form as meaning
● What kind of semiotic resources do they use to communicate something? Which cultural references it relate to?
● What are the resources available? Which one are selected and how they are re-shape? (which other selection could have been possible)
● How the different resources are orchestrated between them?
● Is it coherent? How this coherence have been constructed?
● What she wants to express with it?
Applications: Embodiment & Technologymultimodality / ethnography / psychology
DESIGN● understand the potential that different modes can bring to interaction
(modal affordance; which mode can be apt for this purpose?)● possibilities of identifying novel semiotic resources● ...
ANALYSIS● emphasis on situated meaning making: how people make meaning
in a specific situation● analyze which modes are used in the environment and which
resources are available ● examine and compare the effect of using different modes in learning
context● ...
Embodiment & Technology: case study
goal:METHOD:1) Logging Video Data: descriptive account (time, content, events, modal configuration): vignette, sketches, screen-capture, maps..2) Raw analysis: watch the video in different way (sound, slow/fast motion) → start to generate criteria for sampling3) Sampling: select episodes / phenomena / etc. // category definition4) Transcription 5) Modal analysis:
- break down the modes (ej: position, gaze, hand-action)- look for intermodal relation (ej: pace)
6) Incorporate the context into the analysis
Light Table
Design: participatory design - children production
● What kind of semiotic resources do they use to communicate something? Which cultural references it relate to?
● What are the resources available? Which one are selected and how they are re-shape? (which other selection could have been possible)
● How the different resources are orchestrated between them?
● What she wants to express with it?
Semiotic resources
Semiotic potentials
Motivated Sign