[Event] Beyond Words: Multimodal Encounters in Translation

Beyond Words: Multimodal Encounters in Translation

5 July 2018 – 6 July 2018

SG1 and SG2, Alison Richard Building

Registration for this conference is now open. Fees are £40 (full price) or £20 (student/unwaged); one-day registration is also available. Fees include lunches and refreshments. Registration closes on Sunday 1 July.

A limited number of spaces are also available at the conference dinner, which will be held at Sidney Sussex College on 5 July. The cost of this is £50. It will only be possible to reserve a space at dinner until 15 June, and this option will be removed from the registration form when our capacity limit is reached. 

Please note that papers will be pre-circulated to all attendees ahead of the event, with the intention of allowing as much time for discussion as possible at the conference itself. 



Monica Boria (University of Cambridge)

Angeles Carreres (University of Cambridge)

María Noriega-Sánchez (University of Cambridge)

Marcus Tomalin (University of Cambridge)



Research into the theory and practice of translation has traditionally focused on the conversion of source texts into target texts. However, during the past decade the rise and sprawl of digital media has ensured that interconnections between different visual, aural, and oral modalities have acquired much greater cultural prominence – a development that has destabilised certain time-honoured translation-theoretic paradigms. This shift has directed critical attention towards acts of translation involving more than one modality. Moving beyond text-to-text translation, pioneering work by a number of scholars has begun to explore different kinds of text-to-music, text-to-dance, text-to-image, dance-to-image, music-to-dance, dance-to-text, and image-to-music transfers. While some of these approaches draw upon well-established traditions of ekphrasis and iconology that can be traced back through the centuries, they also probe in new and provocative ways the limits of these activities. Indeed, for some scholars the very term ‘translation’ has become an inadequate one for describing the full range of interactions involving the transfer of meaning from one modality to another. Therefore, terms such as ‘transduction’, ‘transaptation’, ‘transformation’, and ‘transcription’, have sometimes been usefully repurposed. But how do these practices differ from each other, and what are their distinctive respective characteristics? While this is still an inchoate field of enquiry, it has already inspired ground-breaking analytical approaches that deserve careful scrutiny. A core purpose of this event is to bring together both those who produce multimodal ‘translations’ as well as those who theorise about them. By encouraging truly inter and trans-disciplinary dialogue, this conference aspires to impact on research directions in the area of translation and multimodality.

The conference will take the form of a two-day event. Each of the eight main speakers will prepare a paper in advance, and these will be distributed to all the registered delegates two weeks before the start of the event. Each speaker will have a 20-minute slot during the conference in which to give a short verbal summary of their paper, followed by a further 25 minutes of questions from the attendees. In addition, there will be two workshops that will showcase practical multimedia examples of multimodal translation in action. At the end of each day there will be a ‘Round Table’ discussion session, chaired by one of the convenors, that will provide an opportunity to explore in greater detail specific points of connection and disconnection between the presentations and workshops.



Day 1 – Thursday 5 July

9.15 – 9.45 Registration & Coffee
9.45 – 10.00 Welcome & Introduction
10.00 – 11.40 Session 1

Gunther Kress (UCL)

‘Making and re-making meaning: “translation” in a Social Semiotic Multimodal approach’


Klaus Kaindl (University of Vienna)

‘A theoretical framework for a multimodal conception of translation’

11.40 – 12.10 Break
12.10 – 13.00 Session 2

Carol O’Sullivan (University of Bristol)

‘Multimodal Translations: subtitling and image-text relations, 1920-1950’

13.00 – 14.15 Lunch
14.15 – 15.15 Session 3

[Speaker to be confirmed]

15.15 – 15.45 Break
15.45 – 17.00 Roundtable

Day 2 – Friday 6 July

9.15 – 9.30 Welcome & introdution to the day
9.30 – 11.10 Session 4

Helen Julia Minors (Kingston University London)

‘Translations between music and dance: analysing the choreomusical gestural interplay in twentieth and twenty-first century dance works’


Matthew Reynolds (University of Oxford)

‘Translation in the in-betweens between speech, writing, and illustration’

11.10 – 11.40 Break
11.40 – 12.30 Session 5

Elisabetta Adami (University of Leeds)

‘Translation and semiotics between facepalms and thumbs-ups: meaning-making in a world of untranslated signs’

12.30 – 13.30 Lunch
13.30 – 14.20 Session 6

Luis Pérez González (University of Manchester)

‘Subtitling the performance of citizenship in the digital culture: a multimodal perspective’

14.20 – 15.20 Workshop

María Mencía (Kingston University London), with Gabriel Gaudette, Manuel Portela and Arnaud Regnauld

‘Translating electronic literature: a presentation of multimodal transferral of literary works’

15.20 – 15.45 Break
15.45 – 16.15 Closing Roundtable


Elisabetta Adami (University of Leeds)

‘Translation and semiotics between facepalms and thumbs-ups: meaning-making in a world of untranslated signs’

Research in translation studies (e.g., O’Sullivan, 2013; Pérez-González, 2014) is increasingly considering the implications for translation theory and practice that derive from multimodality, both as a field of research and as a phenomenon characterizing the ‘texts’ to be translated. As a social semiotician, I am rather interested in understanding the impacts of translation (in its development as a social practice) onto sign-making in all modes. Hence, the paper will

  • start from the assumption that semiotic resources such as gesture, face expression, clothing, music, still and dynamic image have not been subject to national codification and a history/tradition of translation to the same extent as language,
  • discuss how this has influenced the ways in which we make meaning through these resources when they circulate transnationally, and
  • derive implications for (2) social semiotic research, when addressing issues of culture(s) and translation, and (3) translation practice, faced increasingly with multimodal representations designed for transnational and transmedia circulation.

The discussion will use data samples from work currently being undertaken at the Centre for Translation Studies at the University of Leeds. Data contexts include audio-visual subtitling, document design and localization, deaf/hearing interactions, and sign-making practices in multicultural/multilingual contexts both online and offline.


Klaus Kaindl (University of Vienna)

‘A theoretical framework for a multimodal conception of translation’

In Translation Studies, the debate about multimodality is characterized by a variety of heterogeneous terminologies that ultimately all reflect a monomodal conception of the discipline. The first aim of this presentation is to lay the foundations for a systematic multimodal approach to translation. As a first step, the innovative potential of the concept of multimodality is critically analyzed against the background of the research tradition in semiotics. As a second step, the theoretical, textual and taxonomical needs and consequences of a multimodal understanding of translation are discussed. In order to develop a multimodal understanding of translation, it must be recognized that modes are intimately intertwined with medium and genre. As a last step, the interrelations among these three concepts are highlighted drawing on examples from popular music translation. Exploring the process of transfer, and this is the second aim of my presentation, can contribute to a better understanding of the functional specificity of modes, media and genres.


Gunther Kress (UCL)

‘Making and re-making meaning: “translation” in a Social Semiotic Multimodal approach’

Social Semiotics assumes that meanings are the product of social (inter-) action, hence meaning is seen as arising out of (and intricately linked with) social actions. Social Semiotics aims to outline the semiotic principles underlying meaning-making, and to provide accounts of the means for making meaning material. Two central terms are sign (as the basic unit of meaning) and mode (as the socially shaped material means for making meanings evident): e.g. sound as speechmovement (of part of the body) as gesturemarks on surfaces as image; etc. ‘Social life’ is a constant process of meanings made with apt means in one site and re-constituted with means apt in a different site. That requires accounts of the site-specific mode-resources for making meaning and of the site-specific mode-resources for re-constituting meaning. In my account I will provide examples of meanings made in one site and re-constituted as meanings apt for, and in, a different site. The process outlined in this ‘sketch’ is intended to account for processes of making and re-making of meaning within one ‘culture’ and its resources; and, equally, for processes of making meaning in one culture and its resources, and its re-constitution in another culture with its resources.


María Mencía (Kingston University London), with Gabriel Gaudette, Manuel Portela and Arnaud Regnauld

‘Translating electronic literature: a presentation of multimodal transferral of literary works’

Over a year and a half ago, a group of scholars, programmers, artists and translators started working on a research project focusing on the translation of various works of electronic literature, including María Mencía’s The Poem that Crossed the Atlantic, Luís Lucas Pereira’s Machines of Disquiet and Michael Joyce’s afternoon, a story. In order to identify common and divergent issues depending on the genres, formats and languages of the works under study, they were all examined through the prism of the following concepts: Translinguistic translation (translation between languages), Transcoding (translation between machine-readable codes and between machine-readable codes and human-readable text), Transmedial translation (translation between medial modalities), and Transcreation (translation as a shared creative practice). This workshop will give the members of the project the opportunity to share their observations on their work process, bridging the gap between the practice-based approach and a theoretical perspective on the task of translating electronic literature. In addition to a brief presentation of each work and the specific challenges they raised, the participants will offer key insights into the collective methodologies elaborated throughout the duration of the project. Members of the audience will have an opportunity to explore the works.


Helen Julia Minors (Kingston University London)

‘Translations between music and dance: analysing the choreomusical gestural interplay in twentieth and twenty-first century dance works’

This paper explores the processes of translation within 20/21st-century music-dance works. It aims to illustrate the transfer of sense between the arts for those who are part of the creative process and for scholars interpreting these works. It questions how meaning is produced within and between these temporal expressive arts. It claims that the interplay of musical movement and danced movement relies on various forms of translation which require cognitive mapping, gestural interpretation, and an awareness of the somatic experience. This paper questions: Can a process of translation be read when choreographers work with pre-composed music? How do composers and choreographers utilise a process of translation and transfer when they collaborate in forming new music-dance works? How do spectators read those emergent meanings between the audio-visual elements? Focusing on narrative balletic works, three case studies illustrate three types of translation:

  1. sensory translation is illustrated through a reading of Vaslav Nijinsky’s approach to Debussy’s Prélude à l’après-midi d’un faune and to Igor Stravinsky’s Le sacre du printemps (1912-1913).
  2. language translation is illustrated through music visualisation, where syntax of music-dance are read as analogous to language: examples include Béjart’s Bolero (1960) and Morris’s Dido’s Lament (1995).
  3. cultural translation is illustrated using the revival of Satie’s La Parade (1917) in cultural translation in the new version, P.A.R.A.D.E., produced by National Dance Company Wales (2017) to celebrate the centenary of the Russian Revolution.

Using translation as a critical framework, I show how sense is transferred between the arts of music and dance, and how meaning is emergent through their interplay.


Carol O’Sullivan (University of Bristol)

‘Multimodal Translations: subtitling and image-text relations, 1920-1950’

This chapter analyses interactions between written text and the moving image in films of the first half of the twentieth century, and their implications for translation. It looks at four such kinds of interaction in particular. The first is the use of superimposed text on the image. Used from the 1910s onward, titles on action never became the standard, but functioned as a special effect in silent features including Ben Hur (Fred Niblo, 1925) and Casanova (Alexandre Volkoff, 1927). The second type of multimodal interaction under discussion is the standard subtitle. With the introduction of sound film in the late 1920s, superimposed titles were co-opted to translate foreign-language dialogue. It took some years for titles to settle into a format resembling the current one; in the early years, for instance, subtitles could be placed in different areas of the frame to reflect the position of the speaker, the pattern of light and dark in the frame, etc. The third area I will look at is typography; following Theo van Leeuwen and Michel Chion, I will look at how the typography of silent title cards, subtitles and in-vision text can carry meaning and how this poses challenges for translators. Lastly, I will look at in-vision texts such as telegrams, letters and notices which were frequently reshot in dubbed, or even subtitled films. It will be argued that these multimodal practices posed considerable technical and aesthetic problems for the distribution of films in translation, and that these experiments are mirrored in the proliferating use of written text in the image in film and television today.


Luis Pérez González (University of Manchester)

‘Subtitling the performance of citizenship in the digital culture: a multimodal perspective’

As the ontological shift from referentiality to deconstruction triggered by the transition from ‘recording technologies’ to ‘synthesizing technologies’ (Kress and van Leeuwen 2006) continues to take hold, ‘the need for conceiving of social semiotic practice in terms of rhetoric and design’ (ibid: 219) has become unanimously acknowledged. This need is particularly acute in the digital media ecology, moulded by variable degrees of convergence between industrial and amateur practices. In those habitats where amateur rhetors have gained greater visibility and influence, the performance of citizenship often involves the deconstruction of representation by exposing the cultural and social make-up of specific semiotic resources within the overall multimodal ensemble.

Unlike critical theory and media sociology, translation studies has been slow to tackle the theoretical challenges that arise as digital media content moves away from the normative logic of linguistic referentiality, which assumes that texts should be faithful representations of the reality they draw on, to enable the emergence of a more deliberative and eclectic public sphere (Chouliaraki 2010). This paper draws on a growing body of research showing that the production and consumption of digital media content allows for playful or ethical forms of multimodal self-expression. It then goes on to examine how various processes of multimodal experimentation undertaken during the subtitling of digital media content, not limited to the reproduction of the verbal component of the subtitled text, contribute to opening up ‘alternative spaces’ for the negotiation of subjectivity and, beyond that, constructing – rather than simply recreating (Baker 2013) – the cultural encounters those texts are embedded in.


Matthew Reynolds (University of Oxford)

‘Translation in the in-betweens between speech, writing, and illustration’

This essay and/or talk will begin from the inherent multimodality of language across sight and sound. I will consider the letter ‘I’, offered by Adam in Dante’s Paradiso as the word for God in his long-lost Edenic language, and of course familiar in English as the first-person subject pronoun (sometimes mistaken for number 1). ‘I’ can become a picture of the self, as in the pillars of text formed by Victorian dramatic monologues such as Browning’s ‘Fra Lippo Lippi’ and Tennyson’s ‘St Simeon Stylites’ (and perhaps also in the paintings of Barnett Newman); because of this it poses a particular and suggestive challenge to translation. I will endeavour to trace the translational ramifications of ‘I’, and possibly of other pictures-that-are-in-words, hoping to elucidate the element of illustration that inheres in translation, and vice versa, as well as the pictorial aspect of all writing. Points of reference beyond those already mentioned are likely to include Charlotte Brontë, Samuel Beckett, Christine Brooke-Rose, and Jorge Méndez Blake.


The conference is part of the activities of the research group Cambridge Conversations in Translation, based at the Centre for Research in the Arts, Social Sciences, and Humanities (CRASSH). The event is supported by CRASSH, the Institute of Modern Languages Research (IMLR), and the Cross-Language Dynamics: Reshaping Community (CLDRC) OWRI project (Translingual Strand).


Administrative assistance:conferences@crassh.cam.ac.uk

Unfortunately, we are unable to arrange or book accommodation for registrants. The following websites may be of help: