Shaping music in performance
Table of Contents
The notion that music has or takes shape is widely shared across time and culture. Performers in rehearsal or teaching commonly use metaphors of shape to explain the kinds of effects they wish to produce. There is an obvious relationship to the process through which performers shape sounds with their bodies as they play, gestures that may be both executive and expressive, to the written representation of music (in Western cultures), and above all to the shaping of sound through adjustments in loudness, timing and frequency, studied under CHARM in terms of ‘expressive gesture’. The experience is so common that it seems likely to play a fundamental role in musical creativity, yet it has been little studied.
This project offered two converging approaches towards a more exact understanding: 1) documenting and analysing performers’ reported experiences of the way music ‘takes shape’, and the ways in which a sense of shape influences detailed decision-making in performance; and 2) experimental and observational studies exploring listener-responses to musical sounds through gesture and visualisation.
The project team worked with a substantial number of participants drawn from professional performers (focused on a representative selection of instruments – including voice – and on conductors), as well as from teachers and students. Participants were studied through interviews and questionnaires in order to classify and describe the perceptions of music as shaped. Follow-up studies were designed which answered specific questions arising from the results of the interviews.
Two postdoctoral research fellows and a PhD studentship were associated with this project. The first Research Fellow, Helen Prior, worked at the interface between psychology and sociology and investigated performers’ perspectives of music as shape though interviews, questionnaires and perceptual experiments using both recordings and live performance. The second Research Fellow, Dan Tidhar, enabled computational visualisation of musical sound in order to explore listeners’ responses to music as shaped. A research studentship, funded by King’s College London, was held by Mats Küssner, who analysed listeners’ representations (drawn and enacted) of their sense of shape in a wide variety of music, with input from CMPCP Associate Nicolas Gold (UCL). Alongside the empirical work, mechanisms underlying a sense of music as shaped were theorised by Daniel Leech-Wilkinson, project director, drawing in particular on the work of Daniel Stern (on the shape of experience) and Mark Johnson (on embodiment).
Four project workshops pooled experience from music psychologists and audio software engineers with related expertise, and gathered feedback.
This strand of the project was led by Helen Prior and explored musical performers’ experiences of shaping music in performance. Investigations took three main approaches. Firstly, a search was undertaken for documentary evidence of the use of the word ‘shape’ in relation to music. The discovery of many examples of the use of the words ‘shape’ or ‘shaping’ by musicians in masterclasses, by critics in Gramophone, by musical performers in articles, books, and news reports, and by musicologists, confirmed that both the word and the idea of musical shape have been used by musicians for at least a century, and are still widely used today. This overview revealed that shape is used by musicians in a variety of ways, with meanings ranging from overall form to musical expression and phrasing. It also confirmed that we were not introducing an easily accepted but novel metaphor to musicians. These data will be explored further at a later stage, but some examples may be seen in the notes and slides relating to Helen Prior’s presentation at the first project workshop.
The second main approach involved an online questionnaire with over two hundred participants, who had a range of ages and musical experience. Many of our participants were of professional standard as musicians. We found an extremely positive response to the idea of shape being used in relation to music: participants reported using the term when practising, in rehearsals, and when teaching; also in relation to music from a wide range of genres. We also found that the term was considered to be linked to several other ideas – from musical structure to musical expression, emotion, and tension – and in relation to specific musical features such as phrasing, the melodic line, and dynamics. Overall, then, shape has been found to be highly versatile and multi-faceted. A report of the main findings from the questionnaire was prepared for questionnaire participants, and may be viewed here. A report outlining the extent to which the links between music and shape may be language- or style-specific is available, while others relating to the documentary evidence and questionnaire data are planned.
The third approach involved in-depth interviews with a relatively small number of participants. Five professional violinists were interviewed and were asked to discuss and demonstrate various aspects of musical shaping. Careful consideration was given to the appropriate method of data analysis, which formed the focus of the second project workshop. The main method being used to analyse the data was Interpretative Phenomenological Analysis, or IPA, along with other supporting approaches such as the analysis of musical excerpts using Sonic Visualiser. An overview of the initial findings was presented at the Performance Studies Network conference in July 2011, and can be viewed here. Outputs from these data are envisaged to relate to three main ideas. Firstly, one participant appeared to show particularly close and interesting links between his decisions about musical shaping and his personal and musical identity. Secondly, participants discussed a range of more or less technical approaches to their control of the musical shape, and these were explored in detail. Thirdly, participants used movement-related metaphors and physical gestures to aid them in their descriptions of musical shaping. A report of the findings intended for interview participants is available.
Representations: visualisations of sound and music
Research on this strand of the ‘Shapes’ project was undertaken by Mats Küssner and explored people’s visualisations of sound and music. The active process of shaping music visually allows insights into many aspects of music perception, cognition and behaviour.
In the resulting PhD thesis, Mats sought to understand what kinds of shapes listeners with varying levels of musical expertise perceive in sound and music by engaging them in overt actions. To that end, two empirical studies were carried out. In the first experiment, using a method called MUVISTA (MUsic VISualized by means of a graphics TAblet), developed in close collaboration with Nicolas Gold, who provided the capturing software, a sample of musically trained and untrained participants was asked to represent visually a series of pure tones varied in pitch, loudness and tempo – as well as two short musical excerpts – by means of an electronic graphics tablet. In the second experiment, entitled MUVISKI (MUsic VISualized KInaesthetically), a new sample of musically trained and untrained participants was asked to represent gesturally a series of pure tones varied in pitch, loudness and tempo, as well as sixteen short musical excerpts. In one of two experimental conditions, participants’ gestures – captured with Microsoft® KinectTM and Nintendo® WiiTM Remote Controller – created a real-time visualisation on a screen in front of them. In order to shed light on cross-modal mappings between drawing/gesturing features (x-, y- and z-coordinates) and sound features (pitch, loudness), correlation analyses, as well as more advanced mathematical tools such as Gaussian processes, were applied.
Results revealed that musically trained participants are generally more consistent in representing sound features cross-modally (e.g., pitch–height) but also less diverse in their approaches than untrained participants. Most participants mapped pitch onto the vertical axis and time onto the horizontal axis. Loudness was mostly represented by size in drawings and by various mapping strategies in gestures such as height, size and muscular energy. Representing musical excerpts gesturally led to a wide range of strategies including, dancing, conducting, air instrument playing and tracing of musical features. Findings were discussed in light of embodied music cognition and current theoretical developments within the cognitive sciences.
Visualisations of music and gesture
Research on this strand of the ‘Shapes’ project was led by Dan Tidhar and aimed to explore and develop computational music visualisation methods which were compatible with listener intuitions about musical shape.
Our starting point was the exploration of various visualisation techniques based on automatic extraction of audio features. Such techniques are readily applied in different contexts, and although they do arguably bear relevance to the notion of shape, it was often secondary to the particular purpose of the visualisation, be it artistic expression, entertainment, or audio analysis. Our aim was to produce automatic visualisations which would bring forward the notion of musical shape. We explored two complementary approaches – frame-level visualisation, which gives rise to emergent shapes and phrase-level visualisation, which is based on segmentation and longer-term dependencies.
We were particularly interested in the connection between the dynamic aspects of musical shape and bodily gesture. To this end, an experiment was prepared, in which gesture data was collected and analysed from participants’ responses to musical stimuli. Data collection was carried out using dedicated software facilitating tools such as the Microsoft Kinect and the Wii Remote. While responding to musical stimuli, direct visual feedback may have significant influence on the form and scale of bodily gestures performed by participants. A pilot experiment assessed different visual feedback mechanisms (e.g. a simple geometrical shape following hand movements with a decaying trace, versus disturbances in a physically modelled particle field). Prototyping was done in Processing using the OpenNI Kinect drivers. We used OpenFrameworks for implementing the main experiment.
Once sufficient gestural data had been collected and analysed, they were used for informing and modulating automatic visualisation techniques. In the light of Mats Küssner’s findings on the Representations strand of the project, we considered different ways in which our gesture data could be integrated in feature-based visualisations and so make them more compatible with human intuitions about dynamics, gesture, and shape.
Music and shape: underlying mechanisms
While Helen Prior investigated performers’ experiences of shape, and Dan Tidhar and Mats Küssner looked at listeners’ shape-related responses to music, Daniel Leech-Wilkinson took a more theoretical approach, drawing on existing research in embodiment and cross-modal mapping in order to suggest perceptual mechanisms that may explain the ease with which music seems shaped.
In outline, it is suggested that changes made by performers in the loudness, speed or pitch of notes bring to performances a sense of constant change in the dynamics of the music (dynamics in the sense of changing motion and intensity), and that it is this more abstract sense of dynamic change that is felt as changing shape. In other words, shape in musical performance and response may have more to do with changing feeling-state than a sense that music happens in two- or three-dimensional space. Daniel Stern’s work on forms of vitality offers one way of thinking about the changing shape of feeling (Stern 2004, 2010) while Mark Johnson’s recent work on embodiment offers a way of understanding in general terms how our awarenesses of tension, linearity and amplitude, fundamental to the dynamics of music, ‘are qualities of organism-environment interactions‘ and derive from our experience of living and moving within our environment (Johnson, 2007).
Mechanisms which encourage the brain to map promiscuously between the dynamics of sound, feeling, and embodied knowledge (among many other phenomena) are likely to operate at a pre-conscious level on which these various perceptual domains share qualities of a sort that, precisely because they are pre-conscious, cannot be directly perceived: one perceives a likeness without being able to say why. Cross-modal perception (and its specialised variant, synaesthesia) offer examples of the consequences of the routine operation of these mechanisms. Studies by Näätänen & Winkler (1999) and McLachlan & Wilson (2010) suggest how they may work, while recent studies of multisensory perception (Stein et al., 2004; Stein & Stanford, 2008) and the implications of mirror neurons (Gallese & Lakoff, 2005; Molnar-Szakacs & Overy, 2006) emphasise the importance and ubiquity of cross-modal perception in general.
In this context, the widespread perception that music in performance is continually changing shape can be seen as an inevitable by-product of the way the brain responds to sensory inputs of all kinds.
- Mark Johnson. (2007). The Meaning of the Body: Aesthetics of human understanding. Chicago: University of Chicago Press.
- Vittorio Gallese & George Lakoff. (2005). The brain’s concepts: the role of the sensory-motor system in conceptual knowledge. Cognitive Neuropsychology 22:3, 455-479.
- Istvan Molnar-Szakacs & Katie Overy. (2006). Music and mirror neurons: from motion to ‘e’motion.Social Cognitive and Affective Neuroscience 1, 235–241.
- Risto Näätänen & Istvan Winkler. (1999). The concept of auditory stimulus representation in cognitive neuroscience. Psychological Bulletin 125:6, 826-859.
- Barry E. Stein, Wan Jiang, & Terence R. Stanford. (2004). Multisensory integration in the single neurons of the midbrain. In Gemma A. Calvert, Charles Spence and Barry E. Stein, ed., The Handbook of Multisensory Processes. Cambridge, Mass.: MIT Press, 243-264.
- Barry E. Stein & Terrence R. Stanford. (2008). Multisensory integration: current issues from the perspective of the single neuron. Nature Reviews: Neuroscience 9, 255-266 & corrigendum.
- Daniel Stern. (2004). The Present Moment in Psychotherapy and Everyday Life. New York: Norton.
- Daniel Stern. (2010). Forms of Vitality: Exploring dynamic experience in psychology, the arts, psychotherapy, and development. Oxford: Oxford University Press.