Shaping music in performance

(Daniel Leech-Wilkinson – King's College London)

Project overview | Performers' perspectives | Representations | Visualisations | Mechanisms | Recruitment for current studies | Workshops | Music and Shape conference

Visualisations of music and gesture

Research on this strand of the 'Shapes' project is led by Dan Tidhar and aims to explore and develop computational music visualisation methods which are compatible with listener intuitions about musical shape.  

Our starting point is the exploration of various visualisation techniques based on automatic extraction of audio features. Such techniques are readily applied in different contexts, and although they do arguably bear relevance to the notion of shape, it is often secondary to the particular purpose of the visualisation, be it artistic expression, entertainment, or audio analysis. Our aim is to produce automatic visualisations which would bring forward the notion of musical shape. We are exploring two complementary approaches – frame-level visualisation, which gives rise to emergent shapes and phrase-level visualisation, which is based on segmentation and longer-term dependencies.

We are particularly interested in the connection between the dynamic aspects of musical shape and bodily gesture. To this end, an experiment is currently being prepared, in which gesture data will be collected and analysed from participants' responses to musical stimuli. Data collection will be carried out using dedicated software facilitating tools such as the Microsoft Kinect and the Wii Remote. While responding to musical stimuli, direct visual feedback may have significant influence on the form and scale of bodily gestures performed by participants. A pilot experiment is currently under way, assessing different visual feedback mechanisms (e.g. a simple geometrical shape following hand movements with a decaying trace, versus disturbances in a physically modelled particle field). Prototyping is currently being done in Processing using the OpenNI  Kinect drivers. We are considering using OpenFrameworks for implementing the main experiment.

Once sufficient gestural data have been collected and analysed, they will be used for informing and modulating automatic visualisation techniques. We are currently using the Sonic Visualiser and VAMP plugin set as a platform for exploring existing feature extraction algorithms, and for implementing additional ones. We are considering different ways in which our gesture data can be integrated in feature-based visualisations and so make them more compatible with human intuitions about dynamics, gesture, and shape.