Interactive Sonic System + Motion Capture // accelerometers + EMGs //

Collaboration with: Jules Francoise and Frederic Bevilacqua (IRCAM), Thecla Schiphorst (SIAT)


In this work, we investigate the use of interactive sound feedback for dance pedagogy based on the practice of vocalizing while moving. Our goal is to allow dancers to access a greater range of expressive movement qualities through vocalization. We propose a methodology for the sonification of the Effort Factors, as defined in Laban Movement Analysis, based on vocalizations performed by movement experts. Our approach to movement sonification uses vocalizations performed by Certified Laban Movement Analysts along with movement with various Effort Factors. We developed a sonification system based on a method called Mapping–by–Demonstration. Mapping refers to the relationship between the movement features and the sound parameters. Rather than formulating an analytical description of the relationship between movement and sound, Mapping–by–Demonstration trains a machine learning model based on Multimodal Hidden Markov Models (MHMMs) on examples of performances of movement with the various Laban Effort qualities associated with vocalization sounds in order to learn the motion–sound interaction model. The system is then used for interactive sonification: during the Performance phase, the participants’ live movements qualities are interpreted by the system that continuously estimates the associated voice features in order to re-synthesize the pre-recorded vocalizations using corpus-based sound synthesis.

Based on the experiential outcomes of an exploratory workshop, we propose a set of design guidelines that can be applied to interactive sonification systems for learning to perform Laban Effort Factors in a dance pedagogy context.

Publication: Jules Francoise, Sarah Fdili Alaoui, Frederic Bevilacqua,  Thecla Schiphorst, “Vocalizing Movement for Laban Effort Sonification in Dance Pedagogy”, In Proceedings of ACM conference on Designing Interactive Systems (DIS), Vancouver, 2014. DIS2014-francoiseetal