by ALICIA PEÑALBA, MARÍA-JOSÉ VALLES, ELENA PARTESOTTI, ROSARIO CASTAÑÓN, MARÍA-ÁNGELES SEVILLANO
Background
Digital musical instruments (DMI) are technological devices that disrupt the causal relationship between performance and sound outcomes. A digital instrument is ‘an instrument that includes a separate gestural interface (or gestural controller unit) from a sound generation unit’ (Wanderley & Depalle, 2004, p. 633). This entails a paradigm shift by comparison with acoustic instruments insofar as any gesture may be linked to any sound, so that the relationship itself does not necessarily match a causal logic, but rather allows for some degree of indeterminacy. Among gestural controllers, the alternate controllers are those ‘whose design does not follow that of an established instrument’ (Wanderley & Depalle, 2004, p. 636). MotionComposer (MC) fits into this category and constitutes ‘a device that turns human movement into music using state-of-the-art video based and 3D sensing motion-tracking technology combined with tailor made sound generating software’ (Bergsland & Wechsler, 2013, p. 1). The possibility that any gesture may produce any sound outcome perceived as one’s own possesses great potential for therapeutic, educational and artistic contexts (Peñalba & Wechsler, 2010; Kontogeorgakopoulos, Wechsler, & Keay-Bright, 2014), since it can broaden the range of gesture and self-expression as well as enhance the participants’ capabilities no matter how diverse they are.
Our research is part of the European scheme known as METABODY, ‘a five-year European Project coordinated by Reverso that started in July 2013. (…). [It] will elaborate a critical study of cultural homogenization, social control and global surveillance in the information society’ (Del Val, 2014, p. 149). While technology holds great potential for people with motor impairments or for exploring new sound-gestural possibilities in the artistic domain, it can also become a trap for human gestural control (i.e. technology determining human gesture), since individuals adapt their gesturality to suit the canons promoted by the media and social networks. Del Val (2009, p. 129) has coined the term Panchoreographic, which he defines as ‘a set of technological devices of global distribution characteristic of a culture based on leisure, information and communication (…) that disseminates standard choreographies in bodies’. For this reason, METABODY ‘will develop new technological paradigms that take into account the changing differentials of bodies, contexts and movements in their irreducibility, valuing and highlighting the importance of indeterminacy for a livable life’ (Del Val, 2014, p. 149).
Aims
The goal of this paper is to demonstrate several types of interaction achieved through MC in individuals with differing characteristics in terms of age, disability, gender, training background, experience, etc., by focusing on both standardization and diversity.
Method
There were 170 participants of different ages, gender and training backgrounds, including individuals with disabilities. MC provides six environments involving different sounds and gestural analyses. Only two of them have been employed for the purposes of our study.
The first environment, which comprises a causal type of mapping, is named Fields, and transforms the movements produced by the user into sounds by dividing the space into two parts: on the left there are the wind sounds, while on the right there are the water sounds. In the absence of movement, no sound is produced, while minimal movements (small and discreet) generate short and soft blowing sounds or the sound of water drops falling, depending on which side we are considering. Broader movements, in turn, produce wind- and water-like sounds that range from the very soft to the very intense, depending on the amount of movement. When movement is abrupt, a storm breaking can be heard. The second environment is Tonality (chair): using a non-causal mapping, it produces piano melodies on the individual’s right hand and vibraphone melodies on the left hand that are heard in stereo through each loudspeaker channel.
Each participant improvised for four minutes without following any pre-established pattern, and performances were videotaped while two members of the research team made entries into an observation template. Interviews were also conducted, in order to collect data about participants and their impressions during the experiment. Moreover, we used the SAM test (Self Assessment Manikin Test) (Lang, 1980) to assess moods and degrees of excitement and spot possible changes in these parameters as estimated three times throughout the process.
Results
Our intention was to avoid reductionist taxonomies, so that the kinds of interactions that we propose are in no way impervious or exclusive categories (this should be noted when evaluating the percentages that we provide). The observation of the several types of interaction has been conducted on the basis of the classification suggested by Peñalba (2008): movement-based, sound-based and contingent interaction (Table 1).
Table 1. Types of interaction (N = 170)
A number of participants (46%) moved in stereotypical ways without practically listening to the sound that resulted from their own bodily motions. In general, this tended to be the case with dancers or people with some dance training (77% of all participants in the Fields environment and 85% in the Tonality setting). This kind of interaction is called movement-based.
Other participants attempted to use MC in order to recreate a musical instrument (45%). Thus, they tried to achieve a sound outcome through their own exploratory movements. On a gestural level such movements do not generally appear to be so stereotypical. This kind of interaction was more frequent in people with some musical training or with experience in music-related activities (77% both in the Fields and in the Tonality environments) and is known as sound-based.
Finally, other people interacted in a contingent way (15.5%). This means that their movements produced some kind of sound, yet at the same time that sound inspired them to move in particular ways. This kind of interaction appears to depend on the particular training the individual has received in specific artistic fields.
In all categories there were subjects with and without disabilities and belonging to different age groups and gender, but no significant differences were found concerning the observable interaction types. The results show several significant differences in terms of the kind of gesture displayed by Fields (involving a more causal mapping) and Tonality (which entails a higher level of indeterminacy). Fields promotes a wider diversity of gestures (concerning the body parts involved, location, novelty, use of space), as well as more discreet, slight and abrupt gestures. In Tonality, by contrast, we observed more stereotypical, continuous and fluid movements.
Conclusions
In the group of people with a musical background a sound-based type of interaction prevails. Playing a musical instrument implies recreating a sound ideal that is achieved by means of a specific type of gesture. When using MC there is no sound-based ideal, but the musician tends to put into practice a similar bodily mechanism that is ultimately regulated by listening.
Dancers, on the contrary, present interactions of the movement-based type. In general, dancing is based on a gestural/choreographic ideal that relies on a pre-existing music. With MC, acquired gestural patterns are reproduced, while sound itself and sound-related explorations play a secondary role.
Only a few participants developed a contingent type of interaction where the individual does not start from a sound- or gesture-based ideal, but instead performs his/her creation on the basis of what sound and gesture jointly evoke. This appears to be more frequent in people who have a background in both music and body-related disciplines, because it implies listening and also body-management skills.
Also worth mentioning are the differences between the kind of gestures respectively displayed in the Fields and Tonality settings. Movements in Fields stem from causation: the environment provides participants with a certain degree of control – on the gestural level – of sound-related responses, which in turn feeds back gesturality. In the Tonality mode, however, the sound-based response is much less predictable and encourages a disconnection from sound while focusing instead on the bodily side and largely favouring more stereotypical motor responses.
As has been pointed out, people with and without disabilities do not significantly differ in their interaction with MC. The fact that no interaction patterns are suggested makes it easier for all participants to stay on an equal footing as regards skills and expressive capabilities. It moreover reveals that this device constitutes a learning mediator for any kind of person and an effective tool in order to eliminate barriers. This is truly a breakthrough with regard to other forms of artistic expression where conditions like age, gender or disability may limit or restrain opportunities. The participants’ training or experience does have a bearing on the kind of interaction displayed during the experiment, but not on the ability to interact itself. We have observed that every subject has a unique gestural vocabulary that is part of his/her own personal endowment, and that a device like MC makes it possible to evidence, make explicit and visualize such a vocabulary.
Alternative digital instruments, unlike acoustic ones, exhibit a certain degree of indeterminacy that stems from their design, based on a non-unequivocal mapping, or from the kind of reductionism that results from using data as raw material. Some degree of randomness can be highly interesting for participants to be able to explore new modes of interaction, thus disclosing a certain amount of invisible learning as well as hidden skills that would not be available for exploration in a fully causal environment.
Fields and Tonality provide complementary experiences. While Fields facilitates gestural feedback, body awareness and a broader and more diverse gestural scope, Tonality affords participants an aesthetic experience and further provides access to artistic performance for those who had been previously excluded from it. Interaction through a device like MC makes for a different experience that broadens the individuals’ potential for perception. The experience departs from a Cartesian, single-focus and fixed approach to perception by promoting an enactive, embodied perception that hinges on the connection between sound and the bodily-proprioception.
Notes
Address for correspondence: Alicia Peñalba, University of Valladolid, Facultad de Educación de Palencia, Avenida de Madrid, 44, CP 34004, Palencia (Palencia).
Email: apenalba@mpc.uva.es.
Acknowledgements:attendance to this conference has been sponsored by IMM Holding Group, and the research is part of the European project METABODY developed with the support of the Culture Programme of the European Union. We would like to thank Jaime del Val, the coordinator of METABODY for the opportunity to participate in this research initiative. We are also grateful to IMM Holding Group, and more particularly to Prof. Detlev Müller, for helping with the expenses involved in attending the conference. We are likewise thankful to the whole team behind MotionComposer, especially Robert Wechsler, for the loan of one of their devices for experimentation purposes and for the fruitful exchange of ideas. We wish to acknowledge the collaboration of the following Valladolid-based institutions: Museo Patio Herreriano, Facultad de Educación y Trabajo Social at the University of Valladolid, Centro Obregón de Parálisis Cerebral, Centro El Pino and Colegio Público Antonio Allúe Morer. MC has been developed by Robert Wechsler (based on an idea by Alicia Peñalba), infomus.org (motion tracking software), imm-gruppe.de (industrial product development) and composers Andreas Bergsland, Andrea Cera, Giacomo Lepri, Marcello Lussana, Pablo Palacio and Ives Schachtschabel.
References
Bergsland, A. & Wechsler, R. (2013). Movement-Music Relationships and Sound Design in MotionComposer, an Interactive Environment for Persons with (and without) Disabilities. Proceedings of re-new, Copenhagen 2013 (pp. 56-62).
Del Val, J. (2009). Cuerpo común y guerra de los afectos. Coreografías globales y cuerpos en serie del Afecto capital. Cuadernos de Información y Comunicación, 14, 121-139.
Del Val, J. (2014). Microtimes: Towards a Politics of Indeterminacy. Performance Research: A Journal of the Performing Arts, 19(3), 144-149.
Kontogeorgakopoulos, A., Wechsler, R., & Keay-Bright, W. (2014). Camera-Based Motion Tracking and Performing Arts for Persons with Motor Disabilities an Autism. In G. Kouroupetroglou (Ed.), Disability Informatics and Web Accessibility for Motor Limitations (pp. 294-322). Hersey, PA: IGI Global book series Advances in Medical Technologies and Clinical Practice (AMTCP).
Lang, P. J. (1980). Behavioral treatment and bio-behavioral assessment: computer applications. In Sidowski, J. B., Johnson, J. H. & Williams T. A. (Eds.), Technology in mental healthcare delivery systems (pp. 119-137). Norwood, NJ: Ablex.
Peñalba, A. (2008). El cuerpo en la interpretación musical: un modelo basado en las propiocepciones en la interpretación de instrumentos acústicos, hiperinstrumentos e instrumentos alternativos (Doctoral Dissertation, University of Valladolid). http://uvadoc.uva.es/handle/10324/55
Peñalba, A., & Wechsler, R. (2010). Danza interactiva con niños con parálisis cerebral. In XXVII Congreso de la Asociación Española de Logopedia, Foniatría y Audiología, 129-137. http://www.palindrome.de/content/pubs/CongresoAelfa.pdf
Wanderley, M., & Depalle, D. (2004). Gestural Control of Sound Synthesis. In Proceedings of the IEEE, 92(4), 632-644.