Musical Haptics

Front matter

Springer Series on Touch and Haptic Systems

Musical Haptics

EUROHAPTICS EHS

Preface

The two fields of haptics and music are naturally connected in a number of ways. As a matter of fact, sound is nothing more than the auditory manifestation of vibration. When attending a concert, we are reached not only by airborne acoustic waves but also by related vibratory cues conveyed through the air and solid media such as the floor and seats. Moving from the audience to the performance stage, it is thanks to a complex system of auditory—haptic interactions established between musicians and their instruments that the former can render subtle expressive nuances and develop virtuosic playing techniques, and that being at a concert is such a rewarding experience.

Whereas auditory research has since long addressed the musical scenario, research on haptics has only recently started to consider it. This volume aims to fill this gap by collecting for the first time state-of-the-art contributions from distinguished scholars and young researchers working at the intersection of haptics and music performance. It presents theoretical, empirical, and practical aspects of haptic musical interaction and perception, such as the role of haptics in music performance and fruition, and describes the design and evaluation of digital musical interfaces that provide haptic feedback.

The realization of this volume was originally encouraged by Prof. Manuel Ferre, following the successful organization of a scientific workshop on Musical Haptics by Stefano Papetti at the EuroHaptics 2016 conference. The workshop hosted some of the most renowned world experts in the field and fostered discussion, exchange, and collaboration to help address theoretical and empirical challenges in Musical Haptics research. It was, in a way, the crowning event of the project Audio-Haptic modalities in Musical Interfaces" (2014-2016), an interdisciplinary research funded by the Swiss National Science Foundation, which initiated an exploratory investigation on the role of haptics and the sense of touch in music practice.

‘http://p3.snf.ch/project-150107 (last accessed on Nov 27, 2017).

ix x Preface

The present volume primarily features contributions from presenters at the EuroHaptics workshop. Additional authors were invited based on their established activities and recent outstanding results. Mirroring the implicitly interdisciplinary nature of Musical Haptics, contributions come from a variety of scientific backgrounds, such as music composition and performance, acoustics, mechanical engineering, robotics, sound and music computing, music perception, and cognitive neuroscience, thus bringing diverse viewpoints on a number of common topics.

Following an introduction which sets out the scope, aims, and relevance of Musical Haptics, the volume comprises 12 contributed chapters divided into two parts. Part I examines the relevance of haptic cues in music performance and perception, discussing how they affect user experience and performance in terms of usability, functionality, and perceived quality of musical instruments. Part II presents engineering, computational, and design approaches and guidelines that have been applied to render and exploit haptic feedback in digital musical interfaces. The two parts are distinct yet complementary: studying the perception of haptics requires sophisticated rendering techniques; developing sophisticated rendering techniques for haptics requires a good understanding of its psychophysics. To help the reader, a glossary is included that gathers in one place explanations of concepts and tools recurring throughout the book.

Musical Haptics is intended for haptic engineers, researchers in human—computer interaction, music psychologists, interaction designers, musical instrument designers, and musicians who, for example, would like to gain insight into the haptic exchange between musicians and their instruments, its relevance for user experience, quality perception and musical performance, as well as practical guidelines for the use of haptic feedback in musical devices and other human-computer interfaces. It is hoped that the present volume will contribute towards a scientific foundation of haptic musical interfaces, even though not all aspects have been possible to take into account.

We thank the Institute for Computer Music and Sound Technology (ICST) at the Zurich University of the Arts (ZHdK) for funding the publication of the present volume in Open Access form, along with the Alexander von Humboldt Foundation for supporting C.S. through a Humboldt Research Fellowship. We are especially grateful to ICST Director German Toro-Peréz for his continuous support, as well as to Federico Avanzini and Federico Fontana for their precious organizational advice. Finally, we would like to thank all the authors for their valuable contribution to this book.

Zurich, Switzerland Stefano Papetti Berlin, Germany Charalampos Saitis December 2017

Contents

Stefano Papetti and Charalampos Saitis

Part I Musical Haptics: Interaction and Perception

2 Once More, with Feeling: Revisiting the Role of Touch in Sile O’Modhrain and R. Brent Gillespie

Vincent Hayward

Federico Fontana, Stefano Papetti, Hanna Järveläinen, Federico Avanzini and Bruno L. Giordano

5 The Role of Haptic Cues in Musical Instrument Quality Perception. oi oce FS od ode lad eee hoe ada ee Charalampos Saitis, Hanna Järveläinen and Claudia Fritz

6 A Functional Analysis of Haptic Feedback in Digital Musical Gareth W. Young, David Murphy and Jeffrey Weeter

Sebastian Merchel and M. Ercan Altinsoy

Part II Haptic Musical Interfaces: Design and Applications

8 The MSCI Platform: A Framework for the Design and James Leonard, Nicolas Castagné, Claude Cadoz and Annie Luciani

Glossary and Abbreviations

Force-Feedback Instruments for the Laptop Orchestra of

Louisiana e teaa A eee Be Si sed te educa we eile, edd

Edgar Berdahl, Andrew Pfalz, Michael Blandino and Stephen David Beck

Design of Vibrotactile Feedback and Stimulation for Music

Performance si teed eo a he AER dd ets es

Marcello Giordano, John Sullivan and Marcelo M. Wanderley Haptics for the Development of Fundamental Rhythm Skills,

Simon Holland, Anders Bouwer and Oliver Hédl

M. Ercan Altinsoy and Sebastian Merchel

Implementation and Characterization of Vibrotactile

Stefano Papetti, Martin Fröhlich, Federico Fontana, Sébastien Schiesser and Federico Avanzini

Contents

Contributors

M. Ercan Altinsoy Institut fiir Akustik und Sprachkommunikation, Technische Universität Dresden, Dresden, Germany

Federico Avanzini Dipartimento di Informatica, Universita di Milano, Milano, Italy

Stephen David Beck School of Music & CCT—Center for Computation and Technology, Louisiana State University, Baton Rouge, LA, USA

Edgar Berdahl School of Music & CCT—Center for Computation and Technology, Louisiana State University, Baton Rouge, LA, USA

Michael Blandino School of Music & CCT—Center for Computation and Technology, Louisiana State University, Baton Rouge, LA, USA

Anders Bouwer Faculty of Digital Media and Creative Industries, Amsterdam University of Applied Sciences, Amsterdam, The Netherlands

Claude Cadoz ACROE—Association pour la Création et la Recherche sur les Outils d’Expression & Laboratoire [CA—Ingénierie de la Création Artistique, Institut polytechnique de Grenoble, Université Grenoble Alpes, Grenoble, France

Nicolas Castagné Laboratoire [CA—Ingénierie de la Création Artistique, Institut polytechnique de Grenoble, Université Grenoble Alpes, Grenoble, France

Federico Fontana Dipartimento di Scienze Matematiche, Informatiche e Fisiche, Universita di Udine, Udine, Italy

Claudia Fritz Equipe LAM—Lutheries-Acoustique-Musique, Institut Jean le Rond d’Alembert UMR 7190, Université Pierre et Marie Curie - CNRS, Paris, France

Martin Fröhlich I[CST—Institute for Computer Music and Sound Technology, Ziircher Hochschule der Kiinste, Zurich, Switzerland xiii xiv Contributors

R. Brent Gillespie Mechanical Engineering, University of Michigan, Ann Arbor, MI, USA

Bruno L. Giordano Institut de Neurosciences de la Timone UMR 7289, Aix-Marseille Université-Centre National de la Recherche Scientifique, Marseille, France

Marcello Giordano IDMIL—Input Devices and Music Interaction Laboratory, CIRMMT—Centre for Interdisciplinary Research in Music Media and Technology, McGill University, Montréal, QC, Canada

Vincent Hayward Sorbonne Universités, Université Pierre et Marie Curie, Institut des Systèmes Intelligents et de Robotique, Paris, France

Oliver Hédl Cooperative Systems Research Group, Faculty of Computer Science, University of Vienna, Vienna, Austria

Simon Holland Music Computing Lab, Centre for Research in Computing, The Open University, Milton Keynes, UK

Hanna Järveläinen [CST—Institute for Computer Music and Sound Technology, Ziircher Hochschule der Kiinste, Zurich, Switzerland

James Leonard Laboratoire [CA—Ingénierie de la Création Artistique, Institut polytechnique de Grenoble, Université Grenoble Alpes, Grenoble, France

Annie Luciani ACROE—Association pour la Création et la Recherche sur les Outils d’Expression & Laboratoire ICA—Ingénierie de la Création Artistique, Institut polytechnique de Grenoble, Université Grenoble Alpes, Grenoble, France

Sebastian Merchel Institut fiir Akustik und Sprachkommunikation, Technische Universität Dresden, Dresden, Germany

David Murphy University College Cork, Cork, Ireland

Sile O’Modhrain School of Information & School of Music, Theatre and Dance, University of Michigan, Ann Arbor, MI, USA

Stefano Papetti ICST—Institute for Computer Music and Sound Technology, Zürcher Hochschule der Künste, Zurich, Switzerland

Andrew Pfalz School of Music & CCT—Center for Computation and Technology, Louisiana State University, Baton Rouge, LA, USA

Charalampos Saitis Audio Communication Group, Technische Universitat Berlin, Berlin, Germany

Sébastien Schiesser [CST—Institute for Computer Music and Sound Technology, Zürcher Hochschule der Künste, Zurich, Switzerland

John Sullivan IDMIL—Input Devices and Music Interaction Laboratory, CIRMMT—Centre for Interdisciplinary Research in Music Media and Technology, McGill University, Montréal, QC, Canada

Contributors XV

Marcelo M. Wanderley IDMIL—Input Devices and Music Interaction Laboratory, CIRMMT— Centre for Interdisciplinary Research in Music Media and Technology, McGill University, Montréal, QC, Canada

Jeffrey Weeter University College Cork, Cork, Ireland Gareth W. Young University College Cork, Cork, Ireland

Chapter 1 A) Musical Haptics: Introduction get Stefano Papetti and Charalampos Saitis

Abstract This chapter introduces to the concept of musical haptics, its scope, aims, challenges, as well as its relevance and impact for general haptics and human-computer interaction. A brief summary of subsequent chapters is given.

1.1 Scope and Goals

Musical haptics is an emerging interdisciplinary field investigating touch and proprioception in music scenarios from the perspectives of haptic engineering, human-computer interaction (HCI), applied psychology, musical acoustics, aesthetics, and music performance.

The goals of musical haptics research may be summarized as: (i) to understand the role of haptic interaction in music experience and instrumental performance, and (ii) to create new musical devices yielding meaningful haptic feedback.

1.2 Haptic Cues in Music Practice and Fruition

Whenever an acoustic or electroacoustic musical instrument produces sound, that comes from its vibrating components (e.g., the reed and air column in a clarinet, or the strings and soundboard of a piano). While performing on such instruments, the haptic channel is involved in a complex action—perception loop: The player physically interacts with the instrument, on the one hand, to generate sound by injecting energy in

S. Papetti (È<)

ICST—Institute for Computer Music and Sound Technology, Zürcher Hochschule der Kiinste, Pfingsweidstrasse 96, 8005 Zurich, Switzerland

e-mail: stefano.papetti@zhdk.ch

C. Saitis

Audio Communication Group, Technische Universitat Berlin, Sekretariat E-N 8, Einsteinufer 17c, 10587 Berlin, Germany e-mail: charalampos.saitis @ campus.tu-berlin.de

© The Author(s) 2018 1 S. Papetti and C. Saitis (eds.), Musical Haptics, Springer Series on Touch and Haptic Systems, https://doi.org/10.1007/978-3-3 19-583 16-7_1

2 S. Papetti and C. Saitis the form of forces, velocities, and displacements (e.g., striking the keys of a keyboard, or bowing, plucking, and pressing the strings of a violin), and on the other hand receiving and perceiving the instrument’s physical response (e.g., the instrument’s body vibration, the kinematic of keys being depressed, the resistance and vibration of strings). One could therefore assume that the haptic channel supports performance control (e.g., timing, intonation) as well as expressivity (e.g., timbre, emotion). In particular, skilled performers are known to establish a very intimate, rich haptic exchange with their instruments, resulting in truly embodied interaction that is hard to find in other human-machine contexts. Through training-based learning of haptic cues and auditory-—tactile interactions, musicians develop highly precise auditory— motor skills [7, 28]. They then form a base of highly demanding users who expect top quality interaction (i.e., extensive control, consistent response, and maximum efficiency) with their instruments—tools that extends beyond mere performance goals to emotional and aesthetical outcomes.

In addition to what described above, both the performers and the audience are reached by vibration conveyed through air and solid media such as the floor and the seats of a concert hall. Those vibratory cues may then contribute to the perception of music (e.g., its perceived quality) and of instrumental performance (e.g., in an ensemble, a player could be able to monitor others’ performances also through such cues).

Music fruition and performance therefore present a well-defined framework in which to study basic psychophysical, perceptual, and biomechanical aspects of touch and proprioception, all of which may inform the design of novel haptic musical devices. There is now a growing body of scientific studies of music performance and perception from which to inform research in musical haptics, including topics and methods from the fields of psychophysics [19], biomechanics [11], music education [29], psycholinguistics [32], and artificial intelligence [20].

1.3 Musical Devices and Haptic Feedback

While current digital musical instruments (DMIs) usually offer touch-mediated interaction, they fall short of providing a natural physical experience to the performer. With afew exceptions, they lack haptic cues other than those intrinsically provided by their (passive) mechanics, if any (e.g., the kinematics of a digital piano keyboard)—in other words, their behavior is the same whether they are turned on or off. Such missing link between sound production and active haptic feedback, summed to the fact that even sophisticated sound synthesis cannot (yet?) compete with the complexity and liveliness of acoustically generated sound, generally makes the experience of performing on DMIs less rewarding and rich than playing traditional instruments. Try asking a professional pianist, especially a classically trained one, to play a digital piano and watch out! However, one could argue that establishing a rich haptic exchange between musicians and their digital tools would enhance performance control, expressivity, and user experience, while the music listening experience would be improved by conveying audio-related vibratory cues to the listener. Indeed, a recently renewed

1 Musical Haptics: Introduction 3 interest in advancing haptic interaction design for everyday intelligent interfaces— shared across the HCI and engineering communities, as well as the consumer electronics industry—promotes the idea that haptics has the potential to greatly improve usability, engagement, learnability, and the overall experience of the user, moreover with minimal or no requirements for constant visual attention [15, 17]. For example, haptic feedback is already used to improve robotic control in surgical teleoperation [27] and to increase realism and immersion in virtual reality applications [30].

With regard to applications, haptic musical interfaces may provide feedback on the performance itself or on various musical processes (e.g., representing a score). In addition to enhancing performance control and expressivity, they have a high potential as tools for music tuition, for providing guidance in (intrinsically noisy) large ensembles and remote performance scenarios, and for facilitating access to music practice and fruition for persons affected by somatosensory, visual, and even hearing impairments [6, 13, 21]. A notable example is: The virtuoso and profoundly deaf percussionist Evelyn Glennie explained her use of vibrotactile cues in musical performance, to the point of recognizing the pitch, based on where the vibrations are felt on her body [10]. A further potential application of programmable haptic feedback in musical interfaces is to offer a way of prototyping the mechanical response of components found in traditional instruments (e.g., the kinematics and vibratory behavior of a piano keyboard), thus saving time and lowering production costs, as opposed to traditional hardware development.

Some efforts were made in recent years to define a systematic approach for the design of haptic DMIs and to assess their utility [3, 9, 23]. Some of the developed prototypes simulate the haptic behavior of existing acoustic or electroacoustic instruments, while others implement new paradigms not necessarily linked to traditional instruments. Early examples of haptic musical interfaces consist in piano-like keyboards with computer-driven mechanical feedback for simulating touch responses of various keyboard instruments (e.g., harpsichord, organ, piano) [4, 8]. More recently, a haptic system using magneto-rheological technology was developed that could reproduce the dynamic behavior of piano keyboards [16]. A vibrotactile feedback system for open-air music controllers, based on an actuated ring or a feet stimulator, was proposed in [31]. Haptic DMIs inspired by traditional instruments (violin, woodwinds, monochord, and slide whistle) are described in [2, 18, 22]. In [26], actuators were used on acoustic and electroacoustic instruments to feed mechanical energy back and induce or dampen resonances.

Only a few commercial examples of haptic musical devices are currently found. The Yamaha AvantGrand! series of digital pianos embed vibration transducers simulating the effect of vibrating strings and soundboard, and pedal depression. The system can be turned on or off, and vibration intensity adjusted. The Ultrasonic Audio Syntact” is a midair musical interface that performs hand-gesture analysis by means of a camera, and provides tactile feedback at the hand through an array of

"https://europe.yamaha.com/en/products/musical_instruments/pianos/avantgrand/ (last accessed on Dec 7, 2017).

*http://www.ultrasonic-audio.com/products/syntact.html (last accessed on Dec 7, 2017).

4 S. Papetti and C. Saitis ultrasonic transducers. The Soundbrenner Pulse? is a wearable vibrotactile metronome. The Loflet Basslet* and Subpac° are wearable low-frequency vibration transducers (tactile subwoofers), respectively, in the form of a bracelet and a vest, whose goal is to enhance the music listening experience.

1.4 Challenges

Research in musical haptics faces several challenges, some of which are common to haptic engineering and HCI in general.

From a technology viewpoint, the use of sensors and actuators can be especially problematic because haptic musical interfaces should generally be compact and unobtrusive (to allow for seamless interaction), efficient in terms of power (so they can be compatible with current consumer electronics industrial processes), and offer high fidelity/accuracy (to enable sensing subtle gestures and rendering complex haptic cues). Musical haptics would then gain from further developments in sensing and actuator technology in those directions.

From the perspective of HCI and psychophysics, the details of how the haptic modality is actually involved and exploited while performing with traditional musical instruments or while listening to music are still largely unknown. More psychophysical evidence and behavioral evidence are needed to establish the biomechanics of touch and how haptic cues affect measurable performance parameters such as accuracy in timing, intonation, and dynamics, as well as to better understand the role of vibration in idiosyncratic perceptions of sound/instrument quality by performers and music/sound aesthetics by listeners.

What is more, haptic musical interfaces are interactive systems that require rigorous user experience evaluation to help define optimal configurations between perceptual effects and limitations on the one hand, and technological solutions on the other [5, 12, 33]. Despite the fact that several evaluation frameworks have been proposed [14, 24, 34], the evaluation of digital musical devices and related user experience currently suffers from a lack of commonly accepted goals, criteria, and methods [1, 25].

1.5 Outline

The first part of the book presents theoretical and empirical work in musical haptics with particular emphasis on biomechanical, psychophysical, and behavioral aspects of music performance and music perception. Chapter 2 redefines, with an original perspective, the biomechanics of the musician—instrument interaction as a tight

3http://www.soundbrenner.com (last accessed on Dec 23, 2017). 4+https://lofelt.com/ (last accessed on Dec 7, 2017).

Shttp://subpac.com/ (last accessed on Dec 23, 2017).

1 Musical Haptics: Introduction 5 dynamic coupling, rather than the mere interaction of two separate entities. Chapter 3 introduces basic concepts and functions related to the anatomy and physiology of the human somatosensory system with special focus on the perception of touch, pressure, vibration, and movement. Chapter 4 reports experiments investigating vibrotactile perception in finger-pressing tasks and while performing on the piano. Chapter 5 examines the role of vibrotactile cues on the perception of sound/instrument quality from the perspective of the musician, based on recent psycholinguistic and psychophysical evidence from violin and piano studies. Chapter 6 reports an experiment that uses quantitative and qualitative HCI evaluation methods to assess how various types of haptic feedback on a DMI affect aspects of functionality, usability, and user experience. Chapter 7 considers a music listening scenario for different musical genres and tests how body vibrations—generated from the original audio signal using a variety of approaches—influence the musical experience of the listener.

The second part of the volume presents design examples, applications, and evaluations of haptic musical interfaces. Chapter 8 describes an advanced hardware— software system for real-time rendering of physically modeled virtual instruments that can be played with force feedback, and its use as a creative artistic tool. Chapter 9 examines hardware and computing solutions for the development of haptic force-feedback DMIs through a case study of music compositions for the Laptop Orchestra of Louisiana. Chapter 10 proposes and evaluates the design of a taxonomy of vibrotactile cues and a stimulation system consisting in wearable garments for providing information similar to a score during music performance. Chapter 11 reports a series of experiments investigating the design and evaluation of vibrotactile stimulation for learning rhythm skills of varying complexity, with a special emphasis on multi-limb coordination. Chapter 12 evaluates the use of touchscreen interfaces augmented with audio-driven vibrotactile cues in music production, focusing on performance, user experience, and the cross-modal effect of audio loudness on tactile intensity. Chapter 13 illustrates common vibrotactile actuators technology and provides three examples of audio-haptic interfaces iteratively designed through validation procedures that tested their accuracy in measuring user gesture and in delivering vibrotactile cues.

A glossary at the end of the book provides descriptions (including related abbreviations) of concepts and tools that are frequently mentioned throughout the volume, offering a useful background for those less acquainted with haptic and music technology.

1. Barbosa, J., Malloch, J., Huot, S., Wanderley, M.M.: What does ‘Evaluation’ mean for the NIME community? In: Proceedings of the Conference on New Interfaces For Musical Expression (NIME). Baton Rouge, LA, USA (2015)

2. Birnbaum, D.: The Touch Flute : Exploring Roles of Vibrotactile Feedback in Music Performance. McGill University, Canada, Tech. rep. (2003)

3. Birnbaum, D.M., Wanderley, M.M.: A systematic approach to musical vibrotactile feedback. In: Proceedings of the International Computer Music Conference (ICMC), Copenhagen, Denmark (2007)

S. Papetti and C. Saitis

. Cadoz, C., Lisowski, L., Florens, J.L.: A modular feedback keyboard design. Comput. Music

. El Saddik, A., Orozco, M., Eid, M., Cha, J.: Haptics Technologies. Springer Series on Touch and Haptic Systems. Springer, Berlin Heidelberg, Berlin, Heidelberg, Germany (2011)

. Friedman, N., Chan, V., Zondervan, D., Bachman, M., Reinkensmeyer, D.J.: MusicGlove:

motivating and quantifying hand movement rehabilitation by using functional grips to play music. In: Proceedings of the International Conference of the IEEE Engineering in Medicine and Biology Society (EMBS), pp. 2359-2363. Boston, MA, USA (2011)

. Gabrielsson, A.: The Performance of Music. Academic Press, Cambridge, MA, USA (1999) . Gillespie, B.: The Touchback Keyboard. In: Proceedings of the International Computer Music

Conference (ICMC) (1992)

. Giordano, M., Wanderley, M.M.: Perceptual and technological issues in the design of vibrotactile-augmented interfaces for music technology and media. Lect. Notes Comput. Sci. 7989, 89-98 (2013)

. Glennie, E.: Hearing Essay (2015). https://www.evelyn.co.uk/hearing-essay/ . Goebl, W., Palmer, C.: Temporal control and hand movement efficiency in skilled music per-

. Hatzfeld, C., Kern, T.A. (eds.): Engineering Haptic Devices. Springer Series on Touch and

Haptic Systems. Springer, London, London, UK (2014)

. Israr, A., Bau, O., Kim, S.C., Poupyrev, I.: Tactile feedback on flat surfaces for the visually impaired. In: CHI’ 12 Extended Abstracts on Human Factors in Computing Systems, vol. 1571. ACM (2012)

. Kiefer, C., Collins, N., Fitzpatrick, G.: HCI methodology for evaluating musical controllers:

a case study. In: Proceedings of the Conference on New Interfaces for Musical Expression (NIME), pp. 87-90. Genoa, Italy (2008)

Lévesque, V., Oram, L., Maclean, K., Cockburn, A., Marchuk, N.D., Johnson, D., Colgate, J.E., Peshkin, M.A.: Enhancing physicality in touch interaction with programmable friction. In: Proceedings of the CHI 11 Conference on Human Factors in Computing Systems, pp. 2481-2490. ACM (2011)

Lozada, J., Hafez, M., Boutillon, X.: A novel haptic interface for musical keyboards. In: IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM), pp. 1—6. Zurich, Switzerland (2007)

MacLean, K.E.: Haptic interaction design for everyday interfaces. Rev. Human Fact. Ergonom. 4(1), 149-194 (2008)

Marshall, M.T., Wanderley, M.M.: Vibrotactile feedback in digital musical instruments. In: Proceedings of the Conference on New Interfaces for Musical Expression (NIME), pp. 226- 229. Paris, France (2006)

Merchel, S.: Auditory-tactile music perception. Shaker Verlag, Aachen, Germany (2014) Miranda, E.R. (ed.): Readings in Music and Artificial Intelligence. Routledge, New York and London (2000)

Nanayakkara, S., Taylor, E., Wyse, L., Ong, S.H.: An enhanced musical experience for the deaf: design and evaluation of a music display and a haptic chair. In: Proceedings of the CHI’ 09 Conference on Human factors in Computing Systems, pp. 337-346. ACM, New York, NY, USA (2009)

Nichols, C.: The vBow: development of a virtual violin bow haptic human-computer interface. In: Proceedings of the Conference on New Interfaces for Musical Expression (NIME), pp. 1—4. Dublin, Ireland (2002)

O’Modhrain, S.: Playing by feel: incorporating haptic feedback into computer-based musical instruments. Ph.D. thesis, CCRMA, Music Department, Stanford University, Stanford, CA, USA (2000)

O’Modhrain, S.: A framework for the evaluation of digital musical instruments. Comput. Music J. 35(1), 28-42 (2011)

Orio, N., Wanderley, M.M.: Evaluation of input devices for musical expression: borrowing tools from HCI. Comput. Music J. 26(3), 62—76 (2002)

Musical Haptics: Introduction 7

Overholt, D., Berdahl, E., Hamilton, R.: Advancements in actuated musical instruments. Organised Sound 16(02), 154-165 (2011)

Pacchierotti, C.: Cutaneous Haptic Feedback in Robotic Teleoperation. Springer, Springer Series on Touch and Haptic Systems (2015)

Palmer, C.: Music performance. Annu. Rev. Psychol. 48, 115-138 (1997)

Parncutt, R., McPherson, G.E. (eds.): The Science and Psychology of Music Performance: Creative Strategies for Teaching and Learning. Oxford University Press, New York, USA (2002)

Peer, A., Giachritsis, C.D. (eds.): Immersive Multimodal Interactive Presence. Springer, Springer Series on Touch and Haptic Systems (2012)

Rovan, J., Hayward, V.: Typology of tactile sounds and their synthesis in gesture-driven computer music performance. In: Wanderley, M., Battier, M. (eds.) Trends in Gestural Control of Music, pp. 297-320. IRCAM, Paris, France (2000)

Saitis, C., Fritz, C., Scavone, G.P., Guastavino, C., Dubois, D.: Perceptual evaluation of violins: A psycholinguistic analysis of preference verbal descriptions by experienced musicians. J. Acoust. Soc. Am. 141(4), 2746-2757 (2017)

Samur, E.: Performance Metrics for Haptic Interfaces. Springer Series on Touch and Haptic Systems. Springer, London, London, UK (2012)

Young, G.W., Murphy, D.: HCI Models for Digital Musical Instruments: Methodologies for Rigorous Testing of Digital Musical Instruments. In: Proceedings of the International Symposium of Computer Music Multidiscal Research (CMMR). Plymouth, UK (2015)

Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter’s Creative

Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Part I Musical Haptics: Interaction and Perception

Chapter 2 A) Once More, with Feeling: get Revisiting the Role of Touch

in Performer-Instrument Interaction

Sile O’Modhrain and R. Brent Gillespie

Abstract The dynamical response of a musical instrument plays a vital role in determining its playability. This is because, for instruments where there is a physical coupling between the sound-producing mechanism of the instrument and the player’s body (as with any acoustic instrument), energy can be exchanged across points of contact. Most instruments are strong enough to push back; they are springy, have inertia, and store and release energy on a scale that is appropriate and well matched to the player’s body. Haptic receptors embedded in skin, muscles, and joints are stimulated to relay force and motion signals to the player. We propose that the performer-instrument interaction is, in practice, a dynamic coupling between a mechanical system and a biomechanical instrumentalist. We take a stand on what is actually under the control of the musician, claiming it is not the instrument that is played, but the dynamic system formed by the instrument coupled to the musician’s body. In this chapter, we suggest that the robustness, immediacy, and potential for virtuosity associated with acoustic instrument performance are derived, in no small measure, from the fact that such interactions engage both the active and passive elements of the sensorimotor system and from the musician’s ability to learn to control and manage the dynamics of this coupled system. This, we suggest, is very different from an interaction with an instrument whose interface only supports information exchange. Finally, we suggest that a musical instrument interface that incorporates dynamic coupling likely supports the development of higher levels of skill and musical expressiveness.

S. O’Modhrain (X4)

School of Information & School of Music, Theatre and Dance, University of Michigan, 2051 Moore Building, 1100 Baits Dr, MI 48109-2085

Ann Arbor, MI, USA

e-mail: sileo@umich.edu

R. B. Gillespie

Mechanical Engineering, University of Michigan, 3450 GG Brown Building, 2350 Hayward Street, MI 48109-2525 Ann Arbor, MI, USA

e-mail: brentg@umich.edu

© The Author(s) 2018 11 S. Papetti and C. Saitis (eds.), Musical Haptics, Springer Series on Touch and Haptic Systems, https://doi.org/10.1007/978-3-319-58316-7_2

The mechanics of a musical instrument’s interface—what the instrument feels like—determines a great deal of its playability. What the instrument provides to be held, manipulated by mouth or hand, or otherwise controlled has obvious but also many subtle implications for how it can be used for musical expression. One means to undertake an analysis of playability and interface mechanics is in terms of the mechanical energy that is exchanged between a player’s body and the instrument. For acoustic instruments, mechanical energy injected by the player is transformed into acoustic energy through a process of resonance excitation. For electronic instruments, electrical energy is generally transformed into acoustic energy through a speaker, but controlled by interactions involving the player’s body and some physical portion of the instrument.

Importantly, there exists the possibility for mechanical energy stored in the physical part of the instrument to be returned to the player’s body. This possibility exists for both acoustic and electronic instruments, though in acoustic instruments it is in fact a likelihood. This likelihood exists because most acoustic instruments are strong enough to push back; they are springy, have inertia, and store and return energy on a scale that is roughly matched to the scale at which the player’s body stores and returns energy. Given that energy storage and return in the player’s body is determined by passive elements in muscle and tissues, one can say that the scale at which interface elements of the instrument are springy and have mass is similar to the scale at which muscles and tissues of the player are springy and have mass. That is, the mechanics of most acoustic instruments are roughly impedance matched to the biomechanics of the player’s body. Impedance matching facilitates the exchange of energy between passive elements within the instrument and passive elements that are part of the biomechanics of the player. Thus the player’s joints are moved or backdriven by the instrument, muscle stiffness is loaded, and the inertial dynamics of body segments are excited. In turn, haptic receptors embedded in skin, muscles, and joints are stimulated and relay force and motion signals to the player. It is also no accident that the parts of the body that interact with instruments—lips, fingers, hands—are the most highly populated by haptic receptors.