Sonic Interactions in Virtual Environments

Front matter

Michele Geronazzo Stefania Serafin Editors i Sonic Interactions in Virtual Environments

A Springer

Human-Computer Interaction Series

Preface

Sonic Interaction Design (SID) is the study and exploitation of sound being one of the principal channels conveying information, meaning, esthetic, and emotional qualities in interactive contexts. The field of Sonic Interactions in Virtual Environments (SIVE) extends SID to immersive media, i.e., virtual/augmented/mixed reality (XR). Considering a virtuality continuum, this book mainly focused on virtual reality (VR) also facing occasionally mixed and hybrid reality settings.

The basic and most obvious assumption that motivates this volume is: it is hard to live in a world without sound and it is hard in virtual environments (VE) too. VR without plausible and convincing sounds feels unnatural to users. Auditory information is a powerful omnidirectional source of learning for our interaction in real and virtual environments. The good news brought by this book is that VR finally sounds plausible. Advances in several fields are now able to provide an immersive listening experience that is perceptually indistinguishable from reality which means that immersive sounds could make interaction intrinsically natural. Auralization and spatial audio technologies play a fundamental role in providing immersion and presence in VR applications at an unprecedented level. The combination of recent developments in VR headsets and earables further strengthens the perceptual validity of multimodal virtual environments and experiences.

We can therefore promote a true audio-centered and audio-first design for VR with levels of realism and immersiveness that can even surpass the visual counterpart. Visuals, although rightly emphasized by many studies and products, are often not very effectively enhanced and strengthened by sound. The final result is a weakening of multisensory integration and the corresponding VR potentials that strongly determine the quality and durability of the experience.

The editors would like to identify two starting points in the past 10 years that have given rise and awareness to the SIVE research area and studies. The first episode is symbolic: we would like to anecdotally bring back from our memories the first meeting between us, the two editors of the book. The year was 2011, exactly 10 years ago. Michele had recently started his Ph.D. at the Sound and Music Computing Group of the Department of Information Engineering at the University of Padua, under the supervision of Dr. Avanzini. The Italian Association of Musical Informatics

Vii viii Preface

(AIMI) organized the workshop “Sound and Music Computing for Human-Computer Interaction” at the ninth edition of the Biannual Conference of the Italian ACM SIGCHI Chapter (CHItaly) at the beautiful Alghero in Sardinia in early September. A great period for the seaside.

Michele was asked to write his first conference paper to be presented at the workshop entitled “Customized 3D Sound for Innovative Interaction Design,” An article with a high-sounding title that promises a lot but provides little: in short, an article of which not to be proud. On the other hand, there were some valuable references to the egocentric audio perspective that will be formalized in the introductory chapter of this book. However, the reason why we tell this anecdote is that at his first presentation at a scientific conference for the Ph.D. student Michele Geronazzo, among the very small audience, there was Dr. Stefania Serafin. Ten years ago, we began to discuss issues that connected sonic interaction design with immersive 3D audio in VR. The AIMI president of that time failed to get the workshop’s contributions included in the official ACM CHlItaly proceedings despite a regular peer-review process. The poor Ph.D. student Michele found himself without an official publication, at his first conference, in an unknown scientific community. We like to think that at that event and with that meeting started something much more relevant and impactful: SIVE. We are here to give it a shape in this book edited and structured together.

Another temporal coincidence brings us to connect this story with the second and official starting point of this adventure. Michele’s unpublished conference paper was finally published within his doctoral thesis, defended in 2014, the year in which the IEEE Virtual Reality workshop series “Sonic Interactions in Virtual Environments (SIVE)” started (https://sive.create.aau.dk/). The mission of IEEE VR SIVE was to increase among the virtual reality community and junior researchers the awareness of the importance of sonic elements when designing immersive XR environments. However, we can also identify a certain degree of reciprocity while considering the fragmented nature and specificity of those studies aim at developing immersive XR environments for sound and music. First, we, therefore, refer to our beloved Sound and Music Computing (SMC) network, and then we consider the International Community for Auditory Display (ICAD), the Audio Engineering Society (AES), and the communities linked to the International Conference on New Interfaces for Musical Expression (NIME), the Digital Audio Effects (DAFX), and the Sonic Interaction Design COST Action (COST-SID IC601, ended in 2012). All these communities address aspects of the SIVE topics according to their specificities. No institutional nor contextual references that collect technological developments, best practices, and creative efforts related to the peculiarities of immersive VEs existed before the SIVE workshop. The book follows a similar philosophy trying to give an exhaustive view of those multidisciplinary topics already mentioned in our two recent reviews.! It features state-of-the-art research on real-time auralization, sonic

1 S, Serafin, M. Geronazzo, N. C. Nilsson, C. Erkut, and R. Nordahl, “Sonic interactions in virtual reality: state of the art, current challenges and future directions,’ IEEE Computer Graphics and Applications, vol. 38, no. 2, pp. 31—43, 2018.

S. Serafin et al., “Reflections from five years of Sonic Interactions in Virtual Environments workshops,” Journal of New Music Research, vol. 49, no. 1, pp. 24-34, Jan. 2020.

Preface ix interaction design in VR, quality of the experience in multimodal environments, and applications. We aim to provide an organized starting point on which to develop a new generation of immersive experiences and applications. Since the editors are aware of the very fast social transformation by the acceleration in the development of digital technologies, all chapters should be read as entry points. Future scenarios and solutions will necessarily evolve by combining emerging research areas such as artificial intelligence, ubiquitous and pervasive computing, quantum technologies, as well as continuous discoveries in the neuroscientific field and anthropological reflections on the authenticity of the experience in VR.

For this reason, contributing authors and editors include interdisciplinary experts from the fields of computer science, engineering, acoustics, psychology, design, humanities, and beyond. So that we can give to the reader a broad view and a clear introduction to the state-of-the-art technologies and design principles, and to the challenges that might be awaiting us in the future.

Through an overview of emerging topics, theories, methods, tools, and practices in sonic interactions in virtual environments research, the book aims to establish the basis for further development of this new research area. The authors were invited to contribute to specific topics according to their well-known expertise. They followed a predefined structure outlined by the editors.

The book is divided into four parts:

Part I, Introduction: this theoretical part frames the background and the key themes in SIVE. The editors address several phenomenological foundational issues intending to shape a new research field from an archipelago of studies scattered in different research communities.

Part II, Interactive and Immersive Audio: we cover the system requirement part with four chapters introducing and analyzing audio-related technological aspects and challenges. With some overlaps and connections, the four chapters deal with the plausibility of an immersive rendering able to tackle the computational burden. To do so, we deal with methods and algorithms for real-time rendering considering sound production, propagation, and spatialization, respectively. Finally, the reproduction and evaluation phase allows closing the development loop of new audio technologies.

Part III, Sonic Interactions: a sonic interaction design part devoted to emphasizing the peculiar aspects of sound in immersive media. In particular, spatial interactions are important where we would like to produce and transform ideas and actions to create meaning with VR, as well as the virtual auditory space is an information container that could be shaped by users. As the VR systems enter people’s lives, manufacturers, developers, and creators should carefully consider an embodied experience ready to share a common space with peers, collaboratively.

Part IV, Sonic Experiences: the last part focuses on multimodal integration for sonic experiences in VR with the help of several case studies. Starting from a literature review of multimodal experiments and experiences with sound, this last part offers some reflections on the concept of audio-visual immersion and audiohaptic integration able to form our ecology of everyday or musical sounds. Finally, the potentials of VR to transport artists and spectators into a world of imagination and x Preface unprecedented expression is taken as an exemplar of what multimodal and immersive experiences can elicit in terms of emotional and rational engagement.

In the following, a summary for each chapter is provided to help the reader to follow the proposed narrative structure.

Part I

Chapter 1 illustrates the editors’ vision of the SIVE research field. The main concept introduced here is the egocentric audio perspective in a technologically mediated environment. The listeners should be entangled with their auditory digital twins in a participatory and enacted exploration for sense-making characterized by a personalized and multisensory first-person spatial reference frame. Intra-actions between humans and non-human agents/actors dynamically and fluidly determine immersion and coherence of the experience, participatively. SID aims to facilitate the diffraction of knowledge in different tasks and contexts.

Part II

Chapter 2 addresses the first building block of SIVE, i.e., the modeling and synthesis of sound sources, focusing on procedural approaches. Special emphasis is placed on physics-based sound synthesis methods and their potential for improved interactivity concerning the sense of presence and embodiment of a user in a virtual environment.

In Chap. 3, critical challenges in auralization systems in virtual reality and games are identified, including progressing from modeling enclosures to complex, general scenes such as a city block with both indoor and outdoor areas. The authors provide a general overview of real-time auralization systems, their historical design and motivations, and how novel systems have been designed to tackle the new challenges.

Chapter 4 deals with the concepts of adaptation in a binaural audio context, considering first the adaptation of the rendering system to the acoustic and perceptual properties of the user, and second the adaptation of the user to the rendering quality of the system. The authors introduce the topics of head-related transfer function (HRTF) selection (system-to-user adaptation) and HRTF accommodation (user-to-system adaptation).

Finally, Chap. 5 concludes the second part of the book by introducing audio reproduction techniques for virtual reality, the concepts of audio quality, and quality of the experience in VR.

Preface xi

Part III

Chapter 6 opens the third part of the book devoted to SID within virtual environments. In particular, it deals with space, a fundamental feature of VR systems, and more generally, human experience. In this chapter, the authors propose a typology of VR interactive audio systems, focusing on the function of systems and the role of space in their design. Spatial categories are proposed to be able to analyze the role of space within existing interactive audio VR products.

Chapter 7 promotes the following great opportunities offered by VR systems: to bring experiences, technologies, and users’ physical and experiential bodies (soma) together, and to study and teach these open-ended relationships of enaction and meaning-making in the framework of soma design. In this chapter, the authors introduce soma design and focus on design exemplars that come from physical rehabilitation applied to sonic interaction strategies.

Then, Chap. 8 investigates how to design the user experience without being detrimental to the creative output, and how to design spatial configurations to support both individual creativity and collaboration. The authors examine user experience design for collaborative music-making in shared virtual environments, giving design implications for the auditory information and the collaborative facilitation.

Finally, Chap. 9 explores the possibilities in content creation like spatial music mixing, be it in virtual spaces or for surround sound in film and music, offered by the development of VR systems and multimodal simulations. Authors present some design aspects for mixing in VR, investigating existing virtual music mixing products, and creating a framework for a virtual spatial-music mixing tool.

Part IV

Chapter 10 helps the reader to understand how sound enhances, substitutes, or modifies the way we perceive and interact with the world. This is an important element when designing interactive multimodal experiences. In this chapter, Stefania presents an overview of sound in a multimodal context, ranging from basic experiments in multimodal perception to more advanced interactive experiences.

Chapter 11 focuses on audiovisual experiences, by discussing the idea of immersion, and by providing an experimental paradigm that can be used for assessing immersion. The authors highlight the factors that can influence immersion and they differentiate immersion from the quality of experience (QoE). The theoretical implications for conducting experiments on these aspects are presented, and the authors provide a case study for subjective evaluation after assessing the merits and demerits of subjective and objective measures.

Chapter 12 focuses on audio-haptic experiences, being concerned with haptic augmentations having effects on auditory perception, for example, about how different vibrotactile cues may affect the perceived sound quality. The authors xii Preface review the results of different experiments showing that the auditory and somatosensory channels together can produce constructive effects resulting in a measurable perceptual enhancement.

Finally, Chap. 13 examines the special case of virtual music experiences, with particular emphasis on the performance with Immersive Virtual Musical Instruments (IVMI) and the relation between musicians and spectators. The authors assess in detail the several technical and conceptual challenges linked to the composition of IVMI performances on stage (i.e., their scenography), providing a new critical perspective.

We hope the reader finds this book informative and useful for both research and practice with sound.

Udine, Copenhagen Michele Geronazzo September 2021 Stefania Serafin

Acknowledgements

We would like to thank all authors and people involved in the book for their time and effort. In particular, the co-organizers of the IEEE Virtual Reality Workshop SIVE participated, in different ways, in this book project. Special thanks to Helen Desmond (Springer Computer Science Editor) and the Springer team for allowing us to prepare this volume.

Michele: I take my place, I have been creating my place in SIVE. This book project closes my first 10 years of academic activities, It has allowed me to reflect on my path and interdisciplinary education, challenging my knowledge extraction process. I went through the three “HCI waves” with my own time: ergonomics and engineering in Padova, psychology, and cognition in Verona, embodied design and UX in Copenhagen. I found recognition and maturity in London, and, finally, Udine gave me the time to find my identity. I would like to thank all my mentors and peers who made me grow in the research jungle, on numerous occasions and at different moments in my life. A big thanks go to my beloved family who supports me and always brings me back down to earth.

New challenges in VR are on the horizon and I am ready to make resonate our audio perspective!

Stefania: This year I have been recognized with the Danish Sound Award, for being pivotal in securing Denmark’s role as a leader in fields such as sonic interaction design, sound, and music computing and developing the role of sound in international virtual reality research. This award and this book would not have been possible without the wonderful colleagues and students of the Multisensory Experience Lab at Aalborg University in Copenhagen, who keep me motivated and inspired on a daily basis. It was a pleasure to host Michele as a postdoc in the lab for 2 years, and this book is a result of that. The lab is my second family, that wonderfully complements my first beloved family, both in Italy and in Denmark, that I thank with all my heart for everything they mean to me.

xiii xiv Acknowledgements

List of expert readers: we thank the following researchers to be our first critical readers, providing valuable comments for specific chapters within their area of expertise:

Roberto Barumerli, Acoustics Research Institute, Vienna, Austria

Braxton Boren, American University, Washington, DC, US

Michele Ducceschi, University of Bologna, Bologna, Italy

Isaac Engel, Imperial College London, London, UK

Floriana Ferro, University of Udine, Udine, Italy

Amalia de Götzen, Aalborg University Copenhagen, Copenhagen, Denmark

Niels Christian Nilsson, Aalborg University Copenhagen, Copenhagen, Denmark

Dan Overholt, Aalborg University Copenhagen, Copenhagen, Denmark

Archontis Politis, Tampere University, Tampere, Finland

Sebastian Prepelita, Facebook Reality Labs, Redmond, WA, US

Giorgio Presti, University of Milano, Milano, Italy

Davide Rocchesso, University of Palermo, Palermo, Italy

Lauri Savioja, Aalto University, Espoo, Finland

Ana Tajadura-Jiménez, University College London, London, UK

Silvin Willemsen, Aalborg University Copenhagen, Copenhagen, Denmark

Acknowledgements XV

Open Access Funding

This book project was supported by e the Nordic Sound and Music Computing Network (NordicSMC)—University hub by Nordforsk (Norway),

e the Department of Humanities and Cultural Heritage—University of Udine (Italy) with the recognition of “Department of Excellence” by the Ministry of Education, University and Research (MIUR) of Italy that cover the majority of the open-access publishing costs. We are grateful to the EU project SONICOM (grant number: 101017743, RIA action of Horizon 2020) for its sponsorship that contributes to the full transition to open access.

a gia hi W) NordForsk AALBORG UNIVERSITY cae BE sone music conrcrins ct >a é y sonicom BEA

Part I Introduction

1 Sonic Interactions in Virtual Environments: Michele Geronazzo and Stefania Serafin

Part II Interactive and Immersive Audio

2 Procedural Modeling of Interactive Sound Sources in Virtual e EA ESEE TEPEE TE EEE EE IE ET Federico Avanzini

Nikunj Raghuvanshi and Hannes Gamper

4 System-to-User and User-to-System Adaptations in Binaural AUOM sscirsesirssoirosst ot oisean enaena eaa a Lorenzo Picinali and Brian F. G. Katz

Fabian Brinkmann and Stefan Weinzierl

Part II Sonic Interactions

6 Spatial Design Considerations for Interactive Audio in Virtual RESI cathe ap ee dap EA rR n n EAR TAREA Thomas Deacon and Mathieu Barthet

7 Embodied and Sonic Interactions in Virtual Environments: Tactics and Exemplars scccnie4okneeancenayiiatedengand Ene Sophus Béneé Olsen, Emil Rosenlund Hgeg, and Cumhur Erkut

8 Supporting Sonic Interaction in Creative, Shared Virtual Enyironmenis 25 sess docs seers ee bebe bore ber ecetiaebaneebeces Liang Men and Nick Bryan-Kinns xvii xviii

Anders Riddershom Bargum, Oddur Ingi Kristjánsson, Péter Babó, Rasmus Eske Waage Nielsen, Simon Rostami Mosen, and Stefania Serafin

Part IV Sonic Experiences

Audio in Multisensory Interactions: From Experiments to ENperionees: cc 24 55 366050548 ik aeda okip IRAE EERE

Stefania Serafin

Sarvesh R. Agrawal and Søren Bech

Augmenting Sonic Experiences Through Haptic Feedback ....

Federico Fontana, Hanna Järveläinen, and Stefano Papetti

From the Lab to the Stage: Practical Considerations on Designing Performances with Immersive Virtual Musical

Instruments ....6 csc sevdesceueseuduncudeanweeneaneanasaeees

Victor Zappi, Dario Mazzanti, and Florent Berthaut

Contents

Editors and Contributors

About the Editors

Michele Geronazzo Ph.D., is an Associate Professor at the University of Padova—Dept. of Management and Engineering, and part of the coordination unit of the EU-H2020 project SONICOM at Imperial College London. He received his M.S. degree in Computer Engineering (2009) and his Ph.D. degree in Information & Communication Technologies (2014) from the University of Padova. Between 2014 and 2021, he worked as an Assistant Professor in Digital Media at the University of Udine and a postdoctoral researcher at Imperial College London, Aalborg University, and the University of Verona in the fields of neurosciences and simulations of complex human-machine systems. His main research interests involve binaural spatial audio modeling and synthesis, virtual/augmented reality, and sound in human—computer interaction.

He is an IEEE Senior Member and part of the organizing committee of the IEEE VR Workshop on Sonic Interactions for Virtual Environments since 2015 (chair of the 2018 and 2020 editions). From September 2019, he has been appointed as an Editorial Board member for Frontiers in Virtual Reality, and he served as guest editor for Wireless Communications and Mobile Computing (John Wiley & Sons and Hindawi publishers, 2019). He is a co-recipient of six best paper/poster awards and co-author of more than 70 scientific publications. In 2015, his Ph.D. thesis was honored by the Acoustic Society of Italy (AIA) with the “G. Sarcedote” award.

XX

Editors and Contributors

Stefania Serafin is a Professor of Sonic interaction design at Aalborg University in Copenhagen and the leader of the Multisensory Experience Lab together with Rolf Nordahl. She was previously appointed as Associate Professor (2006-2013) and Assistant Professor (2003-2006) in the same University. She has been visiting researcher at the University of Cambridge and KTH in Stockholm (2003) and visiting professor at the University of Virginia (2002). Since 2014, she is the President of the Sound and Music Computing association, and since 2018, the Project Leader of the Nordic Sound and Music Computing network supported by Nordforsk. She has been a part of the organizing committee of the IEEE VR Workshop on Sonic Interactions for Virtual Environments since the first edition. She is also the coordinator of the Sound and music computing Master at Aalborg University. She received her Ph.D. entitled “The sound of friction: computer models, playability and musical applications” from Stanford University in 2004, supervised by Professor Julius Smith HI. She is the co-author of more than 300 papers in the fields of sound and music computing, sound for virtual and augmented reality, sonic interaction design, and new interfaces for musical expression.

Editors and Contributors xxi

Contributors

Agrawal Sarvesh R. Bang & Olufsen, Struer, Denmark; Technical University of Denmark, Department of Photonics Engineering, Lyngby, Denmark

Avanzini Federico Laboratory of Music Informatics, Department of Computer Science, University of Milano, Milano, Italy

Babó Péter Department of Architecture, Design, and Media Technology, Aalborg University Copenhagen, Copenhagen, Denmark

Barthet Mathieu Centre for Digital Music, Queen Mary University of London, London, United Kingdom

Bech Søren Bang & Olufsen, Struer, Denmark; Department of Electronic Systems, Aalborg University, Aalborg, Denmark

Berthaut Florent University of Lille, Lille, France

Brinkmann Fabian Audio Communication Group, Technical University of Berlin, Berlin, Germany

Bryan-Kinns Nick Queen Mary University of London, London, United Kingdom

Deacon Thomas Media and Arts Technology CDT, Queen Mary University of London, London, United Kingdom

Erkut Cumhur Multisensory Experience Lab, Aalborg University Copenhagen, Copenhagen, Denmark

Eske Waage Nielsen Rasmus Department of Architecture, Design, and Media Technology, Aalborg University Copenhagen, Copenhagen, Denmark

Fontana Federico Department of Mathematics, Computer Science and Physics, University of Udine, Udine, Italy

Gamper Hannes Microsoft Research, Redmond, USA

Geronazzo Michele Department of Engineering and Management, University of Padova, Padova, Italy;

Dyson School of Design Engineering, Imperial College London, London, UK; Department of Humanities and Cultural Heritage, University of Udine, Udine, Italy

Høeg Emil Rosenlund Multisensory Experience Lab, Aalborg University Copenhagen, Copenhagen, Denmark

Ingi Kristjánsson Oddur Department of Architecture, Design, and Media Technology, Aalborg University Copenhagen, Copenhagen, Denmark

Järveläinen Hanna Institute for Computer Music and Sound Technology, Zurich University of the Arts, Zurich, Switzerland xxii Editors and Contributors Katz Brian F. G. Sorbonne Université, CNRS, UMR 7190, Institut Jean Le Rond d’ Alembert, Paris, France

Mazzanti Dario Independent researcher, Genoa, Italy

Men Liang Liverpool John Moores University, Liverpool, United Kingdom

Olsen Sophus Béneé Multisensory Experience Lab, Aalborg University Copenhagen, Copenhagen, Denmark

Papetti Stefano Institute for Computer Music and Sound Technology, Zurich University of the Arts, Zurich, Switzerland

Picinali Lorenzo Imperial College London, London, UK Raghuvanshi Nikunj Microsoft Research, Redmond, USA

Riddershom Bargum Anders Department of Architecture, Design, and Media Technology, Aalborg University Copenhagen, Copenhagen, Denmark

Rostami Mosen Simon Department of Architecture, Design, and Media Technology, Aalborg University Copenhagen, Copenhagen, Denmark

Serafin Stefania Department of Architecture, Design, and Media Technology, Aalborg University Copenhagen, Copenhagen, Denmark

Weinzierl Stefan Audio Communication Group, Technical University of Berlin, Berlin, Germany

Zappi Victor Northeastern University, Boston, MA, United States