Tangible Scores

By Enrique Tomás (Post-doc Researcher) , Martin Kaltenbrunner (Supervisor) , 2014

PhD Thesis Enrique Tomás “The Interface Score”

supervised by Prof. Dr. Martin Kaltenbrunner and Prof.Dr. Gerhard Eckel

This artistic thesis investigates the artistic and philosophical consequences of understanding musical instruments as artifacts embodying a score. In other words, it explores the extent to which musical instruments can be perceived as musical scores. This artistic research examines the cultural and technological nature of digital musical instruments and it contributes to a material account of musical notation. It introduces a novel paradigm for tangible interface design called the “tangible score”. It also describes a practice-based research journey performing tangible musical interfaces.

A “Tangible Score” is a tactile interface for musical expression that incorporates a score in its physical shape, surface structure or spatial configuration. Using sound as a continuous input signal, both synthesis and control are available simultaneously through direct manipulation on the engraved patterns of the physical score. Every interface is conceived from a different graphical score that still represents a musical idea but it has been also specially designed for providing a diverse palette of acoustic signals when touched. But more important, the tactile scores define and propose specific gestural behaviors due to the different affordances and constraints of the object in front. Sound is generated through a polyphonic concatenative synthesis driven by a real-time analysis and classification of input signal spectra. Each of the scores is loaded with a specific sound corpus that defines its sonic identity. Thus, “Tangible Score” provides a implicit visual and haptic feedback in addition to its sonic core functionality, making it intuitive and learnable but as well suitable as an interface for musical improvisation and sonic exploration.

Concept and Technology behind Tangible Scores
Watch tangible scores concept in HD on Vimeo.

But, in which ways is it possible to embody musical works into musical instruments? ‘A Moment of Transition’ is a live electronics work performed with a “tangible score” whose form has been inspired by certain sound gestures. Through the ‘action of the hand’ Tomás’ explores the tactile patterns engraved on the instrument. A machine learning system clusters and recognizes tactile and sonic contents enabling real-time manipulation of digital information previously stored and analyzed in the computer. A generative lighting system augments the paper’s physical materiality while it also inscribes a visual narrative inspiring musical forms. The result of this physical encounter is a live performance where sound, light and interface become both musical notation and instrument.

A Moment of Transition (Trailer Version) is the last iteration of the project Tangible Scores.

This performance shows us how much our body is the central actor empowering those ‘moments of transition’ which transform artistic intention into sonic outcome. Our bodies, embedded in a perpetual action and perception loop, explore physical materiality and inform us about the world. But our bodies are noisy. They do not only enable performance but they also sculpt our ´being-in-the-world´. In ‘A Moment of Transition’ interface´s materiality becomes physical notation provoking performace. Tomás´ noisy hands, freely exploring the affordances of this system, finally produce the sound work we perceive.

Technically, touch sensors and contact microphones are used to capture tactile activity on the paper which has been embossed with a CNC. This information is chuncked and analyzed in real time extracting mathematical descriptors (Mel Frequency Cepstral Coefficients). This information is used to drive a machine learning system resynthesizing captured gesturality into new sound. This is achieved after ordering and concatenating chunks of a digital sound archive previously analyzed and clustered by the same algorithm.

(Produced with the support of ENCAC Network)

A Moment of Transition (Resumed Version) is the last iteration of the project Tangible Scores. A 20 minutes sound performance shows us how much our body is the main actor enabling the ‘moment of transition’ between artistic intentions and perceived sonic outcomes.

This is a thesis in art. Opposed to a thesis on art, such as art history, my research did not have any prescribed methodology. In fact, my artistic production during this PhD may be considered both object and method. As an artist working with technology it was mandatory contextualizing my personal artistic methods. In consequence, this thesis gives answers to the question of how artists can engage artistic research with the field of human-computer interaction (HCI). I defend that the way artists can help HCI is adopting a critical attitude with our research medium -namely tangible and musical interaction design in this PhD- avoiding the instrumentalization of our artistic processes and adopting the format of artistic research.

The notion of the “inherent score” drives the beginning of my research on musical notation. I define it as the material and virtual elements of the instrument shaping and inspiring a player’s performance. This notion gives us language to describe how the instrument mediates embodiment at the exact moment of performance. This is coherent with many improvisers’ intuition affirming that the instrument is the score of what they play. I propose that it is possible to design new interfaces for musical expression emphasizing their “inherent score”.

Watch live from the basement in HD on Vimeo.

For exemplifying this option, I have built and performed a series of digital musical instruments called “Tangible Scores”. They are the center of my artistic practice during this doctorate.This thesis contributes to a vision of digital musical instruments as symbolic machines. In my opinion,the principal characteristic of digital music instruments is their symbolic apparatus. The apparatus always features a program defining all possible musical realizations. For dealing with the complexities of the apparatus, designers have to deal with representations. For this reason, the representational dimension of musical interfaces has been addressed in this thesis. I present a vision about musical interfaces as a collection of related texts and symbols with a given arbitrary meaning. As I show in this thesis, they radically mediate our embodiment with the instrument. For this reason, I contribute towards a non-linguistic and non-representational approach to performing with musical interfaces.

Materiality is another important field of study in this thesis. I contribute to a materialistic accountof musical interfaces defending that matter must not be only understood as the mere support for an interface’s functionality. My thesis asserts that matter is an active agent able to inspire, provoke and shape performances. Therefore, my research aims at developing a new materialist sensibility at tangible and musical interaction design.Finally, this thesis addresses the cultural and political dimensions of musical interface design. I defend that the cultural always re-configures and redefines the technical in musical interface design. Politically, musical interfaces are never transparent technologies. A full range of cultural and political values are scripted into our musical interfaces. They are good examples of cultural interfaces.

Watch tangible scores installation in HD on Vimeo.

In conclusion, this is a thesis dedicated to the study of the issues of musical interface design from an artistic standpoint. The field of study is highly interdisciplinary and cultural studies, technology and design-among many other disciplines- merged at my practice. For this reason, the background of this thesis is art,music, engineering and philosophy.

Lecture @IRCAM: Towards Non-Linguistic Writing for Music – A Performative Approach (New Notations Symposium)

https://medias.ircam.fr/embed/media/x003dae

Some photos of Tangible Scores: