Tangible Signals – Physical Representation of Sound and Embodied Control Feedback

Music brings joy, enables artistic self-expression and stimulates human connections. The research project Tangible Signals explores the physicality of computer music and digital music-making in the context of accessibility and assistive technologies for visually impaired and blind musicians. Using artistic research methods, the project aims to overcome the limitations of vision-centric music software by coupling digital data with the physical domain, creating a more tactile and accessible physical music computing experience. In collaboration with visually impaired and blind experts from the Institute for the Blind in Vienna, this research project investigates whether tangible musical interaction as a human-computer interaction (HCI) modality is a suitable approach towards accessible digital music-making and whether it can inspire creativity and enhance artistic stage performance.

Research Questions

With regard to tangible musical interaction the research project attempts to answer two main research questions in the context of computers as artistic tools and stage-based live performances of visually impaired or blind musicians:

• What are the benefits of tangible musical interaction for visually impaired and blind musicians?
• How does tangible musical interaction contributes to the artistic expression and musical performance of visually impaired and blind musicians?

Accessible Music Software

During the initial workshops at the Institute for the Blind in Vienna (BBI) it became clear, that there is no simple accessible computer music software available, that interacts with the screen reader and can be used to create computer music and electronic patterns playfully in real time. In order to enable the visually impaired and blind pupils as well as to provide a platform for the research project, the simplified web-based live code environment Welle was developed.

Live demo at: https://welle.live

Interactive Physical Sound Displays

Based on the insights of the workshops at the BBI and in collaboration with visually impaired and blind experts, three tangible interfaces were developed, that use sensor technology in combination with mechanical actuators to provide an interactive physical display of sound and music data for people with visual impairments. The interfaces can be used either separately or combined as input and output devices utilizing pin-based, string-based and wheel-based interaction elements.

Demonstration of Sonic Interactions at TEI ‘21

Tangible Signals Interfaces

using the tangible musical interfaces for live composition with custom software

Workshops and Performances

In order to investigate tangible musical interaction and accessibility for visually impaired and blind persons, multiple interviews, workshops and performances were conducted at the BBI as well as during the Austrian Computer Camp (OCC).

Findings & Results

The research project including overview on the research fields and background, documentation of the practical work and the findings is published in the thesis Tangible Signals – Physical Representation of Sound and Embodied Control Feedback (PDF). The results of the research project can be summarized as follows:

• The research project exposes shortcomings in the field of music software and accessibility for non-trained VIB musicians.
  
• The web-based music environment Welle was developed as accessible digital music platform
  
• Three tangible musical interfaces were developed in collaboration with visually impaired and blind experts
• Tangible musical interaction was explored in expert inquiries, workshops and stage performances as a means for accessible digital music-making for VIB musicians.
• The research highlights tangible musical interaction as an expressive and intuitive modality for performative musical expression for visually impaired and blind musicians, that enables intuitive interaction with sonic features and musical controls. The tactile use of the interfaces is perceived as joyful interaction and creates a playful musical experience, that stimulates curiosity and playful experimentation. Tangible musical interaction enables direct physical manipulation which it increases immediacy. The externalization of multiple tangible interfaces increases bandwidth and enables simultaneous access, collaboration, improvisation and spontaneous interaction.

– – – – – –

* This PhD project is funded with the DOC fellowship by the Austrian Academy of Sciences, starting June 2019.

* In collaboration with the Federal Institute of the Blind, Vienna.