The research project “Tangible Signals” investigates the physical representation and haptic feedback control of computer sound data.

From GUI to TUI

The creation and manipulation of sounds is fascinating and challenging. The availability and ease of use of microcontrollers, sensors and easily accessible programming environments supported the transition from Graphical User Interfaces (GUI) to Tangible User Interfaces (TUI). But while many NIMEs and TUIs concentrate on enabling the creation and control of digital data, especially the control of musical parameters through the availability of tangible interaction, yet there are only a few interfaces and devices which are able to dynamically represent data in a physical way and at the same time offer the possibility to control or change this data representation and thus the data itself.

Physical Representation

The physical representation and modification of digital sound data including haptic feedback is the core of this research work Tangible Signals. The audience is at first visually impaired musicians who are limited in their work with GUIs, but also sighted musicians who want to integrate a physical level into their creative process. In addition, this research may also be of interest to other disciplines that benefit from physical representation and modification of data.

Research Questions:

• How can the physical representation of sound and haptic feedback control contribute to the process of sound generation, sound processing and artistic performance?
• How does the physical representation of sound and its haptic control feedback open up new inspiration and creativity in artistic-musical expression?
• How does the interaction with the physical representation of sound and its haptic control feedback compare to GUI-based workflows?

Interactive physical sound displays

Three tangible user interface prototypes were developed, that use sensor technology in combination with mechanical actuators to provide an interactive physical display of sound and music data for people with visual impairments. The interfaces can be used either separately or combined as input and output devices utilizing pin-based, string-based and wheel-based interaction elements.

ACCESSIBLE MUSIC SOFTWARE

For creating computer-based music with children and adults with VI, the simplified web-based live code environment WELLE was developed. WELLE serves also as a tool and interface platform for connecting the various tangible signal prototypes to computer generated sounds and sequences.

– – – – – –

* This PhD project is funded with the DOC fellowship by the Austrian Academy of Sciences, starting June 2019.

* In collaboration with the Federal Institute of the Blind, Vienna.