Enhancing Musical Experiences for the Hearing-Impaired

Consider the kinds of musical behaviours that typical non-musically trained listeners with normal hearing engage in as part of everyday life. Such listeners can tap their foot or otherwise move rhythmically in response to a musical stimulus. They can quickly articulate whether the piece of music is in a familiar style, and whether it is a style they like. If they are familiar with the music, they might be able to identify the composer and/or performers. The listeners can list instruments they hear playing. They can immediately assess stylistic and emotional aspects of the music, including whether or not it is loud, complicated, sad, fast, soothing, or generates a feeling of anxiety. They can also make complicated socio-cultural judgments, such as suggesting a friend who would like the music, or a social occasion for which it is appropriate.

Now, if the listeners are hearing-impaired, what would their musical behaviour be? Partial or profound lack of hearing makes the other ways humans use to sense sound in the environment much more important for the deaf than for people with normal hearing. Sound transmitted through the air and through other physical media such as floors, walls, chairs and machines act on the entire human body, not just the ears, and play an important role in the perception of music and environmental events for all people, but in particular for the deaf. In fact, it has been found that some deaf people process vibrations sensed via touch in the part of the brain used by other people for hearing ( Brain helps deaf enjoy music). This provides one possible explanation for how deaf musicians can sense music, and how deaf people can enjoy concerts and other musical events. These findings suggest that a mechanism to physically 'feel' music might provide an experience to a hearing impaired person that is qualitatively similar to the experience a normal hearing person has while listening to music. However, little research has specifically addressed the question of how to optimise a musical experience for a deaf person. This research focus on designing and evaluating a system to enhance the musical experience for the deaf.

A prototype system has been developed to provide musical information via three senses: touch, vision and sound. A ‘Haptic Chair’ vibrates with the music and provides input via touch and bone conduction of sound; and a computer display generates real-time visual effects based on musical features. The combination helps people with hearing problems to sense beat, pitch, amplitude, key changes and even gives a stereo effect. User studies suggest this is a promising way of enhancing enjoyment of music for both the Deaf and people with normal hearing. This research project is the PhD thesis topic of Suranga Chandima Nanakkara, supervised by Assoc. Prof. Ong Sim Heng (NUS, Faculty Engineering, Dept. ECE), Dr. Elizabeth Taylor (Head MMRL, TMSI) and Assoc. Prof Lonce Wyse (NUS Faculty Arts & Social Sciences / IDMI). The prototype system, especially the Haptic Chair was so enthusiastically received by the Deaf community that it is possible this system might significantly change the way the deaf community experiences music.


A real- time visual effects display
A 3D model of the Haptic Chair