Sensory Substitution in Music: Enhancing Deaf Perception Through Vibrotactile Feedback

Abstract

Music experienced through vibrotactile interfaces is a method of perceiving musical elements through the sense of touch, often involving vibrations. This technology functions by converting audio signals into physical sensations that can be sensed through the skin, typically via a wearable device like a wristband. Beginning with an initial audio file devoid of tactile feedback, the procedure entails altering it through sinusoidal modeling and, if necessary, implementing a Space-Fixed Audio transformation by utilizing the Head-Related Transfer Function (HRTF). In this study, we successfully transformed sound files into tactile stereo vibrations using specialized hardware. This process was rigorously tested and validated through experimentation involving ten individuals. Our findings confirm that psychophysical sensations can indeed be perceptible. Notably, the most consistent responses were observed when applying the Vibrato and Tremolo effect, following an HRTF transformation. The Space-Fixed Audio transformation primarily introduced variations in azimuth, covering 360 in a clockwise direction. Consequently, this processing led to significant spectral changes, effectively rescaling and compressing the audio’s frequencies into lower equivalents. These modified spectral characteristics were subsequently transmitted through vibrotactile actuators, thereby transforming the essence of sound into a tactile experience. This innovative system creates a sensory replacement approach based on the psychophysical sensations perceived on the skin. It has proven to be exceptionally beneficial, particularly for individuals with hearing impairments who may not perceive music in the same way as individuals with typical hearing abilities.