Introduction
This is an adaptation of a presentation I did recently for a Human Library session about learning music even with a hearing loss. The presentation was originally done with a group, but I’ll try to summarize the main points here and also introduce a small project I did with my digital piano. I’m a hard of hearing software engineer based in Japan - my Japanese isn’t very good, so please excuse me writing this in English 🙂
Playing music with a hearing loss
First, let’s address the big question: can deaf people be musicians? The answer is a resounding “yes!” - there are too many examples to give, but let me introduce Evelyn Glennie, Mandy Harvey, and Mariko Takamura, just to name a few. There’s some potential differences that might be interesting to look at - some are prelingual deaf, others are postlingual, or developed a hearing loss after, for instance, repeated exposure to loud sounds (not uncommon for some genres of music), or to illness, etc, but it isn’t too important for the sake of this discussion. Still though, a deaf person playing music still invokes some degree of wonder and surprise - just as much as a blind person drawing art, or someone with limited mobility participating in sports. It’s not a new topic - for instance, see this recent British short film, “A Sonic Pulse” that discusses this very topic. We wonder, “is it really possible?” At times, it almost seems to me that the people most skeptical are the deaf and hard of hearing themselves.
But I can definitely say that it is possible. There are people around me who are hard of hearing or deaf, and also learned to play violin and other instruments. As for me, I’ve been playing piano for over 10 years now, starting as a child, taking a long break, and only recently rediscovering it as an adult, so I can share my own experiences. It wasn’t all smooth sailing though - I’m told that finding a piano teacher initially was tough and my parents had to search for a teacher willing to teach me. I suppose some teachers must balk at the prospect of teaching someone who is deaf, or are inflexible with adapting their teaching methods to suit the needs of that person! But, I’m glad that I did find that first teacher, and ever since then, I’ve never struggled to find another teacher afterwards - it was only the first one that was challenging.
So, what’s tough about playing music with a hearing loss? In my experience, the biggest difficulty has to be the somewhat imprecise judgement of pitch (e.g., no “perfect pitch”). It’s already challenging enough for those with no hearing loss to acquire a good sense of pitch, but it’s even more difficult for the deaf, who have to deal with modern hearing aids, etc that do all manner of frequency compression and other techniques that change the sound to match the user’s hearing loss. That means that a C note could sound a bit off and the intervals between the notes might not be necessarily fixed. However, despite that, even if we can’t hear the expected sound, we can always rely on our “inner voice” that is always 100% in tune 🙂 Thus, reading a music score while listening dramatically helps the brain to process and understand the music. It can be a lot of mental processing in the beginning, but later, once somewhat memorized, I can relax and better appreciate the music next time around. It’s a good mental workout - I can’t claim to have concrete evidence, but I suspect that listening to music in this way helps make your brain develop more connections and work better!
I hope I’ve convinced some of you reading this to consider learning music. Music is about many things - some major components being patterns/rhythms and ideas/motifs, which are universally understood. If I had to make broad generalizations, it might be easier to learn percussion or instruments that use fixed/discrete tones like piano, etc rather than continuous ones such as trombone or violin that require constant pitch adjustment. Learn to read musical notation to help enhance your listening experience. But most importantly, find a style that works for you!
Shared piano experiment
Piano is a wonderful instrument that is almost limitless in versatility - we can play all kinds of musical styles and works arranged for solo piano without needing an ensemble. But it’s not portable at all, and difficult to find opportunities to play with or for other people. That’s why I was excited when I discovered a Chrome Experiment called Shared Piano - if you attached a MIDI-capable digital piano to a computer, you can play on the piano and someone else loading your URL could hear your playing from somewhere else! The piano sounds are a little synthetic, but it started getting me to think about the possibilities of digital instruments and sample libraries. A far more simpler approach would just be to pipe the line out audio of the piano into a computer, but this didn’t seem exciting enough 🙂
On a spare Raspberry Pi, I started looking at all the different options available but I was adamant that I wanted something completely self hosted and based on open source. Here’s an overview of the final system architecture that I built.
The key component here is the JACK audio system on Linux that provides low-latency software defined audio routing between applications. We need to bridge ALSA to JACK at places, to support non-JACK apps, but once configured, the latency up to the Icecast component is low. JACK supports many different plugin hosts, and the one I chose is Carla, which supports the free Salamander Piano sample library. This takes the MIDI data from the piano and converts it to a realistic sounding piano, perhaps even better than the piano sounds on the Casio PX750 itself. However, to deliver it over the web, we need to compress it, and the choice for that is ffmpeg. The last piece of the puzzle is Icecast, which is normally used to build an Internet radio station, but can also be used to stream audio in this case. If I send a link to the Icecast stream to a friend, they can hear the music, just like the original Shared Piano that I was trying to emulate!
One annoyance that I wasn’t able to completely solve is the latency/buffering introduced by Icecast. Icecast isn’t designed for low latency streaming - Internet radio doesn’t need these kinds of specifications and a total system e2e latency of 30 seconds is fine in this case. Even after adjusting all the parameters I could find, there is still a latency of about 5-10 seconds introduced. This could potentially be solved by using a different streaming technique. Also, we have implemented streaming of the piano audio, but it isn’t collaborative in the same way as the original Shared Piano, so some investigation of the actual Web MIDI APIs, and whether we can bridge this into JACK would be a good next step. Finally, given that we are using software instruments, it opens up the possibility for using other kinds of instrument samples or to allow different people to choose their own instrument to play together.
That’s it! If you have any questions or comments, please leave a note below! Thanks for reading 🙂