Cloud Hosting

Prototype used machine learning to classify sounds in real time

Researchers have developed a smartwatch app for deaf and hard-of-hearing people.

November 10, 2020

Smartwatches offer people a private method for getting notifications about their surroundings -- such as a phone call, health alerts or an upcoming package delivery.

Now University of Washington researchers have developed SoundWatch, a smartwatch app for deaf and hard-of-hearing people who want to be aware of nearby sounds. When the smartwatch picks up a sound the user is interested in -- examples include a siren, a microwave beeping or a bird chirping -- SoundWatch will identify it and send the user a friendly buzz along with information about the sound.

The U.S. National Science Foundation-funded team recently presented these findings at the ACM (Association for Computing Machinery) conference on computing and accessibility.

"This technology provides people with a way to experience sounds that require an action -- such as getting food from the microwave when it beeps," said lead author Dhruv Jain, a UW researcher in the Paul G. Allen School of Computer Science & Engineering.

Jain was born hard of hearing. "I use the watch prototype to notice birds chirping and waterfall sounds when I am hiking. It makes me feel present in nature. My hope is that other deaf and hard-of-hearing people who are interested in sounds will also find SoundWatch helpful."

The team started this project by designing a system called HomeSound for deaf and hard-of-hearing people who wanted to know what was going on around their homes.

"I used to sleep through the fire alarm," said Jain.

The researchers tested HomeSound in the Seattle-area homes of six deaf or hard-of-hearing participants for three weeks. Participants were instructed to go about their lives as normal and complete weekly surveys.

Based on feedback, a prototype used machine learning to classify sounds in real time. The researchers created a dataset of over 31 hours of 19 common home-related sounds, such as a dog bark or a cat meow, a baby crying or a knock on the door.

Link:

New smartwatch app alerts deaf and hard-of-hearing users to common home-related sounds - National Science Foundation