So how does it work? An easy way to explain it is that a song’s melody is like its fingerprint: They each he their own unique identity. We've built machine learning models that can match your hum, whistle or singing to the right “fingerprint.”
When you hum a melody into Search, our machine learning models transform the audio into a number-based sequence representing the song’s melody. Our models are trained to identify songs based on a variety of sources, including humans singing, whistling or humming, as well as studio recordings. The algorithms also take away all the other details, like accompanying instruments and the voice's timbre and tone. What we’re left with is the song’s number-based sequence, or the fingerprint.
We compare these sequences to thousands of songs from around the world and identify potential matches in real time. For example, if you listen to Tones and I’s “Dance Monkey,” you’ll recognize the song whether it was sung, whistled, or hummed. Similarly, our machine learning models recognize the melody of the studio-recorded version of the song, which we can use to match it with a person’s hummed audio.
This builds on the work of our Research team’s music recognition technology. We launched Now Playing on the Pixel 2 in 2017, using deep neural networks to bring low-power recognition of music to mobile devices. In 2018, we brought the same technology to the SoundSearch feature in the Google app and expanded the reach to a catalog of millions of songs. This new experience takes it a step further, because now we can recognize songs without the lyrics or original song. All we need is a hum.