Representing Audio Data
We live, as they say, in a society. We also live in an atmosphere. The pressure of the air around us is, I am told, roughly 101,325 pascals. This air pressure is always changing though as the air swirls around us. Our ears have evolved to be sensitive to variations in the air pressure that happen at certain speeds, those that occur between 20 and 20,000 times a second.
The number of occurrences of something per unit of time is called a frequency. The standard unit of frequency is the hertz (Hz). 1 hertz is defined as one event or cycle per second. So we can say that our ears are capable of hearing sounds between 20 and 20,000 Hz.
Microphones are designed to produce a voltage that tracks how far the air pressure is above or below the norm, with a voltage of 0 meaning no deviation. When we record sound through a microphone we take samples of the value of the voltage at regular fixed intervals. The number of intervals per second (or frequency of intervals) is known as the sample rate. 24,000 is a typical sample rate when training text to speech models. CDs use 44,100. Videos tend to use 48,000.
The value of the voltage is rounded to the nearest step on a fixed scale. This is known as quantisation. In a 16-bit file, that step could be between -32,768 and 32,767. The person or audio software doing the recording will set the input volume (known as "gain") so that typical sounds use a good deal of the range without actually hitting the ends.
It might seem unintuitive that loud sounds can take the biggest or the smallest values but remember that a sound wave can push the pressure both ways. It can squeeze the air together in places, and thin it out in others.
The Waveform
Below is a plot of a recording generated by a text to speech model. Each "point" on this plot represents a sample of the amplitude at a particular time. Notice how the loudest parts of the recording create values both far above and far below the norm.
“A microphone measures air pressure thousands of times every second.”
0.00 / 0.00 sLet's zoom in on a few sounds in that recording and see what their waveforms look like.
The waveform of the "i" is made up of a repeating pattern. When a person makes this sound their vocal folds open and close, releasing a puff of air. Sounds made this way are described as voiced. The peaks in the pattern are about 5.9 milliseconds apart, which means about 170 repeats a second. This repetition rate is the pitch, about 170 Hz.
When making a sound like "s" by contrast, our vocal folds stay still. Air is instead forced through a narrow gap between the tongue and teeth which creates the turbulent messy sound that is reflected in the waveform. Sounds like this are described as unvoiced. Because there are no regular repeats there is no sense of pitch. You can't sing the letter "s" higher or lower. We can see though that there is a lot of energy concentrated in higher frequencies as the soundwave crosses the zero line about 9,000 times a second.
The Spectrum
Let's have a look at the waveform of the same vowel, spoken by a different voice.
Notice how a repeating pattern remains at a similar interval, but the shape of the pattern is slightly different.
The name for the "shape" of a sound is its timbre. It's what differentiates the sounds of two musical instruments playing the same note. It's very important to have a mathematical way of describing the timbre of a sound when generating audio to reproduce it.
Let's start with an extremely simple sound. A single note whose waveform is a sine wave like the one below.
And here is an equally simple sound with a frequency that is twice as high.
Now here is a sound whose waveform is the sum of the previous two sounds' waveforms.
Notice how both of the original sine waves are still visible in the wave of the more complex sound.
It turns out that it is possible to perform this operation in reverse and decompose the complex sounds we experience in the real world into the simpler sounds that make them up. The operation that performs this decomposition is the Fourier transform.
The step by step operations that make up a Fourier transform are something I'll write up in a separate page another time. For now, the best resource for getting a high level understanding of why the Fourier transform works is Grant Sanderson's video.
Applying Fourier transforms to the combined waveform, and to the "i" sound from the audio above, produces the following frequency spectra.
The first spectrum has two spikes, at 170 and 340 Hz. So we have retrieved the frequencies we used in the synthetic waveform.
The second has many more spikes, which get smaller as the frequency rises. These spikes are called harmonics. Each appears at a whole number multiple of the vowel’s 170 Hz pitch. This is because the waveform is a puff of air repeated 170 times a second. Frequencies that are whole multiples of 170 Hz will line up together, so their strength will build up. Frequencies in between line up with some puffs and against others. So they end up cancelling themselves out.
The harmonics are also wider than in the first spectrum. The cancelling only works fully over many repeats, and our clip of the vowel is only 43 ms long, about seven puffs. The synthetic tone lasts a full second, so its harmonic spikes are much sharper.
After the third harmonic, each one is less than 4% as strong as the first, so on this scale they all but disappear. To see them we can plot strength in decibels (dB) instead. Here 0 dB is the strongest frequency in the clip, and every 20 dB below that is ten times weaker. This gives each factor of ten the same amount of room on the plot, so the quiet harmonics become visible. This is closer to how we hear, since our ears judge loudness by ratios rather than differences.
How do decibels work?
Decibels describe a ratio on a logarithmic scale, so they always compare a value against a reference. The formula is:
Every 20 dB is a factor of ten, so −20 dB is a tenth of the reference's strength, −40 dB a hundredth and −60 dB a thousandth.
You may have seen decibels used for loudness, where a conversation is about 60 dB and a jet engine 120 dB. In that context 0 dB is defined as the quietest sound a human can hear. Because that reference is fixed and widely agreed on, it usually goes unmentioned.
Now we can return to the two voices from the start of this section. Here is the same "i" sound from each, in decibels.
The spikes sit in almost the same places, because both voices are speaking at almost the same pitch. But their heights rise and fall differently. Those broad hills in the heights are called formants, and they are set by the shape of the speaker's throat and mouth. They are a large part of what makes the two voices sound different, even when saying the same vowel at the same pitch.
The Spectrogram
The spectra in the previous section were for a single synthetic tone, and a steady vowel. The frequency spectrum of a longer recording changes as the sound changes. We can, if we want to, run a Fourier transform on our entire example sentence, and mix it all into a single plot.
This only tells us which frequencies appeared across the sentence overall, not when they happened. To capture each sound in the order it was made we can slide a short window along the waveform and take the Fourier transform of each slice. The trade off here is making the window narrow enough to capture one sound at a time but long enough to fit several repeats of the vocal pitch. In this example, I've used 1,024 samples, which is equivalent to about 43 ms.
If you click play on the animation below you'll see it plot the spectrum of each window as it passes through the waveform.
“A microphone measures air pressure thousands of times every second.”
0.00 / 0.00 sHow is each window cut out?
If you chop a slice out of a waveform, the signal starts and stops abruptly at the edges. The Fourier transform treats finite clips as though they loop. The abrupt jump in amplitude from the end of a slice back to its start gets recreated by the Fourier transform using high frequencies that weren't originally there.
To prevent this, each slice is multiplied by a smooth bell-shaped curve (a Hann window) that starts at zero, rises to one in the middle, and fades back to zero at the end:
where is the sample's position in a window of length .
“A microphone measures air pressure thousands of times every second.”
0.00 / 0.00 sThis tapering squashes the ends of each slice, so adjacent windows are overlapped. In the spectrograms below, each 1,024-sample window steps forward by 256 samples (about 11 ms), so every part of the recording passes through the middle of a window.
To visualise all the data in that animation in a single picture, we can use colour. We plot every window's spectrum alongside one another, rotated. In a linear spectrogram like this:
- The horizontal axis is time.
- The vertical axis is frequency (0 to 12,000 Hz, split into 513 evenly spaced bins; a 24,000 Hz recording tops out at half its sample rate because capturing a wave takes at least two samples per cycle).
- The brightness is the strength of each frequency at that instant.
“A microphone measures air pressure thousands of times every second.”
0.00 / 0.00 sNotice how on the linear spectrogram above, most speech detail is crammed into the bottom third. Half the vertical space goes to 6,000 – 12,000 Hz, mostly the noisy hiss in sounds like "s" and "f".
Human pitch perception responds to ratios rather than absolute differences, much like with loudness. We find it easier to tell low frequencies apart than high ones.
The mel scale (short for melody) warps frequency so equal steps sound like equal steps in pitch, stretching low frequencies and compressing high ones. Grouping the 513 linear bins into 80 evenly spaced bands on this scale gives us a mel spectrogram.
How is a mel spectrogram calculated?
We map frequencies in hertz onto the mel scale, before rebinning them.
“A microphone measures air pressure thousands of times every second.”
0.00 / 0.00 sThe frequencies below 4,000 Hz now take up roughly two thirds of the chart instead of one third. The harmonics and shifting formants are clearer.
Mel spectrograms pack the parts of sound we actually notice into a compact grid. Speech recognition models often take mel spectrograms as input. Many text to speech models generate them as an intermediate step. Because discarding phase and averaging bins into mel bands throws information away, you can't mathematically invert a mel spectrogram back into audio. So systems based on mel spectrograms need a separate neural network (a vocoder) to be trained to turn them back into a waveform.
Linked from
Tags: AI