Vocoders

In the 1930s, Bell Labs developed a system that split speech into pitch and frequency bands to send it over narrow phone lines, before reconstructing it at the other end. It was known as a voice encoder, or "vocoder" for short. In modern text-to-speech, the term is used for the reconstruction half: a model that takes frame-by-frame acoustic representations and converts them into audio waveforms.

In Representing Audio Data, we slid a 1,024-sample Hann window over our audio recording 256 samples at a time, performing a Fourier transform on each slice. This process is called the short time Fourier transform (STFT).

I will do a more detailed exploration of the STFT in another post. For now, the important thing to know is that we get two pieces of information about the waveform from the STFT, represented together as a complex number:

  • The magnitude: how strong a frequency is during the window.
  • The phase: where along its up and down cycle a wave is during the window.

When we create a spectrogram, we discard the information about the phase. This creates a problem. To see why, have a look at the visualisation below. For a single wave, changing the phase just slides it left or right without changing its shape. However, with a more complex wave made up of different frequencies, the phase determines how the frequencies interact with each other, and changes the shape of the overall wave accordingly.

How does a complex number encode magnitude and phase?

Each frequency bin in the STFT is a complex number a+bia + bi, which you can picture as an arrow from the origin to (a,b)(a, b). Its length (a2+b2\sqrt{a^2 + b^2}) is the magnitude and its angle is the phase.

Rather than searching over every possible shift of a wave, the Fourier transform measures how much the signal matches two fixed reference waves: an unshifted cosine (aa) and an unshifted sine (bb). Adding those two dashed waves together — acos⁡(2πft)−bsin⁡(2πft)a\cos(2\pi f t) - b\sin(2\pi f t) — reproduces the wave at any height and phase shift.

170 Hz bin: 0.71 + 0.71i

Re (a)Im (b)length = 1.00 · angle = 45°

Resulting 170 Hz wave

suma·cos−b·sin
051015Time (milliseconds)
One Fourier bin is a complex number a + bi, pictured as an arrow: its length is the magnitude (height of the wave) and its angle is the phase (where the wave starts at t = 0). Adding the two fixed reference waves — a·cos (grey) and −b·sin (orange) — reproduces any shift.

This has big implications for how the reconstructed audio sounds.

Playing: STFT → iSTFT (with phase)0.00 / 0.00 s

1. Inverse FFT of each STFT frame (7 of 7 overlapping slices shown)

4,2000−4,200

2. Overlap-add sum (colour) vs original waveform (grey)

4,2000−4,200020406080100Time across 7 overlapping STFT frames around “i” in “times” (milliseconds)
Each STFT column is inverted back into a 43 ms tapered slice (top) and added 10.7 ms after the previous one (bottom). Wherever four slices overlap (shaded band), the sum snaps onto the original recording in grey (full-sentence max error —).

Tags: AI