Analysis/synthesis is a paradigm of audio processing. In this paradigm, an audio signal is analyzed to produce a model of the signal. Then this model is transformed according to our wishes. Finally, the modified model is synthesized to produce the transformed signal. This paradigm can be used not only for transforming signals, but also for synthesizing them from scratch by means of selecting prerecorded samples from a database. Also, another method is omit the analysis stage and generate the model using statistics or neural networks.

In analysis/synthesis, for any real (i.e. non-synthetic) signal, there will always be a portion of it which cannot be represented well by the model. The exact composition of this component depends on exactly what the model itself models, but in this post I will be focusing on what can not be modeled by any model, that is, stochastic components.

Broadly, even if they are not specifically implemented this way, analysis/synthesis techniques that consider the stochastic part due so by first separating out the part that cannot be modeled by the model. Although the remaining component is stochastic, it is not strictly random. Usually, this means it is either tied to the model, or a separate model that describes a non-random aspect of the stochastic component. For example, a typical case is that the spectrum of then stochastic part is not flat as it would be in white noise, but follows an envelope. In that case, the extraction of just the random part would be done by inverse filtering the stochastic component by that envelope, a process known as spectral whitening.

From there, there are basically two ways to proceed. Either the random signal is preserved, or it is discarded and replaced with white noise at synthesis. The former has the advantage that, depending how the rest of system is implemented, it may be possible to reconstruct the original signal perfectly when no transforms are applied. Additionally, in practice, the 'random' component will in practice always include some amount of signal, so discarding loses some aspect of the original signal. On the other hand, the latter has the advantage of flexibility. Often, transforming the model requires that we now have more or less samples of the random part.

One option would be to interpolate the random component. However, when more samples are required than there were originally (i.e. upsampling), the new signal is smoother than the original. This can be a significant problem, for example if we are slowing down a signal by large factor.

From signal theory, we know that all other interpolation methods approximate the ideal sinc-convolution interpolation. In the frequency-domain, that method corresponds to there being no new spectral components above the old Nyquist frequency when upsampling. Given that our random component is supposed to be white noise, i.e. that it should have a flat spectral envelope, we can now see where the smoothness problem comes from.

Thus, from this observation, I propose a hybrid model. We preserve the original random component, with all the advantages that brings. When requiring a different amount of samples than there were originally, we resample it. However, in the case of upsampling, we replace the new spectral bins with random values instead of zeros.

Thusfar, I have discussed how the spectrum should change under this proposal, however in practice the direct principle cannot be implemented directly. Two practical reasons are that my FFT implementation has fixed maximum size, so it cannot be used on arbitrary-length signals; and that doing so also becomes inefficient with larger and larger FFT size. However, a more fundamental issue is the circular nature of the FFT which results in artifacts whenever the signal is not entirely periodic with a period corresponding to exactly the FFT window. This problem can be either mitigated through techniques such as overlap-and-add; or time-domain convolution (i.e. filtering) can be used instead. For linear transforms, the latter is almost always used because it is much more efficient, and at the same time, it produces far lower levels of artifacts.

Based on that, this proposal could be implemented by lowpass filtering the interpolated signal (i.e. resampling), and adding to that highpass-filtered white noise whose bandwidth corresponds to the difference between the new and old sampling rates. In practice, real lowpass filters are not ideal, so they have a non-zero transition band. Thus, the resulting envelope is not completely flat and would have spikes due to the transition bands. This could be reduced, although not completely, by increasing the filter size, at the cost of additional computational complexity. Another option is to adjust the bandwidths slightly such that total distortion is minimal (i.e. error can in either direction, not just positive).

Another interesting idea would be to see if a window could be used whose frequency response is overlapping. In the time-domain, some windows are said to be overlapping, for example the triangular window, Hanning window, and Gaussian window. What that means is that it is possible to separate the windows by a specific distance, and the resulting of adding them together is one outside of given edges. The Kaiser-Bessel window is almost-but-not-quite overlapping, which is because it is almost-but-not-quite the Gaussian window. If a window's frequency response were to be overlapping, using it as the FIR filter construction window would solve the problem from before. Special care would need to made to ensure that the filter's transition band matches the size needing for proper overlapping as closely as possible. Usually we do not care about this since the transition band being smaller than desired is usually exclusively a good thing.