Hello I'm back with another update to my VOCALOID project. It's not as big an improvement as last time - and in fact, there's no new features - but I felt like it was worth posting. I've been trying to rectify the major issues before I move onto implementing the Excitation plus Resonance model.

The first thing I attempted to tackle was all the added noise at high frequencies.
Here's the original spectrum: https://files.catbox.moe/fq55bo.png
And here's the reconstructed spectrum (with no transforms applied): https://files.catbox.moe/gq7jff.png
You can clearly see the high frequency artifacts. The first thing I tried was something mentioned in the paper. In the paper, specifically the WBVPM section, it was mentioned that there are two approaches for a non-integer size discrete fourier transform. The first one is repeating the signal while second is upsampling it. I went with second as the former is patented and also because the second is easier to implement. It is mentioned that increasing the repetition count of the signal (or in the case of upsampling, the upsampling factor), and then discarding the higher frequencies, can improve the estimation by reducing artifacts. In the case of repetition, it is also mentioned that quadratic interpolation can be used in the resulting spectrum, however I am not sure if this can be done for upsampling and as such, I have not tried to implement it for now.

Here's the result after applying an upsampling factor of 3: https://files.catbox.moe/qcgnzq.png
Here's the original audio: https://files.catbox.moe/f7g8ta.wav
The original reconstruction: https://files.catbox.moe/da0m1i.wav
And now with the improved reconstruction: https://files.catbox.moe/513ycn.wav
You can see an improvement, especially at lower frequency, however the high frequency artifacts largely persist. So they have to be arising elsewhere. I realized the source was the reconstruction of the signal (AKA the "synthesis"). I had previously implemented a synthesis method that was quite different from the one used in the study, because I did not understand the method in the study at first. My synthesis method worked by taking each voice pulse and for each sample where the voice pulse is the closest voice pulse to that sample, setting the value of that sample to the interpolated value of a spline representing a time domain version of the upsampled voice pulse with a step corrospondin between the ratio a sample in the regular time domain and the upsampled time domain. Now, in some cases, estimation inaccuracies and differences from any transformations that were applied result in these regions of samples being bigger than the actual sample itself. In these cases, we take advantage of the period nature of the voice pulse and repeat it (i.e. sampling before the start is equivalent from that offset from the end, and sampling after the end is the same as that offset from the start). However, this method results in discontinuities in some cases.
Here is an example of such a discontinuity: https://files.catbox.moe/jnnxfj.png
I began to try to implement an interpolation system. In this system, we could calculate the gap between pulses - or in the cases of inaccuracies in the other direction (i.e. overlapping pulses) - the overlapping area, and interpolate between one pulse and the other linearly. However, this was approach was complicated significantly by the non-integer (and potentially differing) sizes of the pulses as well as numerous edge cases. For this reason, I struggled to do so and spent over an hour trying to figure out how to do it corrrectly. About half way through, I decided to check the paper again and this time I understood the actual synthesis method properly, largely because of a diagram I had missed the first time.
In the actual method, each pulse is is expanded in a manner similar to that of the border interpolation technique used in WBVPM analysis, except kind of in reverse. In this technique, for each voice pulse, we generate extensions on both sides with each extension having the size of the border interpolation ratio of the size of the voice pulse. Then we apply a trapezoidal window to the voice pulse which starts at zero at each side of the extended voice pulse and becomes 1 on either side after protrusion of twice the border interpolation size on each side. Then we overlap and add the voice pulses.
This technique fixes the discontinuity issue because it effectively results in each border-interpolation-length side of each voice pulse being interpolated with the corrosponding section for the other voice pulse linearly over a period of twice the border interpolation size. However, this only holds perfectly when the fundamental frequency is the same for both voice pulses (and thus they are the same size) and they are spaced out at onsets that are exactly the period of the fundamental frequency apart. However, when this in not the case, some amount of modulation occurs that results in some voice pulses being attenuated while others are accentuated. This is especially noticeable when there are large inaccuracies in the fundamental frequency estimation and/or the voice pulse onset sequence.
Here's the same section from before. Notice how now it does not have a discontinuity: https://files.catbox.moe/p26914.png
Now here's a zoomed-out version: https://files.catbox.moe/zacw8w.png
Now here's a section with large inaccuracies in the MFPA estimation that clearly shows large modulation artifacting: https://files.catbox.moe/efk1vx.png
Here's the new spectrum: https://files.catbox.moe/f94zse.png
You can see that while the high frequency artifacts are now gone, there are now more low frequency artifacts. In fact, the overall amount of artifacts is actually higher than before.
Here's the reconstructed audio: https://files.catbox.moe/ympfi0.wav
While I ended out solving this issue by fixing large inaccuracies in the MFPA system, it is interesting to note that my approach is more resilient to estimation inaccuracies. Perhaps for a future improved vocal synthesizer, it would be worth exploring a variant of my periodic continuation technique adapted with an interpolation method that could handle changes in pulse onset and f0.

The first thing I tried was switching to a magnitude-limited logarithmic scale for the ampltiude in the MFPA function instead of it being linear. However, this resulted in little to no effect. The next thing I tried was adjusting the size in periods of the window used for the peaks that are fed into MFPA, however again this resulted in little to no effect. Next, I tried implementing the harmonic peak selection algorithm I proposed in the previous post, but again this resulted in little to no effect.
Finally, I began looking at the MFPA refinement algorithm instead, and I found something quite interesting:
In this section, these are the per-frame detected onsets: https://files.catbox.moe/f36gno.png
Now here's the onsets chosen by the MFPA refinement algorithm: https://files.catbox.moe/4b7hc0.png
Notice that while one of the onsets in the detected onsets is wrong, there is also a correct one for that voice pulse, and additionally, the incorrect onset chosen was actually for the next pulse. Furthermore, that incorrect chosen onset was actually not even a detected one - the one detected for that frame was correct - so it must have been one of the additional onset candidates considered by the MFPA refinement algorithm. I realized shortly after what the issue was: When I first wrote the MFPA refinement algorithm, I was under the false assumption that it's primary purpose was to compute a superset, rather than a subset, of the detected onsets. Because of this, I realized I could make a simplification to the algorithm. In the paper, it says to calculate the MFPA error by finding the closest MFPA onset to the frame. However, since I thought there should be at most onset per pulse in the detected onsets, we could do this by just getting the onset time at that frame index (where we get the frame index by rounding the time). I believe actually even written the code originally to use a search, but simplified it.
But since now there can be (and usually are) multiple detections per pulse, that assumption is no longer true and by doing that, we may choose a pulse which is not actually the closest. In the case I show above, what probably happened was that the wrong detection in the previous pulse was chosen, resulting in choosing the wrong one for the next pulse. I fixed the issue by making it use a search (and also sorting the detected onsets first), and it fixed that section: https://files.catbox.moe/0oryig.png
Here's the section that was heavily modulated before: https://files.catbox.moe/r0oq2w.png
And here's the spectrum: https://files.catbox.moe/pgfqfh.png
Notice the low frequency artifacts are mostly gone.
And here's the reconsutrcted audio: https://files.catbox.moe/98zbd1.wav

Now here's the pitch transposed audio with the fixes applied: https://voca.ro/1izsfK1EwXD3
Compare to before: https://voca.ro/1mJ5qljrp9hD