So you may know that I have been working on this project to attempt to recreate the VOCALOID2 synthesis engine. Well I have a major update.

So yesterday I was talking about the SPP variant. As I said, I already had an SPP implementation. But it only implemented the basic approach, and only had a few configurable options. Also, I never actually tested it.

So last night I had big trouble sleeping, so I only got about 3.5 hours of sleep. Nonetheless, for most of the day, I have been working on implementing every conceivable SPP variation. Actually, I basically wrote if from scratch. So in total I've written about 1,000 lines of codes. Actually, this new version only has pitch transposition, and no longer implements time scaling, timbre scaling and equalization. While this sounds like a bad thing, it's actually good. I want to isolate the basic algorithm first and make that as close as possible to VOCALOID2. The interaction with those other transforms adds additional uncertainty and complexity, so I have skipped on them for now.

The code still some stuff to be done, and certainly could use refactoring (although arguably I should wait until after I am sure exactly what is used before doing so, so I can collapse the code first). It's basically this insane hellscape of preprocessor directives that let me control every variation of the algorithm I could think of.

Anyway, after fixing a few compile errors and outright crashes/NaN explosions, it actually worked first try. I loaded a database sample from the VOCALOID2 database and fed it in, and the whole pipeline actually worked first try. The sample used was a stationary, but for actual comparison, I will probably use an articulation instead. This is because some patents suggest a special type of whole-sample spectral concatenation interpolation that applies to stationarys.

Here is my prototype type code: http://up.heyuri.net/src/6335.c

And here are the configuration parameters I used:

#define TEST_SPP_WINDOW_TYPE SPP_WINDOW_2048
#define TEST_PITCH_TRANSPOSITION_TYPE PITCH_TRANSPOSITION_SCALE
#define TEST_SPP_USE_PHASE_DISCONTINUITY_HYSTERESIS SPP_USE_PHASE_DISCONTINUITY_HYSTERESIS_NO // Until it's fully implemented
#define TEST_SPP_INTERPOLATION_TYPE SPP_INTERPOLATION_SPLINE
#define TEST_SPP_INTERPOLATION_TYPE2 SPP_INTERPOLATION2_POLAR
#define TEST_SPP_INTERPOLATION_TYPE3 SPP_INTERPOLATION3_PHASE_UNWRAPPED
#define TEST_SPP_INTERPOLATION_TYPE4 SPP_INTERPOLATION4_LOGARITHMIC
#define TEST_SPP_PHASE_PROPGATION_TIME SPP_PHASE_PROPGATION_TIME_AFTER
#define TEST_SPP_PHASE_PROPGATION_METHOD SPP_PHASE_PROPGATION_METHOD_LINEAR
#define TEST_SPP_HARMONIC_AMPLITUDE_ENVELOPE_INTERPOLATION_TYPE SPP_HARMONIC_AMPLITUDE_ENVELOPE_INTERPOLATION_SPLINE
#define TEST_SPP_HARMONIC_PHASE_ENVELOPE_INTERPOLATION_TYPE SPP_HARMONIC_PHASE_ENVELOPE_INTERPOLATION_SPLINE
#define TEST_SPP_PEAK_PICKING_METHOD SPP_PEAK_PICKING_BONADA2008
#define TEST_SPP_SYNTHESIS_HARMONIC_DISTRIBUTION SPP_SYNTHESIS_HARMONIC_DISTRIBUTION_IDEAL
#define TEST_SPP_ANALYSIS_HARMONIC_DISTRIBUTION SPP_ANALYSIS_HARMONIC_DISTRIBUTION_IDEAL
#define TEST_SPP_PEAK_PICKING_CLOSEST_TYPE SPP_PEAK_PICKING_CLOSEST_ROUND
#define TEST_SPP_REGION_DELIMITATION_METHOD SPP_REGION_DELIMITATION_MIDDLE
#define TEST_SPP_REGION_RENDERING_STRATEGY SPP_REGION_RENDERING_STRATEGY_0A_BASIC_STRATEGY
#define TEST_SPP_REGION_RENDERING_ORDER TEST_SPP_REGION_RENDERING_ORDER_FLOOR_FLOOR
#define TEST_SPP_REGION_RENDERING_STRATEGY_4E_FREQUENCY_WARPING_SCALING_TYPE SPP_REGION_RENDERING_STRATEGY_4E_FREQUENCY_WARPING_TYPE_NONLINEAR_SCALING

#define SPP_PEAK_PICKING_BONADA2008_VALLEY_THRESHOLD logf(FROM_DECIBELS(15.0f))

As you can see, I used the frequency-warping variant of SPP.

Here is the original audio (technically, it was also processed with SPP; but the pitch transposition factor was 1.0, and the transform should be near-lossless): http://up.heyuri.net/src/6336.wav

Now here is a 1.5 factor (+702 semitone) upwards pitch transposition with frequency-warping SPP: http://up.heyuri.net/src/6337.wav
Spectrogram: http://up.heyuri.net/src/6338.png

By comparison, here is the same transform with the original/basic SPP technique: http://up.heyuri.net/src/6339.wav
Spectrogram: http://up.heyuri.net/src/6340.png

The difference is subtle, but I feel like the frequency-warping has a slightly more "VOCALOID" timbre to it. Actually both do to me, but that one especially so. Let me know what you think!
This makes me feel like I'm onto something, although I have not done objective tests yet, although I hope to do those soon.

Also, here is an extreme pitch transposition (3x).
Frequency-warping: http://up.heyuri.net/src/6341.wav
Regular SPP: http://up.heyuri.net/src/6342.wav

About the previously discussed idea from the figure image about the non-linear amplitude scaling - I have yet to implement and test that. I am starting to have some doubts about it though, for example because around the true discontinuity, the different sides appear in opposite order after equalization, which would not happen according to my explanation. Furthermore, after very close inspection of many figures, I believe that the curves are most likely hand-drawn now, or at least hand-edited, whereas before I thought they had been imported.

Let me know what you think!