As some of you may know, I have been working on a project to create a singing voice synthesizer. Actually, the actual development is currently on hold, but I am still doing research and planning. The main goal of this project is to create a realistic traditional singing voice synthesizer (although integrating machine learning ideas into it could also be interesting), but side goal is also to recreate VOCALOID2 (actually this was the original goal, but the objective changed early on).

Part of this task is to implement the expression system. I had long realized that there were two separate expression systems, but initially there had been some confusion between what belonged to which. Initially, I worked based on the expression system described in Jordi Bonada's 2008 PhD thesis, in Chapter 3, because this was by far the most complete description. For a long time, I had thought the expression briefly mentioned in the 2003 paper "Sample-Based Singing Voice Synthesizer Using Spectral Models and Source-Filter Decomposition", in part due to the reused figures. Because of this, I referred to this system as the "Bonada 2003" expression system, because that's where I thought it had been described, although only briefly.

Later, I read parts of Jaume Ortola's 2001 Master's Degree. In there, some of the "2001" (or "Ortola 2001") expression should be described. Good detail is provided on the dynamics curves, which use Manfred Clynes' Predictive Amplitude Shaping algorithm. On the other hand, the pitch model is not really described, stating only: "The pitch contour of the singing voice has to be carefully generated in order to obtain a faithful synthesis. So we have designed a mathematical model for reproducing the smooth pitch transitions between notes. This model allows us to control the transition duration and the tuning deviations at the end and the beginning of the notes in accordance with the musical context.".

Nothing about what happens in between the note transitions was stated at all, so this was a total mystery at the time. With what I know now, I am still not fully sure. The basic model is probably either flat lines or linear interpolation between the start and end. On the other hand, looking at the figures, besides the vibrato present in some of them, there is clearly something else as well. In one expired patent I read, the mean was subtracted from timbre variations and those were added on top of interpolated timbre, so perhaps something could be happening like that for pitch, although it is unclear then what would happen for the areas that don't correspond to stationary PhUs. Another possibility is some kind of random noise that is added. Actually, the figure looks suspiciously like something that has been linearly interpolated at the edges. You can clearly seen in the transitions and expressive parts, full pixel-level resolution; on the other hand, in these areas, it looks like straight line interpolation. So perhaps it is random noise that is interpolated.

Anyway, this "mathematical model" in reference in many other places, but not described in any of them. For example, in "Sample-Based Singing Voice Synthesizer Using Spectral Models and Source-Filter Decomposition": "In the case of note transitions, the process is the same but whenever no template is specified, a pitch model is applied that overwrites the absolute pitch track of the score, like shown in Fig. 3, so to avoid pitch discontinuities. This pitch model has to be carefully generated to obtain a natural sounding pitch curve in the output synthesis. A mathematical model has been designed to produce smooth pitch transitions between notes and allow the control of some parameters like duration, shape and synchronization to phonetics and musical rhythm."

So recently I have been reading the expired VOCALOID patents, and made an effort to catalog and then read them all. I read https://patents.google.com/patent/JP2006330615A which describes the expression system described in Bonada's 2008 PhD thesis. Because of this, I am now calling this expression system the Bonada 2005 expression system. Interestingly, this finally described that mathematical model. The mathematical model for the legato transitions is not mentioned at all in Bonada's thesis, so I think it was replaced by the performance-sampling method for producing legato transitions that he also described in that thesis was used instead.

Another interesting thing that this patent mentions is the addition of supplemental points before attacks and after releases. This was not mentioned at all in Bonada's thesis. Interestingly however, the thesis did provide the parameters for these points. Second, another very interesting thing is that in the patent, all of the parameters have fixed values, with the option to randomly scale them. On the other hand, in the thesis, the values were generated according to gaussian distributions. This must have been entirely different methods for generation and not just a writing quirk because, for example, the patents mentions that the points A and C are above the nominal pitch, and B below, while in the thesis, all points' gaussian distributions have a mean of zero. For this, along with other differences, such as the description of the supplemental point pitches being inconsistent with the given parameters in the thesis, I believe these two descriptions actually referred to two different iterations of this expression system, which I am now referring to as Bonada 2005 V1 and Bonada 2005 V2.

This clears some things. On the other hand, there are also things where there is now more ambiguity, so more things to test. Other things are still unclear, such as how the point based model is used to generate the dynamics.