Hello,

As you may know, I am working on a project relating to voice signal processing. There several goals to this project. One is creating a general voice signal processing software library; another is creating a realistic singing voice synthesizer based on said library. However, a side project of this project has been recreating the VOCALOID2 synthesis engine.

Previously, I have discussed this, but I had yet to really attempt it specifically. Recently though, I made a breakthrough that has been an impetus towards actually doing it. This breakthrough was not a scientific breakthrough, but simply something I found about. Previously, I did not attempt this because I realized I'd actually need to obtain the VOCALOID2 software to have a good chance at doing this. Previously, I had hoped that I'd be able to find clean raw audio of VOCALOID2 output online and compare it to my results. However, I quickly realized this was not enough. First, I couldn't actually find that much. Although the little I did find was quite helpful (it was the basis of my last post on pitch and dynamics models), I quickly realized I'd need a lot more, and would need agency to test specific things. Furthermore, I'd need the specific database to test.

The VOCALOID2 software has not been available for sale for over a decade. One option would be to order a physical copy, but these are hard to come by. Besides, it requires internet activation to install, which cannot work now since the activation servers have been shut down. Although some voice libraries are still available, these require the software itself as a prerequisite.

However, I found out after some research that apparently demo versions of voice libraries actually come bundled with a limited version of the actual software. It seemed unlikely though that any of these would still be available. However, after a quick search I found one that was surprisingly still available. After signing up, I was able to download it.

Indeed, it did not require online activation. However, the installer would fail to start, without giving any error message. I tried just about every trick in the book, however none worked. I even tried https://github.com/lifenjoiner/ISx as an attempt to directly extract files to install manually, but that also wouldn't work. Finally, I moved the installer out of the directory it came in, which contained invalid characters. I thought I had already tried this, but it seems I hadn't, because it worked perfectly following this.

So recently, I have been working on the VOCALOID2 recreation. The basic idea was to implement it based on papers and patents, implementing several possible variations in cases of uncertainty. The VOCALOID2 software would be used as a reference to compare the outputs to, and also to decide the few things that weren't really mentioned in papers or patents. The latter was basically just the pitch and dynamics model, which is the single most transparent thing since it can be analyzed just by analyzing the resulting synthesis audio.

Already, a database parser has been implemented in C, that parses the VOCALOID2 database format. That format seems to be known as DaisyDB, according to figures in Bonada's PhD thesis, so I have named it accordingly. Although I expected the process to be a huge hassle, it actually turned out to be surprisingly easy. Basically, just by spending a couple days in a hex editor, and cross referencing it with papers and patents, I was able to decipher pretty much the entire format. The parser can already parse whole databases without issue, and there are only a handful of fields whose use is not yet known, many of which I expect will be resolved in due time.

Anyway, the main focus of this post is something I just realized something about the frequency-warping rendering variant of the Spectral Peak Processing algorithm (alternative strategy 4/e, p. 89 of "Voice Processing and Synthesis by Performance Sampling and Spectral Models, https://www.tdx.cat/bitstream/handle/10803/7555/tjbs.pdf). This is of significance because I suspect that variant may have been used in VOCALOID2. This is for several reasons. First, it is the only one of those alternative rendering strategies that was patented ("Pitch converter", Bonada et al, JP4840141B2; now-expired)*, and said patent was filed in 2004. The latter detail is important, because I have already deduced that the VOCALOID2 engine must have been completed from between late 2004 and early. This can be seen from two angles.

*A later patent ("Singing composition device and singing composition program"; Bonada, Loscos, and Kenmochi et al; JP4839891B2; now-expired) proposes another technique distinct from all those mentioned in thesis; however it was filed in 2006.

On one hand, as I discovered yesterday, VOCALOID2 actually uses the Maximally Flat Phase Alignment algorithm. This seen based on the fact that what I have deduced to be an MFPA harmonic envelope is contained within the SPP frame analysis data within the VOCALOID2 DaisyDB database format. Each SPP frame has chunk identifier "FRM2", presumably meaning "Frame version 2". In Bonada's 1997 Master's thesis ("Desenvolupament D’un Entorn Gràfic per a L’anàlisi, Transformació I Síntesi De Sons Mitjançant Models Espectrals"), a chunk-based hierarchical format is described. In that format, chunks were also assigned four-byte identifiers, and the identifier for the frame chunk was "FRM1". So the "FRM1" vs. "FRM2" distinction could refer to the iteration of the spectral processing technique; the distinction of SMSTools vs. SMSTools2, or perhaps just different iterations of the storage format.

I was able to determine that it was an MFPA envelope because it was exactly equiv to a linear phase shift applied to the harmonic phases, plus phase unwrapping. Furthermore, it could not have just been the method wherein the harmonic phases are shifted such that the fundamental phase is zero (such as in "Spectral Approach to the Modeling of the Singing Voice", 2001, Bonada et al.), because the fundamental phase was non-zero. The MFPA algorithm was first proposed in the paper "High Quality Voice Transformations Based on Modeling Radiated Voice Pulses in Frequency Domain" (alternatively titled "High Quality Voice Transformations Based on Modeling Radiated Voice Pulses in Spectral Domain"), published in October of 2004.

On the other hand, the patent for the first iteration of the Bonada 2005 expression system (JP2006330615A, now-expired) was filed on May 30, 2005. Since this was the main improvement for the third iteration of the VOCALOID, and, I discussed previously, VOCALOID2 did not use it; this means the VOCALOID2's engine must not have been finished any later than that date.

Another reason is that SPP alternative rendering strategy 4 is the only technique mentioned in other papers and patents, for example in "Improvements to a Sample-Concatenation Based Singing Voice Synthesizer". Finally, in Bonada's thesis he noted "Strategy 4 (E) sounds mostly good but smoothes the utterances a bit", which seems to be consistent with how people describe the specific VOCALOID2 resultant timbre quality. Furthermore, he goes on to say "Therefore, it seems that none of the different strategies proposed improve the overall quality of the basic technique. Only the second strategy (C) yields a comparable quality, being in some cases better, especially for breathy utterances.", which would seem to imply that either they either went back to just the basic SPP rendering strategy; or they switched to another one, perhaps alternative strategy 2 or the one described in JP4839891B2. This would also explain why they let the patent expire especially early.

Anyway, onto the main topic of this post, which is what I just noticed about the figure in Bonada's thesis (which was in turn copied from "Improvements to a Sample-Concatenation Based Singing Voice Synthesizer"), picrel. At the top, we see the original spectrum and its harmonic envelope. In the middle plot, we see an intermediate visualization of peaks after being moved, but before equalization. Notably, there now exists one discontinuity due to one peak being used twice. In the bottom plot, we see the rendered spectrum after equalization. Notably, there is still only one discontinuity in the spectrum. The proposed technique eliminates gaps in the spectrum, but not discontinuities, because the SPP algorithm, as described, regardless of the rendering variant, scales all the bins in each region, which would produce a discontinuity at the boundary between regions.

(continued in next post)