Lip-sync on Diffeomorphic models

Lip-sync

In this post we will go through the procedure to create a lip-sync animation for a Diffeomorphic based model.
Note that in general automatic lyp synching might not generate perfect results, and it might need some tweaking. However, it might be a good starting point to enhance your animation with some “talk”.

Requirements

Some things you’ll need:

• obviously a Diffeomorphic model (such as Tifa);
• Papagayo, a lyp synching generator software;
• a .wav sound file to insert in Papagayo. If the format of the file is different, you can find online tools to quickly convert them.

Papagayo

The Papagayo window will look like this.

We can simply drag and drop the wav file. A sound analysis of the wav file will appear on top, and you can write the spoken test in the field below. You can also choose the FPS in the upper right field (this is important to get a correct result, but you can also scale the lip synch in Blender).

Note that, in the example above, the text is not overlapping with the sound. We can solve this modifying the text position. Just keep pressed on any of the vertical grey columns, and drag them. You will also hear the sound where you’re placing your text.
Note that in general, even if you just want to quickly create a lip synch, it is better to make the text overlap at least with the blue part.
Once you finish tweaking your text, you’ll have something like this

Finally, just Export the file as .dat.

Blender

We can import the file with the Diffeomorphic addon. Select the Import Daz tab, in the Morph panel. Go to the Visemen part, and click on the Load Moho button.

A window will appear. Choose the .dat file and check the settings on the right (if they are not available, press N in the import window).
In general all those settings can be left default, with the exception of:

• Emphasis: the generated animation sometime might be too strong. If this happen, go back and import the .dat file with less Emphasis;
• New action: choose where to add the animation. If you don’t ask for a new Action, it will create it in the current one. This obviously overwrites all visemen eventually already there;
• Start At Frame 1: if you disable this, the animation will start at the current frame.

If the import was successful, you now have your animation on the Dope Sheet!

Troubleshooting

• Error when generating the animation in Blender: be sure to select the armature where you want to import the animation, otherwise the import might fail.
• Rest pose before/after the animation are the first/last visemen: obviously the first generated key will be with the first visemen. If you want to start/end from rest “un-talking” pose, you can add some additional keys before/after with all visemens set to value 0. The visemen values can be modified in the same Visemen tab of Diffeomorphic shown above.

Reference

For more informations about Diffeomorphic, check the documentation.