CRIWARE Unity Plugin Manual
Last Updated: 2024-07-12
ADX LipSync Library
What's ADX LipSync?
ADX LipSync is middleware that automatically generates mouth shape information (mouth patterns) by analyzing speech waveforms.
A combination of machine learning and signal processing technology generates natural mouth patterns from speech data.
There are two ways to animate lips based on analyzed voice data: by "calculating the movement in advance" or by "deciding the shape of the mouth from audio played in real-time".
ADX LipSync can be used in both of these cases, contributing to improved mouth motion quality and reduced workload.
ADX LipSync Usage
ADX LipSync can be used in the following two ways depending on how the character's mouth motion is set.
Pre-analysis with ADX LipSync Tools
Real-time analysis with ADX LipSync Library
In "Pre-analysis with ADX LipSync Tools", when an audio file is input to the command line tool, mouth pattern data is output in text format.
By importing this into the DCC tool and editing it, you can use it to edit the facial motion of your character.
Tools for pre-analysis are included in this SDK package.
Refer to the manual of the following path:
cri/tools/ADXLipSync/manual/ja/ADX_LipSync_Tool_Manual_ja.html
In "Real-time analysis with ADX LipSync Library", mouth patterns can be obtained by analyzing audio played in the game in real-time.
By connecting to CriAtomSource, which is a sound source, you can control the mouth motion of the model without complicated coding.
This manual explains how to use ADX LipSync mainly assuming real-time analysis.
Automatic generation of natural mouth patterns from voice data input
Analyzes the waveform of the input voice data and automatically outputs the optimum mouth shape.
It can also be reproduced according to the actual pronunciation, such as the position of the tongue and the opening and closing of the mouth by both lips.
Supports both 2D / 3D models
The generated mouth pattern is output in a format that can be easily incorporated into various model data.
It supports both 2D / 3D, and realizes a richer game experience with a wide range of uses and genres.
Multilingual support independent of language
Mouth patterns can be generated in any language, not just Japanese.
There is no need to prepare dictionary data for each language in advance.
The development of titles that are deployed in multiple languages worldwide is greatly streamlined.
Supports a wide range of voice quality
It can analyze words of various voice qualities, both young and old.
Mouth patterns are generated for any voice quality, regardless of the pitch.
Features and Restrictions
Details concerning the mouth shape analysis
(1)
The Lips library will perform its analysis whenever the Atom's waveform filter callback is called (set by `criAtomExPlayer_SetFilterCallback()).
This is set up in the plugin library by calling CriWare.CriLipsAtomAnalyzer.AttachToAtomExPlayer , etc.
Nothing special needs to be done,
but some restrictions apply to the current features.
(2)
The different pieces of information that can be obtained via the Lips library are the result of independent analyzers (neural networks).
By combining these values, you can achieve a more impressive lip sync.
It is effective to supplement one analysis result with another analysis result.
(3)
You can call CriWare.CriLipsMouth.MorphTargetBlendAmountAsJapanese to play the audio for other languages than Japanese (e.g., English, Chinese).
The analysis results returned by the Lips library are in a format compatible with a character model and do not restrict the audio input.
However, the shapes of the phonemes specific to the English language cannot be returned when calling CriWare.CriLipsMouth.MorphTargetBlendAmountAsJapanese .
If English speech is analyzed via Lips based on the 5 Japanese vowels, the mouth shapes of the Japanese vowels will be used to simulate English speaking.
We recommended checking the actual application behavior (or any test data) to ensure that the results are satisfactory.
For more details about the analysis results of the Lips library, please refer to the following web page:
