Mouth pattern information that can be acquired with ADX LipSync

Available Mouth Patterns

ADX LipSync provides three parameter types for the mouth patterns output as the analysis results.
They can be used based on the mouth control patterns of the character.

Mouth Shape Information (CriLipsMouth.Info)

CriWare.CriLipsMouth.Info
The height of the mouth, the width of the mouth, and the height of the tongue.

/* Mouth Shape Information Structure */
public struct CriLipsMouth.Info {
    public float lipWidth;          /* Mouth Width (0.0f - 1.0f)                      */
    public float lipHeight;         /* Mouth Height (0.0f - 1.0f)                    */
    public float tonguePosition;    /* Tongue Height (0.0f - 1.0f)                    */
    bool isLipWidthReleased;        /* Whether the mouth width is transitioning to the closed state                */
    bool isLipHeightReleased;       /* Whether the mouth height is transitioning to the closed state              */
    bool isLipToungueReleased;      /* Whether the tongue position is transitioning to the closed state              */
}

It is suitable for use in models that control the mouth with direct parameters such as height and width, and when expressing the mouth shape by scaling simple figures.
Also, if you want to express a mouth motion with only one axis of mouth openness,
you can achieve this by using only the vertical openness so that "A" opens greatly but "I" does not open much.

Tongue height information (vertical position) can also be acquired.
0.0f in silent state and 1.0f in tongue attached to gums.
By controlling the tongue as well as the movement of the mouth, more realistic character expression is possible.

Attention

The output of the "tongue height" parameter is a beta version feature under development.
The quality of analysis results is not guaranteed. In addition, changes in specifications and operations may occur in future updates.
Please use it after understanding that it is beta version function.

Note:

In the silent state, the mouth shape information of 0.0f in the vertical and 0.583f in the horizontal width is output. This number can be obtained by CriLipsMouth.GetInfoAtSilence.
The width increases or decreases based on the value of the silent state, such as increasing when "I" is pronounced and decreasing when "U" is pronounced.
Therefore, when referring to CriLipsMouth.Info directly on the application side, please convert the numerical value according to the data structure of the character.

The mouth shape information is adjusted based on the model that opens the mouth considerably at the maximum value like the Face_Sample model included in this sample.

Face_Sample Maximum value of LipHeightOpen model

For this reason, depending on the maximum mouth size of the model to be controlled, if the value of lipHeight is used as is, the mouth may not open too much and the movement may appear like muttering.

For example, by using the log function, you can process the value obtained from CriLipsMouth.Info.lipHeight and perform conversion so that it will be a value suitable for your model, which can create a more natural expression.

1-Dimensional Mouth Shape Information (CriLipsMouth.OpenInfo)

CriWare.CriLipsMouth.OpenInfo
1-dimensional mouth shape information

/* 1-dimensional Mouth Shape Information Structure */
public struct CriLipsMouth.OpenInfo {
    /* 1-dimensional lip-sync mouth shape open ratio (0f ~ 1.0f) */
    public float openY;
}

For either 2D or 3D characters, the mouth shape information is the choice to generate lipsync if the character only has one pattern of mouth shape (morph target) or the mouth's opening is controlled with a single parameter.
Lip-sync is performed via the opening and closing of the mouth for each sound.

The value of CriLipsMouth.Info.lipHeight is used internally to synchronize the mouth movement to be more realistic.
This is the intermediate mouth shape movement.
Therefore, with this 1-dimensional mouth shape information, better lip-sync can be achieved for unrealistic characters (e.g., characters designed without a chin) .

It is also effective for character models with fixed facial expressions and morph targets such as a "smiling" face without the Japanese 5 vowels (A, I, U, E, O), or those based on volume values (such as RMS).

Note:

This value is based on CriWare.CriLipsMouth.MorphTargetBlendAmountAsJapanese, which combines multiple mouth movements to be more realistic.
Therefore, the following restrictions apply:

• Because of the smoothing/blending, it takes longer to transition to silent state than with Info or MorphTargetBlendAmountAsJapanese .
• The silent state of OpenInfo is influenced by CriWare.CriLipsMouth.IsAtSilenceForMouthOpen .
  

Japanese 5 Vowel Morph Target Blend Quantity (CriLipsMouth.MorphTargetBlendAmountAsJapanese)

CriWare.CriLipsMouth.MorphTargetBlendAmountAsJapanese
The blend amount of Japanese vowels ("AIUEO").

/* Japanese 5 Vowel Morph Target Blend Quantity Structure */
public struct CriLipsMouth.MorphTargetBlendAmountAsJapanese {
    public float a;                 /* Blend amount of morph target "A" (0.0f - 1.0f) */
    public float i;                 /* Blend amount of morph target "I" (0.0f - 1.0f) */
    public float u;                 /* Blend amount of morph target "U" (0.0f - 1.0f) */
    public float e;                 /* Blend amount of morph target "E" (0.0f - 1.0f) */
    public float o;                 /* Blend amount of morph target "O" (0.0f - 1.0f) */
}

This information is suitable for control of models with blend shapes of the 5 Japanese vowels.
Depending on the utterance, the blend amount of the two vowels at the same time will increase or decrease within a range of 1.0f or less.
By combining shapes that express two vowels, a natural mouth shape can be expressed even for non-Japanese speech.

Combination of multiple mouth patterns

The mouth shape information and the Japanese 5 vowel morph target blending can be obtained in a single analysis.
This allows you to use a combination of different patterns, such as "AIUEO + tongue height".

Next: Facial mouth shape suitable for ADX LipSync