Researchers from Northeastern University and Google Research have developed a new model called OmniControl to tackle the challenge of combining spatial control signals in human motion production. While current diffusion-based techniques can generate lifelike human motion, they struggle to incorporate variable spatial control signals, which are important for many applications. For example, a model needs to accurately position the hand to pick up an object or adjust the head height to avoid accidents in a room with low ceilings.
To address this issue, researchers propose OmniControl, which allows flexible spatial control signals for any joint at any given time. The model also includes realism guidance to ensure realistic and consistent movements. By converting the relative motion produced by the model to global coordinates, they can compare it directly with the input control signals and make necessary adjustments. This overcomes the limitations of previous approaches and improves control precision.
However, using spatial guidance alone can result in drifting issues and abnormal movements. To address this, researchers introduce realism guidance, which modifies the whole-body motion to produce realistic and coherent movements. Both spatial and realism guidance are essential for achieving control precision and motion realism.
The effectiveness of OmniControl is demonstrated through experiments using HumanML3D and KIT-ML datasets. The model outperforms other text-based motion generation techniques in terms of motion realism and control accuracy, particularly in controlling the pelvis. Additionally, OmniControl allows for controlling multiple joints collectively, making it suitable for various applications such as integrating human motion with surrounding scenery and objects.
In summary, OmniControl is a groundbreaking approach that combines flexible spatial control signals and realism guidance to produce realistic and controlled human motion. This model sets a new standard in motion generation and opens up possibilities for various applications. Researchers have provided a detailed paper and project for further exploration.