Neural Radiance Fields (NeRF) is a powerful AI technique that captures 3D scenes and objects from 2D images or sparse 3D data. It consists of two main components: the “NeRF in” and the “NeRF out” network. The “NeRF in” network takes the 2D coordinates of a pixel and the camera pose as input and produces a feature vector. The “NeRF out” network uses this feature vector to predict the 3D position and color of the corresponding 3D point.
NeRF-based human representations can be created by capturing images or videos of a human subject from different viewpoints. These images can come from cameras, depth sensors, or other 3D scanning devices. HumanNeRF has many applications including virtual avatars for games and virtual reality, 3D modeling for animation and film production, and medical imaging for creating 3D models of patients.
However, creating a NeRF-based human representation can be computationally intensive and requires a lot of training data. To address this, researchers have proposed a new method called ActorsNeRF. It is a category-level human actor NeRF model that can generate high-quality views of novel actors with only a few images. ActorsNeRF uses a 2-level canonical space method and a skinning weight network to achieve realistic character movements and deformations.
To achieve generalization across different individuals, the researchers trained the category-level NeRF model on a diverse set of subjects. During the inference phase, they fine-tuned the model using a few images of the target actor, allowing it to adapt to specific characteristics.
The results show that ActorsNeRF outperforms the HumanNeRF approach by maintaining valid shape for the less observed body parts. It can synthesize unobserved portions of the body smoothly. ActorsNeRF has been tested on multiple benchmarks and has shown superior performance in generating novel human actors with unseen poses.
For more information, you can check out the paper and the project page. Credit goes to the researchers on this project. Don’t forget to join our ML Subreddit, Facebook Community, Discord Channel, and Email Newsletter for the latest AI research news and cool AI projects.