The development of 3D assets is important for various commercial applications like gaming, cinema, and AR/VR. However, the traditional process of creating 3D assets is time-consuming and requires specialized knowledge. With recent advancements in AI technology, text-to-3D pipelines have emerged as a way to automatically generate 3D assets from textual descriptions, reducing the time and skill requirements.
Improving Material Generation
However, one challenge with text-to-3D pipelines is accurately capturing object materials. Existing techniques struggle to restore high-fidelity materials, limiting their use in real-world applications like relighting. Previous attempts to model material properties have not been successful in identifying appropriate materials based on natural distribution, especially in fixed light conditions.
The MATLABER Approach
To address this issue, researchers from Shanghai AI Laboratory and S – Lab, Nanyang Technological University, have developed a new approach called Material-Aware Text-to-3D through LAtent BRDF auto EncodeR (MATLABER). This approach uses rich material data, such as BRDF material datasets, to learn a unique text-to-3D pipeline that separates material from ambient lighting and creates realistic materials that match text prompts.
MATLABER uses a latent BRDF auto-encoder that predicts BRDF latent codes instead of BRDF values. The auto-encoder is trained with real-world BRDF priors, allowing MATLABER to focus on selecting the most appropriate material. This approach ensures the realism and coherence of object materials and produces high-quality 3D assets.
Benefits and Applications
Accurate material estimation enables activities like scene modification, material editing, and relighting, which were previously difficult to perform. This opens up possibilities for more practical 3D content generation. Additionally, MATLABER can also infer tactile and sonic information from acquired materials, enhancing the overall virtual experience.
Overall, the MATLABER approach improves the efficiency and quality of 3D asset generation in text-to-3D pipelines. It provides a solution for accurately capturing object materials and opens up new possibilities for practical applications in various industries.