TEXGen: a Generative Diffusion Model for Mesh Textures

1The University of Hong Kong, 2VAST, 3Beihang University, 4Tsinghua University
SIGGRAPH Asia 2024 (Journal Track)

Abstract

While high-quality texture maps are essential for realistic 3D asset rendering, few studies have explored learning directly in the texture space, especially on large-scale datasets. In this work, we depart from the conventional approach of relying on pre-trained 2D diffusion models for testtime optimization of 3D textures. Instead, we focus on the fundamental problem of learning in the UV texture space itself. For the first time, we train a large diffusion model capable of directly generating high-resolution texture maps in a feed-forward manner. To facilitate efficient learning in high-resolution UV spaces, we propose a scalable network architecture that interleaves convolutions on UV maps with attention layers on point clouds. Leveraging this architectural design, we train a 700 million parameter diffusion model that can generate UV texture maps guided by text prompts and single-view images. Once trained, our model naturally supports various extended applications, including text-guided texture inpainting, sparse-view texture completion, and text-driven texture synthesis.

Fast Forward

Details

Text-Condition Texture Generation

Method Overview

BibTeX

@article{10.1145/3687909,
        author = {Yu, Xin and Yuan, Ze and Guo, Yuan-Chen and Liu, Ying-Tian and Liu, Jianhui and Li, Yangguang and Cao, Yan-Pei and Liang, Ding and Qi, Xiaojuan},
        title = {TEXGen: a Generative Diffusion Model for Mesh Textures},
        journal = {ACM Trans. Graph.},
        volume = {43},
        number = {6},
        year = {2024},
        issn = {0730-0301},
        doi = {10.1145/3687909},
        articleno = {213},
        numpages = {14},
        keywords = {generative model, texture generation}
      }