Abstract

This work studies learning from a synergy process of 3D Morphable Models (3DMM) and 3D facial landmarks to predict complete 3D facial geometry, including 3D alignment, face orientation, and 3D face modeling. Our synergy process leverages a representation cycle for 3DMM parameters and 3D landmarks. 3D landmarks can be extracted and refined from face meshes built by 3DMM parameters. We next reverse the representation direction and show that predicting 3DMM parameters from sparse 3D landmarks improves the information flow. Together we create a synergy process that utilizes the relation between 3D landmarks and 3DMM parameters, and they collaboratively contribute to better performance. We extensively validate our contribution on full tasks of facial geometry prediction and show our superior and robust performance on these tasks for various scenarios. Particularly, we adopt only simple and widely-used network operations to attain fast and accurate facial geometry prediction.

Framework

The framework is shown as follows.

Advantage

👍 SOTA on all 3D facial alignment, face orientation estimation, and 3D face modeling.

👍 Fast inference with 3000fps on a laptop RTX 2080.

👍 Simple implementation with only widely used operations.

Facial Alignment on AFLW2000-3D (NME of facial landmarks):

Face orientation estimation on AFLW2000-3D (MAE of Euler angles):

3D face modeling (Point-to-plane RMSE)

Overall Results

Acknowledgement

We thank Jingjing Zheng, Jim Thomas, and Cheng-Hao Kuo for their concrete comments and advice on this work.