Patches Detection and Fusion for 3D Face Cloning.
DOI:
https://doi.org/10.14738/aivp.35.814Keywords:
Structured mesh, Patches detection and fusion, 3D Morphable Face Model, FittingAbstract
3D face clones are used in many fields such as video games and Human-Computer Interaction. However, high-resolution sensors generating high quality clones are expensive and not accessible to all. In this paper, we propose to make a fully automated and accurate 3D reconstruction of a face with a low cost RGB-D camera. For each subject, we capture the depth and RGB data of their face in different positions while performing a rotational movement of the head. We fit a 3D Morphable Face Model on each frame to eliminate noise, increase resolution and provide a structured mesh. This type of mesh is a mesh which the semantic and topological structure is known. We propose to only keep the suitable parts of each mesh called Patch. This selection is performed using an error distance and the direction of the normal vectors. To create the 3D face clone, we merge the different patches of each mesh. These patches contain relevant information on the specificity of individuals and lead to the construction of a more accurate clone. We perform quantitative tests by comparing our clone to ground truth and qualitative tests by comparing visual features. These results show that our method outperforms the FaceWarehouse process of Cao et al [2]. This 3D face clone on a structured mesh can be used as pretreatment in applications such as emotion analysis [13] or facial animation.
References
(1) Besl, P., McKay, N.D., 1992. A method for registration of 3-d shapes. Pattern Analysis and Machine Intelligence, IEEE Transactions on 14, 239–256.doi:10.1109/34.121791.
(2) Cao, C., Weng, Y., Zhou, S., Tong, Y., Zhou, K., 2014. Facewarehouse: A 3d facial expression database for visual computing. IEEE Transactions on Visualization and Computer Graphics 20, 413–425. URL: http://dx.doi.org/10.1109/TVCG.2013.249, doi:10.1109/TVCG.2013.249.
(3) Chen, Y., Medioni, G., 1992. Object modelling by registration of multiple range images. Image Vision Comput. 10, 145–155.URL: http://dx.doi.org/10.1016/0262-8856(92)90066-C,doi:10.1016/0262-8856(92)90066-C.
(4) Cui, Y., Schuon, S., Thrun, S., Stricker, D., Theobalt, C., 2013. Algorithms for 3d shape scanning with a depth camera. Pattern Analysis and Machine Intelligence, IEEE Transactions on 35, 1039–1050. doi:10.1109/TPAMI. 2012.190.
(5) Debevec, P., Hawkins, T., Tchou, C., Duiker, H.P., Sarokin, W., Sagar, M., 2000. Acquiring the reflectance field of a human face, in: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques,ACM Press/Addison-Wesley Publishing Co., New York, NY, USA.pp. 145–156. URL: http://dx.doi.org/10.1145/344779.344855,doi:10.1145/344779.344855.
(6) Fleishman, S., Drori, I., Cohen-Or, D., 2003. Bilateral mesh denoising. ACM Trans. Graph. 22, 950–953. URL: http://doi.acm.org/10.1145/882262.882368, doi:10.1145/882262.882368.
(7) Funes Mora, K.A., Odobez, J., "Gaze estimation from multimodal Kinect data," Computer Vision and Pattern Recognition Workshops (CVPRW), 2012 IEEE Computer Society Conference on , vol., no., pp.25,30, 16-21 June 2012
(8) Ralph Gross, Iain Matthews, and Simon Baker, "Generic vs. person specific active appearance models," Image and Vision Computing, Vol. 23, No. 11, November, 2005, pp. 1080-1093.
(9) Arman Savran, Neşe Alyüz, Hamdi Dibeklioğlu, Oya Çeliktutan, Berk Gökberk, Bülent Sankur, and Lale Akarun. 2008. Bosphorus Database for 3D Face Analysis. In Biometrics and Identity Management, Ben Schouten, Niels Christian Juul, Andrzej Drygajlo, and Massimo Tistarelli (Eds.). Lecture Notes In Computer Science, Vol. 5372. Springer-Verlag, Berlin, Heidelberg 47-56.
(10) Pascal Paysan, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter. 2009. A 3D Face Model for Pose and Illumination Invariant Face Recognition. In Proceedings of the 2009 Sixth IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS '09). IEEE Computer Society, Washington, DC, USA, 296-301.
(11) Catherine Soladié, Nicolas Stoiber, and Renaud Séguier. 2013. Invariant representation of facial expressions for blended expression recognition on unknown subjects. Comput. Vis. Image Underst. 117, 11 (November 2013), 1598-1609.
(12) Väljamäe, A., Larsson, P., Västfjäll, D. och Kleiner, M. (2004) Auditory Presence, Individualized Head-Related Transfer Functions, and Illusory Ego-Motion in Virtual Environments.
(13) Jun Wang; Lijun Yin; Xiaozhou Wei; Yi Sun, "3D Facial Expression Recognition Based on Primitive Surface Feature Distribution," Computer Vision and Pattern Recognition, 2006 IEEE Computer Society Conference on , vol.2, no., pp.1399,1406, 2006 doi: 10.1109/CVPR.2006.
(14) Zollhöfer, M., Thies, J., Colaianni, M., Stamminger, M., Greiner, G., 2014. Interactive model-based reconstruction of the human head using an rgb-d sensor. Journal of Visualization and Computer Animation 25, 213–222.
(15) Sun, Q., Tang, Y., Hu, P., Peng, J., 2012. Kinect-based automatic 3d high resolution face modeling, in: Image Analysis and Signal Processing (IASP), 2012 Inter
(16) Blanz, V.; Scherbaum, K.; Seidel, H.-P., "Fitting a Morphable Model to 3D Scans of Faces," Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on , vol., no., pp.1,8, 14-21 Oct. 2007
(17) Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohli, P., Shotton, J., Hodges, S., Fitzgibbon, A., 2011. Kinectfusion: Real-time dense surface mapping and tracking, in: Proceedings of the 2011 10th IEEE International Symposium on Mixed and Augmented Reality, IEEE Computer Society,Washington, DC, USA. pp. 127–136.
(18) B. Curless and M. Levoy. A volumetric method for building complex models from range images. ACM Trans. Graph., 1996
(19) S. Schuon, C. Theobalt, J. Davis, and S. Thrun. “Lidarboost: Depth super-resolution for toF 3D shape scanning,” In CVPR, pp. 343-350, 2009.
(20) Y. Cui, S. Schuon, D. Chan, S. Thrun, and C. Theobalt, “3D Shape Scanning with a Time-of-Flight Camera,” Proc. IEEE Conf. Computer Vision and Pattern Recognition, pp. 1173-1180, 2010.
(21) Michael Zollhöfer, Michael Martinek, Günther Greiner, Marc Stamminger, and Jochen Süßmuth. 2011. Automatic reconstruction of personalized avatars from 3D face scans. Comput. Animat. Virtual Worlds 22, 2-3 (April 2011), 195-202.
(22) Rusinkiewicz, S., Levoy, M., 2001. Efficient variants of the ICP algorithm, in: Third International Conference on 3D Digital Imaging and Modeling (3DIM).
(23) Cootes, T. F. and Taylor, C. J. and Cooper, D. H. and Graham, J., 1995. Active Shape Models-Theirs Training and Application. Comput. Vis.Image Underst.
