Regular observation and recording of the changes in body appearance are essential for the process of the treatment of plastic surgery and dermatology, especially aesthetic surgery. Usually, physicians treat patients with medical interviews, pictures of the patient's faces before and after their treatment, anatomical data that including size, location, and color of the affected skin. However, it is difficult to capture the affected area under the same conditions every time because the captured range varies depending on the imaging angle and distance. There is a need to record three-dimensional shape of face parts such as cheek, nose, eye, and chin. Therefore, in this study, the face shape and the skin color were measured using the infrared depth camera and the RGB camera built in the smartphone three-dimensionally. We measured before and after modulating the shape and color of the face, and then, the change in volume and the change in skin pigment of skin color was calculated and visualized. This method makes it possible to analyze the skin shape and color independently of the viewing angle and the illumination direction. In this study, the depth sensor built in the smartphone showed the potential to monitor changes in facial shape and skin color. In the future, it is expected to contribute to the development of telemedicine, in which the patient measures their face at home and gets medical treatment consultation remotely.