In the present paper, we propose a technique to control the appearance on an object in real time using a high-luminance PC projector and graphics hardware. We have previously proposed an image projection technique to reproduce the appearance of a real object on a mock object using a high-luminance projector. By controlling the projected image, the reflected radiance on the mock object is matched to that on the real object. However, in our previous study, only preliminary experiments were performed by matching the appearance empirically. The present paper uses ray tracing to reproduce the distribution of the reflected radiance for appearance matching. It is necessary to measure the bi-directional reflectance distribution function of the objects and the geometry between the projector and the observer's eyes. Since the observer's eyes move with the head in evaluating the appearance of the object, we performed a real-time reproduction of gloss appearance with the movement of the observer's position. The observer's position is detected by an electromagnetic position sensor. Graphics hardware is used in the real-time reproduction to calculate the ray tracing at high speed and render the appearance according to the observer's eye position in real time. Observer rating revealed that little difference in appearance was perceived between the real object and the projected mock object.