An approach is proposed for the reliable appearance reconstruction of fluorescent objects under arbitrary conditions of material and illuminant based on reference geometric factors. First, a large set of spectral images is acquired from a variety of scenes of fluorescent objects paired with a mutual illumination effect under different conditions. The target fluorescent object is constructed using a cube and a flat plate supporting it, and is subsequently illuminated using a directional light source. We produce many target objects of the same size with different fluorescent materials and observe them under different illumination conditions. The observed spectral images are subsequently decomposed into five components, combining the spectral functions and geometric factors. The reference geometric factors are independent of the material, illuminant, and illumination direction change; instead, they are only dependent on object geometries. A reliable estimation method of reference geometric factors is presented using the whole spectral images observed under various conditions. Further, we propose an algorithm for reconstructing a realistic appearance including mutual illumination effect under arbitrary conditions of material, illuminant, and illumination direction. Finally, the reliability of the proposed approach is examined experimentally.