In recent years, buildings in urban areas are frequently being demolished for a variety of reasons. For example, demolitions happen when buildings are rebuilt because the building was sold during an asset sale by a corporation in financial difficulties, because the building was built during the period of high economic growth and was aging, or because the building was damaged in a natural disaster, which happens frequently in recent years. However, construction waste is still being sorted by hand. Therefore, it is desired to reduce labor costs and to improve safety of workers. In order to overcome these issues, this study investigated methods for automatically recognizing waste materials. However, these methods had several problems, such as low recognition accuracy and an inability to handle metals, such as iron. In this research, we propose a method for automatically recognizing waste materials using sensor fusion. In the proposed method, information regarding the color, brightness, and shape of the object is acquired from images obtained using imaging sensors. In addition, we also focus on differences in the thermal conductivity of different materials and use a thermal sensor to measure the temperature of the target object to obtain thermal information. We performed a material recognition experiment in which only camera images were used, and a material recognition experiment in which sensor fusion was used. The results show that the recognition accuracy was approximately 10% higher overall in the experiment conducted using the latter method compared to the experiment conducted using the former method. These results show that the proposed method is effective.