Understanding multimodal interaction and its effects on user experience and behavior is becoming increasingly important with the rapid development of immersive mixed-reality applications. While significant attention has been devoted to enabling haptic feedback in multisensory extended reality systems, research on how the visual and tactile properties of materials and objects interact is still limited. This study investigates how the color appearance of texture images affects observers' judgments of different tactile attributes. For this purpose, we captured images of different texture samples and manipulated the color of the images based on the previously reported consistent mapping of tactile descriptors onto color space. The observers were asked to rate these textures on different haptic properties to test the effects of color on their perception of materials. We found that the effect of changing color is most significant for the perception of heaviness, warmness, and naturalness of textures. For these attributes, a strong correlation between ratings of textures and ratings of uniform color patches of similar colors was also observed, while other attributes, such as hardness, dryness, or pleasantness, showed low or no correlation. The results can increase our understanding of the role of color as a visual cue in estimating material properties of visual textures and for designing and rendering surface properties of objects in 3D printing and virtual and augmented reality applications, including online shopping and gaming.