The cross-modal association is a specific connection experienced between stimuli, attributes, or perceptual dimensions. Most previous studies related to the cross-modal association of color and temperature use subjective report methods, and the underlying mechanisms behind this association are unclear. This study used the Implicit Association Test (IAT) paradigm combined with event-related potentials (ERPs) elicited by color stimuli to explore the cross-modal association between color, in terms of hue, saturation, and brightness, and temperature. Cross-modal associations between hue and temperature, as well as saturation and temperature were found. There was no evidence for the existence of a brightness-temperature cross-modal association. Additionally, significant differences in the average N1 amplitudes were observed in the hue IAT task. Significant differences in the average N2 and N400 amplitudes were observed in both the hue and saturation IAT tasks, indicating that both corresponding cross-modal associations have a semantic basis to some degree.