We hypothesize that the semantics of image content affects how humans judge the perceptual quality of images. The recognition of image content has been shown to be processed within the first 500 ms of observation (and mostly in a pre-attentive stage). We look at whether or not participants are also able to detect impairments and judge their annoyance at early attentive stages. As the presence of impairments may slow down the early semantic recognition process, we investigate whether or not different semantic content impacts people’s judgment of image quality. Our results show that participants do recognize image content despite the presence of impairments even at very early stages of vision (within the first fixation). In addition, we show that semantic categories have an influence on people’s detection of image impairments at early attentive stages. People seem to be able to correctly detect very obvious impairments within one fixation, but more subtle impairments are not perceived. Finally, we show that people are more tolerant toward impairments on images portraying outdoor scenes than images portraying indoor scenes; additionally, users seem to be more critical toward images containing animate objects (humans or animals) in the region of interest compared with those with inanimate objects.