In this article, we propose a knowledge-based taxonomic scheme of the objective image quality assessment metrics including the key concepts involved for each approach. Our classification is constructed according to six criteria based on the information available at each stage of the design process. The novelty of the present classification scheme is that the six layers are linked via a single concept where each layer represents a single type of knowledge about: 1) the reference image, 2) the degradation type, 3) the visual perception field, 4) the human visual physiology and psychophysical mechanisms, 5) the processes of the visual information analysis, and finally 6) knowledge about perceptual image representation and coding. The first layer helps delineate boundaries between full-reference (FR) image quality assessment metrics, that are further classified through layers 2–6, and other families (reduced-reference [RR] and no-reference [NR]). In addition, gradual degrees are considered for knowledge about specific areas related to visual quality evaluation processes. The proposed taxonomic framework is intended to be stepwise, to help sorting out the fundamental ideas behind the development of objective image quality metrics often working on the luminance channel or marginally on the RGB channels. The aim is to congregate the already published classification schemes and to methodologically expand new aspects according to which an efficient and straightforward classification of the image quality assessment algorithms becomes possible. This is significant because of the increasing number of developed metrics. Furthermore, a systematic summarization is necessary in order to facilitate the research and application of image quality techniques.