A salient image region is defined as an image part that is clearly different from its surround. This difference is measured in terms of a number of attributes, namely, contrast, brightness and orientation. By measuring these attributes, visual saliency algorithms aim to predict the regions in an image that would attract our attention under free viewing conditions. As the number of saliency models has increased significantly in the past two decades, one is faced with the challenge of finding a metric that can be used to objectively quantify the performance of different saliency algorithms. To address this issue in this article, first, the state of the art of saliency models is revisited. Second, the major challenges associated with the evaluation of saliency models are discussed. Third, ten frequently used evaluation metrics are examined and their results are discussed for ten latest state-of-the-art saliency models. For the analysis, a comprehensive open source fixations database has been quantitatively examined.