To improve the efficiency and accuracy of Content Based Image Retrieval (CBIR) for specific images, a new method is presented in the paper. The method focuses on 3 key problems. Firstly, considering the impaction of saliency point near the attention focus, an improved saliency region extraction algorithm is proposed to locate object of interest more accurately. Then, the construction of Bag-of-Features (BoF) feature vector is improved by our visual attention model to extract features more effectively. Finally, Particle Swarm Optimization (PSO) is introduced to optimize the learning process of the feedback model based on Support Vector Machine (SVM) to boost the accuracy and efficiency of the image retrieval. Experiments and comparison between typical algorithms based on Caltech 101 dataset and self-collection dataset demonstrate that the method proposed in this paper can improve the accuracy and efficiency of content based image retrieval.