The past few years have witnessed the impressive performance of sparse representation based classification (SRC) for visual recognition. However, the SRC technique may lead to high residual error and poor performance due that the training samples in each class contribute equally to the dictionary in the corresponding class. This inspired the emergence of class specific dictionary learning algorithm. In this paper, we propose a novel approach—class specific dictionary learning combined with linear discriminant analysis constraints in Reproducing Kernel Hilbert Space (KCSDL-LDA), which modifies and extends the conventional class specific dictionary learning (CSDL) algorithm in several aspects. First, we propose a novel class specific dictionary learning scheme that considers the weight of each sample for each class when generating the dictionary in that class. Second, we extend the novel class specific dictionary learning scheme to the Reproducing Kernel Hilbert Space, in which nonlinear structure can be extracted and represented to improve the classification accuracy. Finally, we further enhance the classification performance by combing class specific dictionary learning with linear discriminant analysis constraints in Reproducing Kernel Hilbert Spaces. Extensive experimental results on several face recognition benchmark datasets, such as Extended YaleB dataset, CMU PIE dataset and AR dataset, demonstrate the superior performance of our proposed KCSDL-LDA.