Smile detection in the unconstrained real-world scenario has attracted much attention due to its importance for mobile applications and human computer interaction. However, in many applications, how to determine the attractiveness of a smile face image (i.e., smile elegance) is an interesting task. In this paper, we present a real-time smile elegance detection system based on feature-level fusion and support vector machine (SVM). Specifically, three types of features, including local intensity histogram (LIH), central symmetry local binary pattern (CS-LBP) and Gabor wavelet, are firstly employed to characterize the global and local relationships of the face image. Then, a feature-level fusion method is leveraged to effectively combine these features. Finally, a specific SVM classifier is learnt to classify the smile face image into elegance or inelegance. Since there is no available public smile elegance database, we establish one for the first time. Experimental results on our collected smile elegance database demonstrate the effectiveness and efficiency of the proposed smile elegance detection system.