In this paper, we propose the method of emotion monitoring using physiological signals such as RR intervals and blood volumes obtained by analyzing hemoglobin concentration from facial color images. The emotion monitoring has a great potential in areas such as market research, safety measure, medical, and robot systems. Recently, the most popular method of emotion monitoring is by using physiological signals. However, the previous studies are not practical due to using special instruments such as electrodes or laser speckle flowgraphy to obtain physiological signals. Therefore, the proposed method uses simple RGB camera. Using 27 kind of features calculated from the physiological signals from the camera, we classified five different emotional states (amusement, anger, disgust, sadness and surprise). As a result, we could classify emotions with 94% accuracy by selecting features.