In this research, we propose a method to assess images of the tongue captured using a polarized light camera for diagnostic use in Kampo Medicine. The polarized light camera is used to simultaneously capture glossy and non-glossy images of the tongue. Data augmentation was performed by modulating the color and gloss, through which the number of images was increased from 11 to 275. A diagnostic assistance module was built to evaluate a given disease by learning a specialist's assessment of it along with feature values obtained from the captured image using a machine learning technique. The resulting mean absolute error of the assessment of five diseases was sufficiently small for it to be accurate.