This work compares the material classification performance of Mueller matrix polarization imaging to RGB imaging. White painted wood and white fabric samples are selected to create a classification task that is challenging for RGB imaging. A Mueller Matrix Imaging Polarimeter with a 30° full field of view is used to capture the Mueller Matrix images at nominal red, green, and blue wavelengths across multiple specular scatter angles. A Bayesian ideal observer model is used to evaluate classification performance. Performance is quantified by the Area under (AUC) the Receiver Operating Characteristic (ROC) curve. An AUC = 1 is perfect detection and AUC = 0.5 is the performance of guessing. The ensemble average AUC does not exceed 0.70 for RGB irradiance data. The ensemble average AUC for all 16 individual Mueller elements is greater than 0.95. Various combinations of Mueller matrix elements are also tested. Elements related to diattenuation and polarizance are nearly perfect classifiers for large scatter angles but the AUC minimum is 0.60 at 20°. Depolarization index is the highest performing parameter out of all tested polarization parameters for scatter angles ≥70° where AUC ≥0.98.