Mercury (Hg) and Arsenic (As) have been recognized as chemical threats to human health. Still, the detection of lower contamination levels using traditional image analysis remains challenging due to the small number of available data samples and the insufficient utilization of the spatial information contained in the sensor pad images. To overcome this challenge, we use the spectra data of the colorimetric response pads and propose two kinds of classification models for differentiating contaminant levels with high test accuracy. In the first model, we use the SMOTE method to solve the imbalanced data problem, then apply the sequential forward selection algorithm to select optimal wavelength features in combination with the k-NN classifier to discriminate five contaminant levels. The second technique comprises principal component analysis (PCA) used as a dimensionality reduction technique combined with the random forest (RF) classifier to classify five contaminant levels. Our proposed system is trained and evaluated on a limited dataset of 126 spectral responses of five contamination levels. Our algorithms can yield 77% and 87% average accuracy, respectively. We will present an overview of the base model, the pipelines and the comparison of our proposed two classification models, and the phone-based narrow-band spectral imaging system that can obtain the camera spectral response for accurate and precise heavy metals analyses with the aid of narrow bandpass filters in front of a cell phone’s camera lens.