It is time-consuming and labor-intensive to detect rice diseases manually. The purpose of this research is to develop a convolutional neural networks (CNNs)-based system to automatically detect the diseased rice leaf infected with rice leaf blast, helminthosporium leaf blight, and bacterial leaf blight. The sizes of rice leaf spots vary with the severity of disease infection. A single model based CNN cannot effectively classify images, especially for images with objects of small size as well as multiple object scales, and complicated image background. In this research, a multiscale serial convolution neural network (MSSCNN) and a multiscale parallel convolution neural network (MSPCNN) are proposed to identify diseased rice leaf spots based on multi-modal fusion to extract different perception features and combine them to improve the performances obtained by using only one modality. Experimental results delineate that MSSCNN and MSPCNN can get better performance in identifying the diseased rice leaves. In MSPCNN, the features of tiny spots on diseased rice leaves can be completely preserved. MSPCNN can hence offer better performances than MSSCNN. Additionally, MSPCNN architecture is suitable for parallel computing environment.