Accurate classifications of colorectal cancer (CRC) lymph node metastasis (LNM) could assist radiologists in increasing the diagnostic accuracy and help surgeons establish a correct surgical plan. This study aims to present an efficient pipeline with deep transfer learning for CRC LNM classification. Hence, 11 deep pre-trained models have been investigated on a CRC LN dataset. The dataset of this experiment is from Harbin Medical University Cancer Hospital. This dataset contains samples of 619 patients. Among these samples, 312 were positive and 307 were negative. In addition, datasets with different dimensions and various training epochs were also studied to ascertain the minimum training dataset and training times. In order to improve the interpretability of the model classification performance, a visual convolution layer feature map was first established to compute the similarity distance between the feature map and original data. The experimental results revealed that resnet_152 was the best deep pre-trained model for the classification of CRC LNM, with an accuracy of 97.2%, with 600 raw data samples being the minimum dimension of a dataset and 30 epochs the minimum training times in the CRC LNM classification. This study suggests that the proposed deep transfer learning pipeline could classify the CRC LNM with high efficiency, without requiring sophisticated computational knowledge for radiologists.