With the development of technology, medical image examination methods are steadily diversified, including ultrasound, pathology, endoscopy, computed tomography, computed radiography, magnetic resonance imaging, radionuclide imaging, positron emission computed tomography, angiography, etc. More than 70% of clinical diagnoses depends on medical imaging. Owning to the fact that medical image data grows faster, the problem of lack of radiodiagnosis doctors are more serious. Although the accuracy of doctors’ diagnosis is high, the accuracy of judgment is questionable due to energy constraints, emotional fluctuations and other factors. To assist accurate and effective diagnosis and reduce the probability of misdiagnosis, in this paper, a novel end-to-end model with an encoder network and a decoder network (MS-UNet) is proposed. The encoder network extracts feature information from the input image, and the decoder network restores the encoded content to the input size. Among them, a multi-scale module (MSM) is built to help extract more accurate information that represents the feature maps of objective segmentation by fusing multiple different convolution information at different stages. The experimental results on two well-known data sets show that the proposed model outperforms other state-of-the-art algorithms, and the introduction of MSM can significantly improve performance.