Recently, deep learning-based models have been widely applied in medical image segmentation, while those methods often focus on creating deeper and more intricate networks to enhance accuracy, ignoring the inference speed, parameter size, and computational complexity, thus these models were difficult to deploy on resource-constrained devices. In this paper, we build a novel architecture named ECPNet that trade-off well between them by using conditionally parameterized convolutions and factorized convolutions. In the encoding process, the residual module of the decomposed convolutional kernel is used to reduce the number of parameters required for extracting target features, supplement the local or global feature information that may be lost in the network layer during the feature learning process, and alleviate the gradient problem caused by stacking network layers. Moreover, the utilization of conditionally parameterized convolution, consisting of multiple expert knowledge, enables the learning of unique convolutional kernels for each channel in the encoding network, which enhances the feature representation capability of neurons. With a parameter size of only 2.02 M, this model is well suited for deployment on a variety of resource-constrained devices. Finally, extensive experiments conducted on three widely used medical datasets demonstrate that the proposed model achieves a superior balance among accuracy, parameter efficiency, memory usage, and inference speed compared to existing models.