The 3D extension of the High Efficiency Video Coding (3D-HEVC) standard has improved the coding efficiency for 3D videos significantly. However, this improvement has been achieved with a significant rise in computational complexity. Specifically, the encoding process for the depth map in the 3D-HEVC standard occupies 84% of the total encoding time. This extended time is primarily due to the need to traverse coding unit (CU) depth levels in depth map encoding to determine the most suitable CU size. Acknowledging the evident texture distribution patterns within a depth map and the strong correlation between encoding size selection and the texture complexity of the current encoding block, an adaptive depth early termination convolutional neural network, named ADET-CNN, is designed for the depth map in this paper. It takes an original 64 × 64 coding tree unit (CTU) as the input and provides segmentation probabilities for various CU sizes within the CTU, which eliminates the need for exhaustive calculations and the comparison for determining the optimal CU size, thereby enabling faster intra-coding for the depth map. Experimental results indicate that the proposed method achieves a time saving of 58% depth map encoding while maintaining the quality of synthetic views.