Rain removal is essential for achieving autonomous driving because it preserves the details of objects that are useful for feature extraction and removes the rain structures that hinder feature extraction. Based on a linear superposition model in which the observed rain image is decomposed into two layers, a rain layer and a non-rain layer, conventional rain removal methods estimate these two layers alternatively from an observed single image based on prior modeling. However, the prior knowledge used for the rain structures is not always correct because various types of rain structures can be observed in the rain images, which results in inaccurate rain removal. Therefore, in this article, a novel rain removal method based on the use of a scribbled rain image set and a new shrinkage-based sparse coding model is proposed. The scribbled rain images have information about which pixels have rain structures. Thus, various types of rain structures can be modeled, owing to the abundance of rain structures in the rain image set. To detect the rain regions, two types of approaches, one based on reconstruction error comparison (REC) via a learned rain dictionary and the other based on a deep convolutional neural network (DCNN), are presented. With the rain regions, the proposed shrinkage-based sparse coding model determines how much to reduce the sparse codes of the rain dictionary and maintain the sparse codes of the non-rain dictionary for accurate rain removal. Experimental results verified that the proposed shrinkage-based sparse coding model could remove rain structures and preserve objects’ details due to the REC- or DCNN-based rain detection using the scribbled rain image set. Moreover, it was confirmed that the proposed method is more effective at removing rain structures from similar objects’ structures than conventional methods.