This study focuses on real-time pedestrian detection using thermal images taken at night because a number of pedestrian–vehicle crashes occur from late at night to early dawn. However, the thermal energy between a pedestrian and the road differs depending on the season. We therefore propose the use of adaptive Boolean-map-based saliency (ABMS) to boost the pedestrian from the background based on the particular season. For pedestrian recognition, we use the convolutional neural network based pedestrian detection algorithm, you only look once (YOLO), which differs from conventional classifier-based methods. Unlike the original version, we combine YOLO with a saliency feature map constructed using ABMS as a hardwired kernel based on prior knowledge that a pedestrian has higher saliency than the background. The proposed algorithm was successfully applied to the thermal image dataset captured by moving vehicles, and its performance was shown to be better than that of other related state-of-the-art methods.