In this paper, we introduce a multi-pedestrian tracking algorithm for tracking from a moving vehicle. The method is based on online learning of a random ferns (RF) tracker model using the output features of a convolutional neural network (CNN). For real-time application in vehicles, an online method is applied within the tracking-by-detection framework where data association between detections and trackers is conducted online. To predict the tracker's position, we perform particle filtering with tracker models inferred from a shallow CNN. In this study, You Only Look Once (YOLO), a real-time object detection system, was adopted as the pre-trained model. Although YOLO has an accurate network for object classification, it is not appropriate for real-time multi-pedestrian tracking. Therefore, we use modified YOLO to obtain a shallow version (S-YOLO) having fewer convolutional layers and fewer filters in these layers. To update the tracker in every frame, positive and negative samples are applied to the S-YOLO and retraining is performed. Then, we extract feature descriptors from the first fully connected layer of S-YOLO to train the RF tracker models. The proposed algorithm was successfully applied to various pedestrian video sequences and yielded a more accurate tracking performance than other existing method.