To improve the driving safety triggered by driver’s behavior recognition in an in-car environment, we propose to use depth cameras mounted in a car to generate behavior models generated by a deep learning algorithm for a driver’s behavior classification. The contribution of this paper is trifold: 1) The proposed multi-view driver behavior recognition system can handle the occlusion problem happened in one of the cameras; 2) Using the recurrent neural network can effectively recognize the continuous time behavior; 3) the average recognition accuracy of proposed systems can achieve 83% and 88%, respectively.