For maritime surveillance, collecting information about vessels and their behavior is of vital importance. This implies reliable vessel detection and determination of the viewing angle to a vessel, which can help in analyzing the vessel behavior and in re-identification. This paper presents a vessel classification and orientation recognition system for maritime surveillance. For this purpose, we have established two novel multi-class vessel detection and vessel orientation datasets, provided to open public access. Each dataset contains 10,000 training and 1,000 evaluation images with 31,078 vessel labels (10 vessel types and 5 orientation classes). We deploy VGG/SSD to train two separate CNN models for multi-class detection and for orientation recognition of vessels. Both trained models provide a reliable F1 score of 82% and 76%, respectively.