Object detection and tracking is an indispensable module in airborne optoelectronic equipment, and its detection and tracking performance is directly related to the accuracy of object perception. Recently, the improved Siamese network tracking algorithm has achieved excellent results on various challenging data sets. However, most of the improved algorithms use local fixed search strategies, which cannot update the template. In addition, the template will introduce background interference, which will lead to tracking drift and eventually cause tracking failure. In order to solve these problems, this article proposes an improved fully connected Siamese tracking algorithm combined with object contour extraction and object detection, which uses the contour template of the object instead of the bounding-box template to reduce the background clutter interference. First, the contour detection network automatically obtains the closed contour information of the object and uses the flood-filling clustering algorithm to obtain the contour template. Then, the contour template and the search area are fed into the improved Siamese network to obtain the optimal tracking score value and adaptively update the contour template. If the object is fully obscured or lost, the YoLo v3 network is used to search the object in the entire field of view to achieve stable tracking throughout the process. A large number of qualitative and quantitative simulation results on benchmark test data set and the flying data set show that the improved model can not only improve the object tracking performance under complex backgrounds, but also improve the response time of airborne systems, which has high engineering application value.