This paper proposes a landslide detection method by UAV-based visual analysis. The fundamental strategy is to detect ground surface elevation changes caused by landslides. Our method consists of five steps: multi-temporal image acquisition, ground surface reconstruction, georeferencing, elevation data export, and landslide detection. In order to improve efficiency, we use Visual Simultaneous Localization and Mapping for ground surface reconstruction. It can perform faster than conventional methods based on Structure-from-Motion. In addition, we introduce convolutional neural network (CNN) to detect landslides robustly in the multi-temporal elevation data. The experimental results in a simulation environment show that the proposed method runs 5.5 times as fast as the conventional methods. In addition, the CNN-based model achieved F1 score of 0.79-0.84, showing robustness against reconstruction noise and registration error.