As applications of drone proliferate, it has become increasingly important to equip drones with automatic sense and avoid (SAA) algorithms to address safety and liability concerns. Sense and avoid algorithms can be based upon either active or passive sensing methods. Each of them has advantages when compared to the other but neither is sufficient by itself. Therefore, especially for application such as autonomous navigation where failure could be catastrophic, deploying both passive and active sensors simultaneously and utilizing inputs from them become critical to detect and avoid objects in a reliable way. As part of the solution, in this paper, we present an efficient SAA algorithm based on input from multiple stereo cameras, which can be implemented on a low-cost and low-power embedded processor. In this algorithm, we construct an instantaneous 3D occupancy grid (OG) map at each time instance using the disparity information from the stereo cameras. Then, we filter noise using spacial information, and further filter noise using a probabilistic approach based on temporal information. Using this OG Map, we detect threats to the drone in order to determine the best trajectory for it to reach a destination.