Estimating optical flow presents unique challenges in AV applications: large translational motion, wide variations in depth of important objects, strong lens distortion in commonly used fisheye cameras and rolling shutter artefacts in dynamic scenes. Even simple translational motion can produce complicated optical flow fields. Lack of ground truth data also creates a challenge. We evaluate recent optical flow methods on fisheye imagery found in AV applications. We explore various training techniques in challenging scenarios and domain adaptation for transferring models trained on synthetic data where ground truth is available to real-world data. We propose novel strategies that facilitate learning robust representations efficiently to address low-light degeneracies. Finally, we discuss the main challenges and open problems in this problem domain.