For nonlinear inverse problems that are prevalent in imaging science, symmetries in the forward model are common. When data-driven deep learning approaches are used to solve such problems, such intrinsic symmetries can cause substantial learning difficulties. In this paper, we explain how such difficulties arise and, more importantly, how to overcome them by preprocessing the training set before any learning, i.e., symmetry breaking. We take the far-field Fourier phase retrieval, which is central to many areas of scientific imaging, as an example and show that symmetric breaking can substantially improve data-driven learning performance. We also formulate the principle of symmetry breaking that can lead to efficient learning.