In recent years, with the rapid development of stereoscopic display technology, its applications have become increasingly popular in many fields, and, meanwhile, the number of audiences is also growing. The problem of visual fatigue is becoming more and more prominent. Visual fatigue is mainly caused by vergence–accommodation conflicts. An evaluation experiment was conducted, and the electroencephalogram (EEG) data of the subjects were collected when they were watching stereoscopic content, and then the stereoscopic fatigue state of the subjects during the viewing process was analyzed. As deep learning is proved to be an effective end-to-end learning method and multi-task learning can alleviate the problem of lacking annotated data, the authors establish a user visual fatigue assessment model based on EEG by using multi-task learning, which can effectively obtain the user’s visual fatigue status, so as to make the comfort designs to avoid the harm caused by user’s visual fatigue.