Back to articles
Volume: 33 | Article ID: art00005
GG-Net: Gaze Guided Network for Self-driving Cars
  DOI :  10.2352/ISSN.2470-1173.2021.17.AVM-171  Published OnlineJanuary 2021

Imitation learning is used massively in autonomous driving for training networks to predict steering commands from frames using annotated data collected by an expert driver. Believing that the frames taken from a front-facing camera are completely mimicking the driver’s eyes raises the question of how eyes and the complex human vision system attention mechanisms perceive the scene. This paper proposes the idea of incorporating eye gaze information with the frames into an end-to-end deep neural network in the lane-following task. The proposed novel architecture, GG-Net, is composed of a spatial transformer network (STN), and a multitask network to predict steering angle as well as the gaze map for the input frame. The experimental results of this architecture show a great improvement in steering angle prediction accuracy of 36% over the baseline with inference time of 0.015 seconds per frame (66 fps) using NVIDIA K80 GPU enabling the proposed model to operate in real-time. We argue that incorporating gaze maps enhances the model generalization capability to the unseen environments. Additionally, a novel course-steering angle conversion algorithm with a complementing mathematical proof is proposed.

Subject Areas :
Views 57
Downloads 16
 articleview.views 57
 articleview.downloads 16
  Cite this article 

M. Abdelkarim, M.K. Abbas, Alaa Osama, Dalia Anwar, Mostafa Azzam, M. Abdelalim, H. Mostafa, Samah El-Tantawy, Ibrahim Sobh, "GG-Net: Gaze Guided Network for Self-driving Carsin Proc. IS&T Int’l. Symp. on Electronic Imaging: Autonomous Vehicles and Machines,  2021,  pp 171-1 - 171-8,

 Copy citation
  Copyright statement 
Copyright © Society for Imaging Science and Technology 2021
Electronic Imaging
Society for Imaging Science and Technology
IS&T 7003 Kilworth Lane, Springfield, VA 22151 USA