Skin oxygenation level is an important indicator for the anesthesiology and psychophysiology of a wide range of skin diseases. The non-contact patient monitoring approaches rely on traditional least square method which are not accurate and can’t be deployed in clinical practices. In this paper, we exploited the power of deep learning to measure the skin oxygenation level from 16 channel spectral filter array cameras (SFA). Our architecture named SpectraNet consist of three important block i.e. a chain of Convolutional Neural Network (CNN) for feature extraction from the spectral data, an channel attention network for selecting the most informative channel selection and a bidirectional Long-Short Term Memory (LSTM) for incorporating the spatial and temporal information for estimating the final oxygenation curve from the input multi-spectral video. To show the validity of our proposed network, a clinically practiced oxygenation monitoring method (INVOS) is used as the reference. The subjective and objective evaluation shows that the techniques achieve promising results and can be deployed in the clinical practices. Moreover, due to a highly optimized nature of the proposed network, a fully trained model can be incorporated in a smartphone app for a real-time oxygenation measurement.