Spectral signal recovery from RGB-images based on modern deep learning techniques demonstrated promising results in recent years and offers a feasible alternative to costly or otherwise more complex spectral imaging devices. The state-of-the-art deep learning is formed by approaches that learn a direct end-toend mapping from RGB to spectral images from given RGB and spectral image pairs. Any prior knowledge, most importantly a known spectral responsivity of the imaging device, is not taken into account by the vast majority of deep learning based methods. Although attempts have been made to include prior knowledge with respect to the camera response functions, it remains unclear how to do so in a robust and constructive way. In this work, we propose a hybrid processing method utilizing a handcrafted linear map to directly obtain a good estimate on the spectral signal. Deep learning is only used for a subsequent signal refinement. In contrast to previous work, our linear estimate on the spectral signal is not subject to any network optimization and relies on explicit knowledge on the camera response. It is finally demonstrated that the proposed hybrid processing strategy reduces spectral reconstruction errors.