This paper proposes a unified framework for capsule video endoscopy image enhancement with an objective to enhance the diagnostic values of these images. The proposed method is based on a hybrid approach of deep learning and classical image processing techniques. Given an input image, it is decomposed spatially into multi-layer features. We estimate the base layer with pre-trained deep edge aware filters that are learned on the flicker dataset. The detail layers are estimated by the spatio-temporal retinex-inspired envelope with a stochastic sampling technique. The enhanced image is computed by a convex linear combination of the base and the detail layers giving detailed and shadow surface enhanced image. To show its potential, performance comparison between with and without the proposed image enhancement technique is shown using several video images obtained from capsule endoscopy for different parts of the digestive organ. Moreover, different learned filters such as Bilateral and Lo norm are compared for enhancement using an objective image quality metric, BRISQUE, to show the generality of the proposed method.