Traditional depth from focus (DFF) methods obtain depth image from a set of differently focused color images. They detect in-focus region at each image by measuring the sharpness of observed color textures. However, estimating sharpness of arbitrary color texture is not a trivial task especially when there are limited color or intensity variations in an image. Recent deep learning based DFF approaches have shown that the collective estimation of sharpness in a set of focus images based on large body of training samples outperforms traditional DFF with challenging target objects with textureless or glaring surfaces. In this article, we propose a deep spatial–focal convolutional neural network that encodes the correlations between consecutive focused images that are fed to the network in order. In this way, our neural network understands the pattern of blur changes of each image pixel from a volumetric input of spatial–focal three-dimensional space. Extensive quantitative and qualitative evaluations on existing three public data sets show that our proposed method outperforms prior methods in depth estimation.