Abstract The color of a print is affected by ink spreading and lateral light scattering in the substrate, making printed dots appear larger. Characterization of physical and optical dot gain is crucial for the graphic arts and paper industries. We propose a novel approach to separate physical from optical dot gain by use of a high-resolution camera. This approach is based on the histogram of microscale images captured by the camera. Having determined the actual physical dot shape, we estimate the modulation transfer function (MTF) of the paper substrate. The proposed method is validated by comparing the estimated MTF of 11 offset printed coated papers to the MTF obtained from the unprinted papers using measured and Monte Carlo simulated edge responses.