The authors present a novel method, Hierarchical Manifold Sensing, for adaptive and efficient visual sensing. As opposed to the previously introduced Manifold Sensing algorithm, the new version introduces a way of learning a hierarchical partitioning of the dataset based on k-means clustering. The algorithm can perform on whole images but also on a foveated dataset, where only salient regions are sensed. The authors evaluate the proposed algorithms on the COIL, ALOI, and MNIST datasets. Although they use a very simple nearest-neighbor classifier, on the easier benchmarks, COIL and ALOI, perfect recognition is possible with only six or ten sensing values. Moreover, they show that their sensing scheme yields a better recognition performance than compressive sensing with random projections. On MNIST, state-of-the-art performance cannot be reached, but they show that a large number of test images can be recognized with only very few sensing values. However, for many applications, performance on challenging benchmarks may be less relevant than the simplicity of the solution (processing power, bandwidth) when solving a less challenging problem. © 2016 Society for Imaging Science and Technology. [DOI: 10.2352/J.ImagingSci.Technol.2016.60.2.020402]