Face recognition abilities are impacted by exposure to faces belonging to distinct categories over the lifespan. Specifically, biased exposure to faces of particular races and ages frequently leads to impaired face memory and discrimination such that faces observers do not frequently see are substantially more difficult to process effectively than faces that are closer to their typical visual experience. Here, we considered the possibility that variation in the sheer amount of faces participants see during the course of their development may also systematically impact face processing. That is, if you grow up seeing a limited set of faces, are you generally less able to process faces effectively? To examine this question, we recruited participants who grew up in very small communities and compared their behavioral and neural responses to face and object images to the responses made by participants from larger communities. We find that observers with limited face exposure do show poorer face memory and also neural responses consistent with limited face-specific processing.