The lived environment is the arena where our cognitive skills, preferences, and attitudes come together to determine our ability to interact with the world. The mechanisms through which lived environments can benefit cognitive health in older age are yet to be fully understood. The existing literature suggests that environments which are perceived as stimulating, usable and aesthetically appealing can improve or facilitate cognitive performance both in young and older age. Importantly, optimal stimulation for cognition seems to depend on experiencing sufficiently stimulating environments while not too challenging. Environmental complexity is an important contributor to determining whether an environment provides such an optimal stimulation. The present paper reviews a selection of studies which have explored complexity in relation to perceptual load, environmental preference and perceived usability to propose a framework which explores direct and indirect environmental influences on cognition, and to understand these influences in relation to aging processes. We identify ways to define complexity at different environmental scales, going from micro low-level perceptual features of scenes, to design qualities of proximal environments (e.g., streets, neighborhoods), to broad geographical areas (i.e., natural vs. urban environments). We propose that studying complexity at these different scales will provide new insight into the design of cognitive-friendly environments.