At a high level of abstraction, causally connecting population growth and environmental degradation is intuitively appealing. However, while it is clear that population size is a critical factor in the size and power of social systems, and hence in environmental impact, the relationship between human numbers and environmental change is complex. In particular, the long timescales involved in population growth and decline, along with the shifting role of economic development in both population growth itself and environmental impact, obfuscate the role of population size as a multiplier of impact. Moreover, the protracted nature of demographic change makes population size seem like an intractable problem, the outcome of natural processes which are not only beyond choice, but, critically, morally perilous. In this review of the role of population size in environmental impact, I argue that choices, norms, and values, as well as material factors, are interwoven and inseparable in the environmental impact of our species. Furthermore, the consideration of human welfare and wellbeing is central to arguments regarding an environmentally sustainable population.