We investigate the variation of the band gap across the Ruddlesden-Popper (RP) series (An+1BnX3n+1) in model chalcogenide, oxide, and halide materials to understand the factors influencing band gap evolution. In contrast to the oxides and halides, we find the band gap of the chalcogenides evolve differently with the thickness of the perovskite blocks in these natural superlattices. We show that octahedral rotations (i.e. deviation of the B-X-B bond angles from 180) and quantum confinement effects compete to decide the band gap evolution of RP phases. The insights gained here will allow us to rationally design layered perovskite phases for electronics and optoelectronics.