Recent advances in the crystal structure determination of ligand-protected metal clusters have revealed that their electronic structures and optical features are essentially governed by the nuclearity and geometries of the inorganic frameworks. In this Perspective, we point out the definite effects of the exterior ligand moieties on the properties of small gold clusters. On the basis of systematic experimental studies on the optical properties of Au8 and Au13 clusters with various anionic ligands, it was shown that not only the "through-bond" electronic effects of coordinating atoms but also the nonbonding interaction with neighboring heteroatoms and the electronic coupling with π-systems cause substantial perturbations. We also suggest that the steric rigidity of the ligand environments affects their photoluminescence efficiencies. These findings imply the feasibility of the facile modulation of the cluster properties through the appropriate choice of ligand modules, which may lead to the evolution of novel cluster-based materials with unique properties and functions.