Social interaction increases significantly the performance of a wide range of cooperative systems. However, evidence that natural swarms limit the number of social connections suggests potentially detrimental consequences of having an excess of social behavior. We study the repercussions that such excess has on the responsiveness of a swarm facing local environmental perturbations. Using a canonical model of collective behavior, we find a decrease in susceptibility when the amount of social interaction exceeds a certain threshold. In addition, we identify the relation between susceptibility and capacity of a natural swarm to perform actions critical to its survival such as predator avoidance. This connection between social activity, responsiveness and predator avoidance capacity presents a plausible picture of how natural selection could favor organisms that self-limit their sociality, as flocking starlings seem to do. Beyond natural systems, the benefits of limited social activity are also uncovered for other canonical models of collective behavior. These results have far-reaching implications for the design of artificial swarms or interaction networks.