Chimeric antigen receptors (CARs) targeting CD19 have mediated dramatic anti-tumor responses in hematologic malignancies, but tumor regression has rarely occurred using CARs targeting other antigens. It remains unknown whether the impressive effects of CD19 CARs relate to greater susceptibility of hematologic malignancies to CAR therapies, or superior functionality of the CD19 CAR itself. We discovered that tonic CAR CD3ζ phosphorylation, triggered by antigen-independent clustering of CAR scFvs, can induce early exhaustion of CAR T cells that limits anti-tumor efficacy. Such activation is present to varying degrees in all CARs studied, with the exception of the highly effective CD19 CAR. We further identify that CD28 costimulation augments, while 4-1BB costimulation ameliorates, exhaustion induced by persistent CAR signaling. Our results provide biological explanations for the dramatic anti-tumor effects of CD19 CARs and for the observations that CD19.BBz CAR T cells are more persistent than CD19.28z CAR T cells in clinical trials.