Many researchers consider costs in shifting attention and mental set to reflect a basic ability to use top-down goal information to guide action. Although switch costs have been used as measures of individuals' executive function, whether common abilities underlie task set switching across different types of shifting tasks has not been well studied. In 249 participants, we studied whether switch costs in a novel two-choice reaction time task were correlated across variations in two variables: thelocus of representation (stimuli were either perceptually available or stored in working memory [WM]) and which of two judgment tasks was performed. Switch costs were asymmetrical, in that it was easier to switch to the easier judgment, and were related to overall and relative processing speed: Switch costs were higher when the task was more difficult. These factors should be accounted for when one is measuring individual differences in switch costs. After controlling for these effects, we found evidence for a common ability underlying switch costs that involved both task set preparation and response selection; however, residual shift costs, which involve only response selection, were uncorrelated across tasks. Correlations among switch costs were substantially higher within task type (e.g., correlations of WM shifting tasks with other WM shifting tasks and of perceptual tasks with perceptual ones), suggesting that there are also processes unique to switching within WM and switching among visible stimuli.