Television (TV) viewing time is associated with abnormal glucose metabolism, the metabolic syndrome, and risk of type 2 diabetes; associations are stronger and more consistent in women. One explanation of this difference may be that TV viewing is a marker of an overall pattern of sedentary behavior in women. We sought to examine associations of TV viewing time with other sedentary behaviors and with leisure-time physical activity in a large sample of Australian adults. Adults aged between 20 and 65 years (n = 2,046) completed a self-administered questionnaire on TV viewing, five other leisure-time sedentary behaviors, and leisure-time physical activity. Mean adjusted time spent in other sedentary behaviors and in physical activity was compared across TV-time categories previously shown to be associated with abnormal glucose metabolism. After adjustment for body mass index and socio-demographic variables, women's time spent watching TV was associated positively with time in other sedentary behaviors and negatively with leisure-time physical activity, but no such associations were observed in men. TV viewing time may be a robust marker of a sedentary lifestyle in women but not in men. Gender differences in the pattern of sedentary behaviors may explain at least in part the gender differences in the previously reported associations of TV viewing time with biological attributes related to type 2 diabetes.