Despite releases of governmental guidelines for promoting physical fitness among the youth in China, the performance of college students in fitness tests has been declining over the past three decades. Obesity and physical inactivity have been proposed as two main causes. However, their relative importance for improving physical fitness remains unclear. To address this knowledge gap, we collected longitudinal data spanning four consecutive years on the physical fitness test for students from Nanjing University, China. Physical education classes of two hours per week were mandatory for the first two years. Using mixed effects models, we quantify the within-subject effects of weight, muscular endurance, sex, and mandatory physical education courses, among other variables, on physical fitness total score. We found that, in spite of the dominance of normal weight among the students, losing weight was positively associated with the total score, with significant sex differences in the associations. Compulsory exercise provided by physical education classes per week had strong positive impacts on the total score, comparable to losing weight of roughly 15–17 kg for males and 5–10 kg for females. Half sex difference in the total score was explained by male students’ poor performance in the muscular endurance represented by pull-ups. Our results suggest that college students in China should engage in physical activity of higher levels to improve their physical fitness, with a heightened awareness of extra fat under normal weight and insufficient muscular endurance.