Review of 'Student Evaluations of Teaching (Mostly) Do Not Measure Teaching Effectiveness'

We thank Professor Cetinkaya-Rundel for her thoughtful review.

Most of her concerns are already addressed in the paper; others we agree with in part but not entirely. For instance, Prof. Cetinkaya-Rundel asserts that

“while some of the analyses in the previous study required normally distribution (sic) of data and equal variances, others didn’t.”

She points to Welch’s two-sample t-test and MANOVA as examples. Both of these tests rely on the assumption that data are sampled independently at random from populations with normal distributions. That stochastic assumption contradicts the actual randomization in the MacNell et al. experiment and the as-if randomization in the SciencesPo natural experiment: the data arise from randomizing a fixed group of individuals into different treatments, not from sampling from larger populations at random. And the variables involved do not have normal distributions; indeed, they are dichotomous (e.g., gender), ordinal categorical (e.g., SET), or have compact support (e.g., grades). The null hypotheses for Welch’s t-test and MANOVA are simply not relevant to the problem. Asymptotically, those tests might have the correct level under the randomization actually performed, but for finite populations, the actual levels may differ considerably from the nominal levels, as Table 8 of the paper shows.

Prof. Cetinkaya-Rundel writes:

“Another issue is that the authors do not compare their findings to those from MacNell et al. [2014]. It should be explicitly stated in the paper whether the findings agreed or disagreed.”

Table 8 compares our results to those from MacNell et al. [2014] (the rightmost columns show the p-values from our permutation test side-by-side with the p-values from t-tests in MacNell et al.) and we discuss the differences near the end of page 8, in the section entitled “SET and perceived instructor gender.” Our omnibus test of the hypothesis that perceived instructor gender plays no role in SET matches the spirit of the MANOVA test (but for a more relevant null hypothesis); they do not report details of those tests, only that the p-values were less than 0.05. The permutation omnibus test we performed gives a p-value of essentially zero for the hypothesis that students rate instructors the same, regardless of the gender of the instructor’s name.

Prof. Cetinkaya-Rundel raises the concern that instructors and students appear in the data more than once, across years and across courses, producing correlation. In the Neyman model, the responses are fixed, not random: only the assignment of instructors to class sections is random. There is no assumption that responses random at all, much less independent. Moreover, instructors appear at most once per course per year—i.e., once per stratum. Because the natural experiment amounts to randomizing within each stratum, independently across strata, there is no dependence across strata, regardless of the identity of the instructors. Prof. Cetinkaya-Rundel continues:

“ A regression based approach, instead of a series of tests, would allow for this exploration, as well as conclusions that evaluate the relationship between SET and certain factors while controlling for others.”

A regression-based approach requires far stronger modeling assumptions (e.g., linearity of effects and IID additive errors), assumptions for which there is no basis here. Indeed, those assumptions seem rather far-fetched, for instance, because SET is an ordered categorical scale, not a linear scale. A model that posits that SET (or other variables) are observed with independent, identically distributed additive errors is also unjustified for these data: the randomness is in the assignment of students to instructors, not measurement noise. The Neyman model allows each individual (student or class section) to have its own response to each treatment condition, with no assumption about the relationship among responses of different individuals. Randomization in the experimental design mitigates confounding with no need to model the effect of covariates: that is the power of a randomized experiment. Moreover, the variables one would presumably want to control for are pretreatment variables, such as prior knowledge of the subject matter, while, the covariates available (e.g., final exam scores) are measured post-treatment.

Prof. Cetinkaya-Rundel also advocates reporting effect sizes. We reported empirical mean effect sizes in the tables (e.g., column 2 of Tables 3, 4, 5, 7, and 8). We intentionally did not report confidence intervals for mean effect sizes. Confidence intervals could be constructed by inverting the permutation tests, but that would require strong assumptions about the structure of the effect that have no basis here. For instance, it would suffice to assume that the effect of instructor gender on SET is the same for every student—but the data give evidence that the effect varies by student gender, at least. Indeed, one of our main conclusions is that there are such differences! Finally, from a policy perspective, effect size per se isn’t particularly interesting: the issue is disparate impact. The data show that the effect of gender on SET can overwhelm differences in actual instructor effectiveness, as measured by exam performance.