Student Evaluations of Teaching (Mostly) Do Not Measure Teaching Effectiveness

SET are biased against female instructors by an amount that is large and statistically significant.
The bias affects how students rate even putatively objective aspects of teaching, such as how promptly assignments are graded.
The bias varies by discipline and by student gender, among other things.
It is not possible to adjust for the bias, because it depends on so many factors.
SET are more sensitive to students’ gender bias and grade expectations than they are to teaching effectiveness.
Gender biases can be large enough to cause more effective instructors to get lower SET than less effective instructors.

These findings are based on nonparametric statistical tests applied to two datasets: 23,001 SET of 379 instructors by 4,423 students in six mandatory first-year courses in a five-year natural experiment at a French university, and 43 SET for four sections of an online course in a randomized, controlled, blind experiment at a US university.

Related collections

Most cited references 24

Record: found
Abstract: not found
Article: not found

On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9

T Speed, D. Dabrowska, Jerzy Splawa-Neyman (1990)

0 comments Cited 8 times      Review now

Bookmark

Record: found
Abstract: not found
Article: not found

Student Perceptions of and Expectations for Male and Female instructors: Evidence relating to the Question of Gender Bias in Teaching Evaluation

SK Bennett, Sheila K. Bennett (1982)

0 comments Cited 4 times      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Hot or not: do professors perceived as physically attractive receive higher student evaluations?

Todd C Riniolo, N Misso, Brad T. Sherman … (2005)

Previous research investigating the influence of perceived physical attractiveness on student evaluations of college professors has been limited to a handful of studies. In this study, the authors used naturally occurring data obtained from the publicly available Web site www.ratemyprofessors.com. The data suggested that professors perceived as attractive received higher student evaluations when compared with those of a nonattractive control group (matched for department and gender). Results were consistent across 4 separate universities. Professors perceived as attractive received student evaluations about 0.8 of a point higher on a 5-point scale. Exploratory analyses indicated benefits of perceived attractiveness for both male and female professors. Although this study has all the limitations of naturalistic research, it adds a study with ecological validity to the limited literature.

0 comments Cited 4 times      Review now

Bookmark

All references

Author and article information

Affiliations

[1]OFCE, SciencesPo, Paris, France

[2]PSL, Université Paris-Dauphine, LEDa, UMR DIAL, Paris, France

[3]Department of Statistics, University of California, Berkeley, CA, USA

Author notes

[*]Corresponding author's e-mail address: pbstark@123456berkeley.edu

Contributors

Anne Boring: (View ORCID Profile)

Kellie Ottoboni: (View ORCID Profile)

Philip B. Stark: (View ORCID Profile)

Journal

Journal ID (publisher-id): SOR-EDU

Title: ScienceOpen Research

Publisher: ScienceOpen

ISSN ( Electronic ): 2199-1006

Publication date: 07 January 2016

Volume: 0 (ID: 818d8ec0-5908-47d8-86b4-5dc38f04b23e)

Issue: 0

Pages: 1-11

License:

This work has been published open access under Creative Commons Attribution License CC BY 4.0, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Conditions, terms of use and publishing policy can be found at www.scienceopen.com.

Counts

Figures: 0, Tables: 11, References: 23, Pages: 11

Product

Self URI (pdf): 3507_XE6680747344554310733.pdf

Comments

Emil Ole William Kirkegaard wrote:

I have some comments:

Use confidence intervals, not p values (cf. 'New Statistics'). These can be obtained by bootstrapping. If you really want p values, then report both.
Put the effect sizes with confidence intervals in the abstract.
Alter the empahsis to be more about the lack of correlation between SET and teacher ability. The focus on gender bias is not warranted with the small effect. It is clearly not "large" as claimed. mean r = 0.09 ≈ d 0.20, which by Cohen's standards is small. There is too much social justice warrior about this article.
It is possible to adjust for the gender bias by simply adding a little to the females SET multiplied by the proportion of male students.
Note which gender was the biased one in the abstract (male against females, not female against males, cf. Table 5).
A simpler recommendation is that SETs should not be used at all since they apparently do not correlate with actual performance. Why have them? This side-steps the entire gender bias issue.
Add to the abstract that the gender effect could not be explained by male instructors being better (i.e. findings from Table 4).
The technical aspect of the analyses seemed fine to me.

Hope these comments are useful. Overall, I liked the article.

2016-03-22 12:29 UTC

One person recommends this

Comment on this article

Register to benefit from advanced discovery features on more than 33,000,000 articles

| Sign in using LinkedIn | Sign in using Facebook | Sign in using ORCID

ScienceOpen disciplines:	Labor law, Assessment, Evaluation & Research methods
Keywords:	disparate impact, nonparametric statistics, gender bias, permutation tests

ScienceOpen: research and publishing network