- Record: found
- Abstract: found
- Article: found
Assessing faculty professional development in STEM higher education: Sustainability of outcomes
research-article
Author(s):
Terry L. Derting
1
,
* ,
Diane Ebert-May
2 ,
Timothy P. Henkel
3 ,
Jessica Middlemis Maher
4 ,
Bryan Arnold
5 ,
Heather A. Passmore
1
Publication date (Electronic):
18 March 2016
Journal:
Science Advances
Read this article at
There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.
Abstract
New faculty sustained the implementation of learner-centered courses in biology following professional development.
Abstract
We tested the effectiveness of Faculty Institutes for Reforming Science Teaching IV (FIRST), a professional development program for postdoctoral scholars, by conducting a study of program alumni. Faculty professional development programs are critical components of efforts to improve teaching and learning in the STEM (Science, Technology, Engineering, and Mathematics) disciplines, but reliable evidence of the sustained impacts of these programs is lacking. We used a paired design in which we matched a FIRST alumnus employed in a tenure-track position with a non-FIRST faculty member at the same institution. The members of a pair taught courses that were of similar size and level. To determine whether teaching practices of FIRST participants were more learner-centered than those of non-FIRST faculty, we compared faculty perceptions of their teaching strategies, perceptions of environmental factors that influence teaching, and actual teaching practice. Non-FIRST and FIRST faculty reported similar perceptions of their teaching strategies and teaching environment. FIRST faculty reported using active learning and interactive engagement in lecture sessions more frequently compared with non-FIRST faculty. Ratings from external reviewers also documented that FIRST faculty taught class sessions that were learner-centered, contrasting with the teacher-centered class sessions of most non-FIRST faculty. Despite marked differences in teaching practice, FIRST and non-FIRST participants used assessments that targeted lower-level cognitive skills. Our study demonstrated the effectiveness of the FIRST program and the empirical utility of comparison groups, where groups are well matched and controlled for contextual variables (for example, departments), for evaluating the effectiveness of professional development for subsequent teaching practices.
Related collections
Most cited references5
- Record: found
- Abstract: found
- Article: not found
Barriers to Faculty Pedagogical Change: Lack of Training, Time, Incentives, and…Tensions with Professional Identity?
Sara E. Brownell, Kimberly D. Tanner (2012)
The time has come for all biology faculty, particularly those who teach undergraduates, to develop a coordinated and sustainable plan for implementing sound principles of teaching and learning to improve the quality of undergraduate biology education nationwide. (Vision and Change, 2011, xv) Recent calls for reform, such as Vision and Change: A Call to Action, have described a vision to transform undergraduate biology education and have noted the need for faculty to promote this change toward a more iterative and evidence-based approach to teaching (American Association for the Advancement of Science [AAAS], 2011). A key challenge is convincing many faculty—not just a handful of faculty scattered across the country but the majority of life sciences faculty in every institution—to change the way they teach. Few would disagree that this is an ambitious goal. Change is difficult in any setting, but changing academic teaching appears to be especially tricky. Calls for change imply that the pedagogical approaches our own professors and mentors modeled and taught us might not be the best way to engage large numbers of diverse populations of undergraduates in our discipline. This effort potentially also involves telling faculty that what they have been doing for the past 5, 10, or even 30 yr may not the most effective approach, especially for today's students. Widespread change in undergraduate biology teaching—or in any of the sciences for that matter—has been documented to be difficult (Henderson et al., 2011). The general perception is that while there are pockets of change driven by individual faculty, there is little evidence that the majority of our faculty members are reconsidering their approach to teaching, despite dozens of formal policy documents calling for reform, hundreds of biology education research publications on the subject, and the availability and award of substantial amounts of external grant funding to stimulate change toward evidence-based teaching (Tagg, 2012). In fact, it is somewhat perplexing that we as scientists are resistant to such change. We are well trained in how to approach problems analytically, collect data, make interpretations, form conclusions, and then revise our experimental hypotheses and protocols accordingly. If we are experts at making evidence-based decisions in our experimental laboratories, then what forces are at play that impede us from adopting equally iterative and evidence-based approaches to teaching in our classrooms? What can we—as members of a community of biologists dedicated to promoting scholarly biology teaching—do to identify and remove barriers that may be impeding widespread change in faculty approaches to teaching? A substantial body of literature has highlighted many factors that impede faculty change, the most common of which are a lack of training, time, and incentives. However, there may be other barriers—unacknowledged and unexamined barriers—that might prove to be equally important. In particular, the tensions between a scientist's professional identity and the call for faculty pedagogical change are rarely, if ever, raised as a key impediment to widespread biology education reform. In this article, we propose that scientists’ professional identities—how they view themselves and their work in the context of their discipline and how they define their professional status—may be an invisible and underappreciated barrier to undergraduate science teaching reform, one that is not often discussed, because very few of us reflect upon our professional identity and the factors that influence it. Our primary goal in this article is to raise the following question: Will addressing training, time, and incentives be sufficient to achieve widespread pedagogical change in undergraduate biology education, or will modifying our professional identity also be necessary? FOCUSING ON THE BIG THREE: LACK OF TRAINING, TIME, AND INCENTIVES Insufficient training, time, and incentives are among the most commonly cited barriers for faculty change, and the focus of most of the current efforts to understand and promote faculty pedagogical change (Henderson et al., 2010, 2011; AAAS, 2011; Faculty Institutes for Reforming Science Teaching [FIRST] IV, 2012; National Academies of Science/Howard Hughes Medical Institute [NAS/HHMI], 2012). In terms of training, many faculty have indicated they feel ill-equipped to change the way they teach and thus would like access to structured, formal training. Unsurprisingly, we as faculty may not be knowledgeable about what constitutes a student-centered classroom (Hativa, 1995; Miller et al., 2000; Winter et al., 2001; Hanson and Moser, 2003; Luft et al., 2004; Yarnall et al., 2007) or we may be unconvinced as to whether new teaching methods are really more effective than traditional instruction (Van Driel et al., 1997; Miller et al., 2000; Winter et al., 2001; Yarnall et al., 2007). Even if faculty are aware of reform efforts, science faculty will most likely not have had training in these types of teaching methods (Rushin et al., 1997; Handlesman et al., 2004; Ebert-May et al., 2011). Vision and Change specifically highlights the need for training of early-career scientists, including postdoctoral fellows and assistant professors (AAAS, 2011). Efforts such as the NSF-funded FIRST IV program and the NAS/HHMI Summer Institutes for Undergraduate Biology Education are examples of programs intended to provide postdoctoral scholars and faculty of all ranks, respectively, with the needed expertise in innovative teaching through hands-on training (FIRST IV, 2012; NAS/HHMI, 2012). Although it is too early to gauge the long-term success of these programs, one wonders whether some of these training efforts may be hindered by the lack of buy-in from the home institutions. After faculty go to nationally or regionally organized training workshops and become excited about implementing new teaching strategies, are they met with support or resistance from their colleagues upon return to their home institutions? Furthermore, trying to achieve pedagogical change through 1-d or even 1-wk training sessions seems incongruent with the notion that pedagogical change for any instructor is an iterative and ongoing process. Even the most well intentioned of us forget what we learned, need extra practice, and often revert to our old habits when we are, inevitably, pressed for time. So although it is necessary to provide scientists with training opportunities demonstrating new ways of teaching, training alone is likely insufficient by itself to achieve lasting pedagogical change. What about issues of time? With the often-competing demands of research and teaching, faculty often find it difficult to carve out sufficient time to reflect deeply upon their teaching. While faculty at different types of institutions have varying degrees of teaching responsibilities, faculty at most 4-yr institutions are also required to do research and obtain significant external grant funding. Although this expectation is most explicit at R1 research institutions, it also exists at many comprehensive institutions, and even at small liberal arts colleges. Regardless of current faculty teaching loads, there is no doubt that the process of changing an instructional technique is time- and labor-intensive (Krockover et al., 2002; Howland and Wedman, 2004; Stevenson et al., 2005; Schneider and Pickett, 2006; Malicky et al., 2007). Additionally, research has shown that interactive teaching, as compared with traditional lecturing, typically takes more preparation time (Miller et al., 2000; Hanson and Moser, 2003; Pundak and Rozner, 2008). Thus, not only will the actual process of change take more time, but we are asking faculty to shift to a method that might be, by its very nature, more time-consuming. Institutional recognition of this fact, and corresponding allowance in faculty schedules, will thus be critical to accomplishing widespread adoption of evidence-based teaching strategies. In addition, for such changes to be made, there needs to be an incentive for faculty to modify their pedagogical approach; even though time is necessary, time alone is likely not sufficient for widespread change to occur. Incentives likely drive most of our professional decisions, and teaching is no exception. If we as faculty are indeed provided the requisite training and time to enact changes in our teaching, then there must also be a concomitant reason why we should want to change. Research has demonstrated that even if faculty are interested in changing their pedagogical approach, few incentives are available to spur this action (Hativa, 1995; Walczyk and Ramsey, 2003; Gibbs and Coffey, 2004; Weiss et al., 2004; Wilson, 2010; Anderson et al., 2011). Many argue that if change takes time and training, then faculty need to be compensated for their efforts in the form of lower teaching loads, financial benefits, recognition for tenure, teaching awards, or even, at the most basic level, verbal acknowledgment from colleagues and supervisors. Research has shown that in many universities there are few to no rewards for teaching in novel ways or introducing evidence-based strategies (Kember and McKay, 1996; Frayer, 1999; Krockover et al., 2002; Romano et al., 2004). In fact, there are some reports that change in instruction can lead to poor teaching evaluations, due to student resistance to change, which can negatively affect progression to tenure (Anderson, 2002, 2007). Until universities reward teaching as much as research (Hannan, 2005; Porter et al., 2006) or find ways to better integrate teaching and research (Kloser et al., 2011), the pressure is on faculty, in particular pretenure faculty, to spend the majority of their time on research, sometimes at the expense of high-quality teaching or any attention to the constant calls for change in teaching practice. The needs for training, time, and incentives are the most commonly cited impediments to widespread change in undergraduate biology faculty teaching practice, and indeed these are real and present barriers. However, let us pause. Imagine a university that provides faculty with all the training, all the time, and all the incentives faculty needed—would that be enough for all biology faculty or even the majority of biology faculty to adopt or build on pedagogical reform? While these “big three” factors are likely necessary for change to occur, it is far from clear that they are sufficient for it to happen. Focusing our efforts exclusively on training, time, and incentives ignores at least one additional and potentially key barrier to faculty change that is largely absent from change discussions: the role of a scientist's professional identity. INTRODUCING THE CONCEPT OF A SCIENTIST'S PROFESSIONAL IDENTITY The process by which we become scientists is often so long and arduous that few of us may have actually taken the time to reflect what constitutes our professional identities as scientists. In the midst of mastering laboratory techniques and crafting research grants, we are also learning, often subconsciously and implicitly, what professional norms we need to obey, or at least tolerate, to be perceived as successful academic scientists. Identity is most often thought about in the social sciences in terms of personal identity or how a person thinks of himself or herself in the context of society. Based on the ideas of Mead (1934) and Erikson (1968), identity is not a stagnant property, but rather an entity that changes with time, often going through stages, and is continuously modified based on the surrounding environment. It has been described as “being recognized as a certain kind of person in a given context” (Gee, 2001, p. 99). For the purposes of this article, we consider scientists’ professional identities to be how they view themselves and their work in the context of their disciplines and how they accrue status among their professional colleagues as academic scientists. These aspects are heavily influenced by the training specific to academic scientists, including course work, laboratory experiences, and the everyday culture and rewards of the scientific profession. Peer acceptance, or more formally the process of peer review, is also closely tied to the development of a professional identity in the sciences. Both the publication of the research we accomplish and garnering the resources we need for experimental work, either at our institution or from national funding agencies, are generally dependent on positive peer review and a shared professional identity with these peers. Thus, the development of a professional identity is not unlike the development of a personal identity but is situated in the context of a discipline and thus framed by the “rules of membership” of that discipline. If you are an academic scientist, then it is likely you were either explicitly told the rules of academic science, or you were able to somehow infer them and make choices to fit in or at least make others think that you fit in. Frustratingly, these rules of professional membership are not always obvious or intuitive, sometimes inadvertently keeping out those who are not afforded opportunities to learn the rules, expectations, and currencies of status within a particular discipline. This has been previously documented as a pivotal problem in the sciences, in particular in attracting and retaining women and people of color in the field (Carlone and Johnson, 2007; Johnson, 2007). While a professional identity is by definition an internalized identity, it guides our external actions and decisions in our profession, including the decisions we make about how we teach. If a scientist has a professional identity that does not encompass teaching at all, or if a scientist has a professional identity he or she feels could be put at risk in his or her discipline and among his or her peers by embracing innovative approaches to teaching, then professional identity becomes a critical barrier in efforts to promote widespread change in undergraduate biology education. WHAT ARE THE TENSION POINTS BETWEEN MAINTAINING ONE'S SCIENTIFIC PROFESSIONAL IDENTITY AND PARTICIPATING IN PEDAGOGICAL CHANGE? Several lines of inquiry support why a scientist's professional identity might interfere with his or her willingness to participate in pedagogical change. We describe here three tension points that individual faculty may commonly encounter when deciding whether or not to participate in biology education change efforts: 1) training cultivates a primarily research identity and not a teaching identity, 2) scientists are afraid to “come out” as teachers, and 3) the professional culture of science considers teaching to be lower status than research and positions scientists to have to choose between research and teaching. Each of these tension points, along with research literature that explores its origins, is presented below. TRAINING CULTIVATES PRIMARILY A RESEARCH IDENTITY AND NOT A TEACHING IDENTITY The first tension point between professional identity and pedagogical change efforts is that scientists are trained in an atmosphere that defines their professional identities primarily as research identities to the exclusion of teaching identities. A scientist's professional identity is shaped by a number of factors, but this socialization into the discipline of science often begins in graduate school (Austin, 2002). For undergraduates who spend considerable time in research labs for summer research projects or honors theses, socialization may begin earlier. However, graduate school is when all future scientists formally enter a learning period about the scientific profession and the cultural norms of the profession, often leading aspiring young scientists to adopt the values, attitudes, and professional identities of the scientists who trained them. Graduate school is the shared playground, where scientists learn the culture and values of the field, as well as how to play the game of professional science. Over the past 30 yr, doctoral and postdoctoral training at research institutions has put a tremendous emphasis on research, immersing students in the culture of research for a scientific discipline, while often ignoring teaching (Fairweather et al., 1996; Boyer Commission on Educating Undergraduates in the Research University, 2002). While some time spent as a teaching assistant may be required, in general there is no requirement for evidence of developing competency in teaching. Consequently, it has been asserted that there is a profound disconnect between the training that students are receiving in doctoral programs and the careers that many of these students will ultimately enter (Tilghman, 1998; Golde and Dore, 2001; Austin, 2002; Dillenburg, 2005; Dillenburg and Connolly, 2005; Fuhrmann et al., 2011). Faculty positions at most colleges and universities are primarily teaching positions, and even faculty positions at research institutions require some teaching, but the majority of graduate students in the sciences are only taught how to do research. What support is given to those graduate students who are interested in developing teaching skills in graduate school? A growing number of institutions have graduate student and faculty teacher-training programs (Rushin et al., 1997; Austin et al., 2008; Ebert-May et al., 2011). However, despite recommendations for the implementation of pedagogy-focused training in graduate school, programs focused on innovative teaching strategies are often voluntary and serve only a small percentage of the overall population of graduate students. Currently, there are no federal mandates associated with training grants that would require pedagogical training for future scientists. As a result, most graduate students still learn how to teach through an “apprenticeship of observation” (Lortie, 1975; Borg, 2004). They model their own teaching approaches after their professors. Students without explicit training tend to teach “naively” (Cross, 1990), often relying on inaccurate assumptions about teaching and learning. Most college classes in the sciences are taught in the traditional lecture format, so the majority of beginning science instructors equate teaching with lecturing, both linguistically and conceptually (Mazur, 2009). Without explicit training during graduate school, postdoctoral training experiences, or even early faculty years, these inaccurate assumptions about teaching appear to persist and become solidified. Additionally, even if a scientific trainee or early-career faculty member is interested in adopting pedagogical approaches different than the norm, there may be peer pressure from scientific colleagues to conform to traditional methods of teaching (Van Driel et al., 1997; Gibbs and Coffey, 2004). Not only is teaching not a formal or recommended component of postdoctoral training, some faculty advisors even view teaching as completely ancillary to, and a distraction from, the training that postdoctoral scholars need, ostensibly to become professors. The National Institutes of Health's Institutional Research and Academic Career Development Awards (NIH IRACDA) postdoctoral program is a notable exception to this. IRACDA postdoctoral fellows conduct research in basic science at R1 institutions and concurrently have formal, mentored teaching experiences at minority-serving institutions (IRACDA, 2012); however, IRACDA currently serves only a limited number of postdocs. Additionally, the FIRST IV program also seeks to provide postdoctoral fellows with training and mentored teaching experiences as they transition to faculty roles, but again, this is an option for a limited number of postdocs (FIRST IV, 2012). Both of these programs could serve as models for the more widespread integration of teaching and research into the scientific training and professional identity development of postdoctoral fellows. If scientists do not consider teaching part of their professional identities, then how can we expect them to change their own teaching and, even more importantly, support and encourage others to change as well? SCIENTISTS ARE AFRAID TO “COME OUT” AS TEACHERS A second tension point between maintaining one's professional identity and participating in pedagogical change is that embracing a teaching identity as part of one's scientific professional identity can be perceived as a liability and something to be hidden. Mark Connolly and colleagues have documented that some graduate students who are interested in teaching are afraid to “come out” as teachers (Connolly, 2010). They fear that they will be marginalized and discriminated against by their scientific peers and mentors. Some faculty advise graduate students to hide their interest in teaching; these mentors worry that the rest of academia will not take such students seriously as researchers (Connolly, 2010). There have been reports that some research professors, upon learning their graduate students are interested in teaching, no longer spend the same amount of time mentoring them. Significantly, some doctoral students have faculty advisors who do not allow them to engage in any activities outside laboratory work (Wulff et al., 2004). Some advisors are of the mentality that graduate students should always be at the bench and that any time devoted to teaching negatively affects research, despite a recent study indicating that teaching while doing research might improve research skills (Feldon et al., 2011). Unfortunately, this approach leaves students with both a skill set and perspective on science that is very narrowly focused. Postdoctoral scholars often face similar problems but often without the larger support structure that many graduate students have. Because postdocs tend to be fairly isolated in individual labs, they are even more dependent on their research mentors for guidance about career paths. If graduate students and postdoctoral scholars fear the ramifications of admitting that teaching is part of their identity, an interest in teaching can be internalized as something illicit, to be kept hidden from peers and mentors. Even those who are interested in continuing in academia to become professors are encouraged to limit the amount of teaching they do. This implicit, if not explicit, research-centric norm of graduate school can result in a student's internal conflict between developing a professional identity as a research scientist and a desire to also develop part of a professional identity as a teacher. As students struggle to reconcile these aspirations, they can fall prey to believing that teaching is inherently inferior to research and that if they are to succeed in the academic world of science, they should focus exclusively on research. For a graduate student with a strong interest in teaching, this could even result in doubts about his or her ability as a scientist. In the process of embracing a teaching identity, budding scientists potentially risk their status as researchers, as well as their professional identities, status, and even membership within the scientific community. THE PROFESSIONAL CULTURE OF SCIENCE CONSIDERS TEACHING TO BE LOWER STATUS THAN RESEARCH AND POSITIONS SCIENTISTS TO HAVE TO CHOOSE BETWEEN RESEARCH AND TEACHING Finally, a third tension point between maintaining one's professional identity and participating in pedagogical change is that teaching is often regarded as lower status than research in the scientific disciplines (Beath et al., 2012). A large part of this disparity in status originates from the culture of individual laboratories, departments, institutions, and even the discipline as a whole (Cox, 1995; Quinlan and Akerlind, 2000; Marbach-Ad et al., 2007). However, it is also reinforced by the general salary and status structures with regard to teaching within our society, in which teaching is generally considered to be not as well compensated for or afforded as much respect as many other professions. Faculty members who want to be perceived as successful and “real” scientists may have purposely avoided integrating teaching into their professional identities, because they feel it could undermine their scientific status with their colleagues, their departments, and their institutions. These actions might even be subconscious, a natural result of years of being surrounded by other faculty who view research as superior to teaching and hearing the age-old adage “those who can, do; those who can't, teach.” This contributes to a professional identity that deemphasizes teaching specifically to maintain high professional status, both within the confines of the institution and within the larger context of the discipline. It is perhaps unsurprising then that the community of science itself does not generally assume that a research identity and a teaching identity can coexist within the same individual. Unfortunately, participation in teaching or research is often seen as a choice, as a set of alternatives rather than an integrated whole. A recent finding from the Longitudinal Study of STEM Scholars (Connolly, 2012) concluded that graduate students are interested in pursuing careers that involve teaching. However, when this finding was reported more widely, it was misinterpreted to mean that these students did not want to do research. Quite the contrary, these students were expressing an increased interest in teaching that was independent of their commitment to or interest in research (M. Connolly, personal communication). Similarly, a recent publication in PLoS One also reinforced this tension point through a survey asking graduate students to rate the attractiveness of certain career paths and gave the choices of “a faculty career with an emphasis on teaching” and “a faculty career with an emphasis on research” with no option for “a faculty career that involves equal amounts of teaching and research,” thereby, likely unknowingly, setting up the mutually exclusive choice between teaching and research (Sauermann and Roach, 2012). Many scientific trainees and current faculty may want careers that involve a balance of both, and the perception that they need to choose one or the other makes it even harder for them to adopt teaching identities without feeling they must sacrifice their research identities, which are likely their primary source of professional status. Unfortunately, in the professional culture of science, an emphasis on teaching in one's professional career can often be mischaracterized as a choice made because one either cannot do research or does not want to do research. BRINGING PROFESSIONAL IDENTITY TO THE FOREFRONT OF CHANGE DISCUSSIONS: SHIFTING FROM AN INSTITUTIONAL DEFICIT MODEL TO A DISCIPLINE DEFICIT MODEL Given the tension points described above, professional identity may not be just one additional barrier to faculty pedagogical change; it could be hypothesized to be a key underlying reason why change strategies addressing training, time, and incentives have to date had only limited success in engaging broad groups of faculty in widespread biology education reform. If biology faculty are potentially entrenched in a professional identity grounded in a research identity to the exclusion of a teaching identity, then it would behoove us, as a community, to consider the possibility that professional identity could undercut all our efforts centered on the “big three” change strategies. As a scientist grounded in a research identity, one may view pedagogical training with skepticism, considering it to be a waste of time and effort, in particular if the training tries to promote teaching methods that depart from the cultural teaching norm in science: lecturing. In addition, it follows that extra time might not be the answer to promoting faculty change, if tensions with professional identity are at play. If we have extra time in the day, we may more likely spend that time on research activities that raise our status with professional colleagues and are aligned with our professional identities. Finally, tensions between a professional scientific identity and teaching reform may, unfortunately, trivialize any teaching incentives that are developed. If scientists have professional identities that are predominantly research identities, then a Nature report or Science article will always be viewed as higher status than a departmental, university-wide, or even a national teaching award. Giving incentives for teaching will likely only have positive effects if we, as a scientific community, somehow begin to value those incentives to the same degree as research-based incentives. A common approach when we think about the reasons why faculty might not change the way they teach is to raise questions about the culture of individual institutions. We assume that the department or institution does not offer training opportunities, release time to develop new courses, or incentives for teaching in scientific ways. This could be broadly classified as an “institutional deficit model,” in which the institution lacks what is needed for reform. Certainly such problems can be inhibiting, and where they exist, institutional reform may be necessary to promote widespread involvement of faculty in pedagogical change. Many of the current pedagogical change strategies and frameworks operate within this model (Henderson et al., 2010, 2011). However, if we approach the issue of faculty change through the lens of professional identity, we will also want to consider a “discipline deficit model.” Faculty are not only members of their campuses, but also of their national professional societies and the professional community of scholars working in their particular fields. Perhaps it is not only a matter of institutions needing to provide training, time, and incentives, but also a need for a disciplinary culture shift, such that there are both a sufficient level of status attached to teaching and a critical mass of individuals who have professional identities that include teaching. Some might argue that regardless of what institutions offer, most faculty will not change the way they teach, because they view teaching as accessory to their professional identities, derived not from their institutions, but rather from their disciplines, which are cross-institutional. Finally, there is clearly a need for much more empirical research on all the potential barriers to faculty pedagogical change, but especially on the role of professional identity in determining whether a scientist chooses to participate in biology education reform efforts. Would efforts to broaden the professional identities of scientists to include teaching accelerate pedagogical change? To what extent do graduate or postdoctoral pedagogical training programs alter the professional identities of these early-career scientists? What are the long-term impacts of programs such as FIRST IV, NIH's IRACDA, or the HHMI/NAS Summer Institutes, in particular in terms of whether participants are more or less likely to engage in pedagogical reform compared with others? How would biologists—with a range of involvement in teaching and biology education reform efforts—themselves describe their professional identities and how these identities shape their professional choices and aspirations? LOOKING FORWARD: HOW COULD WE ALTER OUR PROFESSIONAL IDENTITIES TO BE MORE INCLUSIVE OF TEACHING? To achieve widespread pedagogical change toward more iterative and evidence-based approaches, it appears that we need to find ways to challenge the assumption that a scientist's professional identity should be primarily research-focused and consider ways in which teaching could become more integrated into the fabric of the discipline. Three possible areas for action are explored below. First, one place to start would be to broaden the goals and content of doctoral and postdoctoral training. Instead of having a handful of unstructured teaching requirements, students could be enrolled in training programs specifically designed to give them mentorship and support to teach in scientific ways. Specific faculty could be identified as teaching mentors for graduate students, who in turn could be given increased teaching opportunities and responsibilities as they progressed through the program. An important caveat is that these teaching mentors would themselves need to be properly trained in scientific teaching. In addition to excellence in research, excellence in teaching would also be an expected outcome of graduate education. One could envision a requirement in which dissertations included a chapter that provided evidence of scholarship and achievement in teaching. Those agencies and foundations that fund graduate education in the life sciences could take the lead in requiring such pedagogical training and deep experiences with teaching for the graduate students they support. By better integrating teaching within the current structure of scientific training, one could provide the next generation of scientists with a better foundation and skill set and also foster a teaching identity as part of their professional identities. A second way to better align professional identity with the goals of widespread pedagogical change may be to target the place where many faculty derive and maintain their professional identities: scientific journals. Publication and peer review in these journals is an important aspect of professional identity. Some scientific journals are beginning to include education sections, but these are often commentary, rather than research articles. An exception to this is Science magazine, in which a number of education articles have appeared as research reports over the past few years. By including articles about scholarly teaching and education research, scientific journals can influence scientists to view scientific teaching as a part of their professional activities. Notably, a number of scholarly journals that maintain high standards of peer review and national/international distribution have been developed in recent years that provide biologists with a venue for publication of their pedagogical research. CBE—Life Science Education, supported by the American Society for Cell Biology and the HHMI, is a good example of growth in this area. There has been a recent push to integrate peer-reviewed education articles from journals such as CBE-LSE into the tables of contents of scientific journals of professional societies, to provide more faculty easier access to education articles most relevant to their fields. This may enable scientists to view education articles and often by association, teaching, as important characteristics of their professional identities. Third, a key venue in which scientists construct and maintain their professional identities is at scientific professional meetings. These meetings were generally founded with a research focus, but many professional societies now have education sections within their annual meetings. Unfortunately, these are often not well integrated into the rest of the scientific meeting—sometimes entailing additional costs and being located in different venues and held on different days—reinforcing the concept that the education meeting is distinct from the research meeting. In addition, how are education research findings presented at these conferences? Ironically, the oral presentations are almost always presented as lectures, even when the topic of the talk is about how lecturing is not very effective! This illustrates how prevalent and influential the assumptions are about the expected norms of behavior and interaction at a scientific conference. Even biologists who have strong teaching identities and are well aware of more effective ways to present findings choose, for whatever reason (professional culture? professional identity?), not to employ evidence-based teaching and communication methods in the venue of a scientific conference. And while workshops and poster sessions would allow a higher level of interaction and dialogue—both generally more effective means of conveying information than oral presentations—these venues are often perceived as less important, lower status, and less stringent for high-quality data in the culture of scientific conferences. IN CONCLUSION… The challenge of addressing tensions between professional identity and pedagogical reform is a complicated issue. Importantly, we need to keep in mind that we as scientists ourselves are the ones responsible for the current state of our professional identities. We as academic scientists set up the tenure structure, publication requirements, and training requirements and dictate the group norms and expected modes of interaction in our own disciplines. We have created and contributed to a culture of science in which research generally has higher status than teaching. Some faculty continue to perpetuate the myth that a researcher should not want to teach and broadcast that value judgment to new graduate students, who are trying to forge their way as scientists. But we, as a professional community, also have the opportunity to take steps to broaden our professional identities and in doing so, address a potentially critical barrier in achieving widespread biology education reform.
- Record: found
- Abstract: found
- Article: not found
The Other Half of the Story: Effect Size Analysis in Quantitative Research
INTRODUCTION Quantitative research in biology education is primarily focused on describing relationships between variables. Authors often rely heavily on analyses that determine whether the observed effect is real or attributable to chance, that is, the statistical significance, without fully considering the strength of the relationship between those variables (Osbourne, 2008). While most researchers would agree that determining the practical significance of their results is important, statistical significance testing alone may not provide all information about the magnitude of the effect or whether the relationship between variables is meaningful (Vaske, 2002; Nakagawa and Cuthill, 2007; Ferguson, 2009). In education research, statistical significance testing has received valid criticisms, primarily because the numerical outcome of the test is often promoted while the equally important issue of practical significance is ignored (Fan, 2001; Kotrlik and Williams, 2003). As a consequence, complete reliance on statistical significance testing limits understanding and applicability of research findings in education practice. Therefore, authors and referees are increasingly calling for the use of statistical tools that supplement traditionally performed tests for statistical significance (e.g., Thompson, 1996; Wilkinson and American Psychological Association [APA] Task Force on Statistical Inference, 1999). One such tool is the confidence interval, which provides an estimate of the magnitude of the effect and quantifies the uncertainly around this estimate. A similarly useful statistical tool is the effect size, which measures the strength of a treatment response or relationship between variables. By quantifying the magnitude of the difference between groups or the relationship among variables, effect size provides a scale-free measure that reflects the practical meaningfulness of the difference or the relationship among variables (Coe, 2002; Hojat and Xu, 2004). In this essay, we explain the utility of including effect size in quantitative analyses in educational research and provide details about effect size metrics that pair well with the most common statistical significance tests. It is important to note that effect size and statistical significance testing (which we will shorten to “significance testing,” also known as hypothesis testing) are complementary analyses, and both should be considered when evaluating quantitative research findings (Fan, 2001). To illustrate this point, we begin with two hypothetical examples: one in biology and one in education. Effect Size and Statistical Significance Testing: Why Both Are Necessary Imagine that a researcher set up two treatment conditions: for example, unfertilized and fertilized plants in a greenhouse or, similarly, reformed and traditional teaching approaches in different sections of an introductory biology course. The researcher is interested in knowing whether the first treatment is more or less effective than the second, using some measurable outcome (e.g., dried plant biomass or student performance on an exam); this constitutes the research hypothesis. The null hypothesis states that there is no difference between the treatments. Owing to sampling variation in a finite sample size, even if the two treatments are equally effective (i.e., the null hypothesis is true), one sample mean will nearly always be greater than the other. Therefore, the researcher must employ a statistical significance test to determine the probability of a difference between the sample means occurring by chance when the null hypothesis is true. Using the appropriate test, the researcher may determine that sampling variability is not a likely explanation for the observed difference and may reject the null hypothesis in favor of the alternative research hypothesis. The ability to make this determination is afforded by the statistical power, which is the probability of detecting a treatment effect when one exists, of the significance test. Statistical power is primarily determined by the size of the effect and the size of the sample: as either or both increase, the significance test is said to have greater statistical power to reject the null hypothesis. The basis for rejection of the null hypothesis is provided by the p value, which is the output of statistical significance testing that is upheld as nearly sacred by many quantitative researchers. The p value represents the probability of the observed data (or more extreme data) given that the null hypothesis is true: Pr(observed data|H0), assuming that the sampling was random and done without error (Kirk, 1996; Johnson, 1999). A low value of p, typically below 0.05, usually leads researchers to reject the null hypothesis. However, as critics of significance testing have pointed out, the abuse of this rather arbitrary cutoff point tends to reduce the decision to a reject/do not reject dichotomy (Kirk, 1996). In addition, many researchers believe that the smaller the value of p, the larger the treatment effect (Nickerson, 2000), equating the outcome of significance testing to the importance of the findings (Thompson, 1993). This misunderstanding is likely due to the fact that when sample size is held constant, the value of p correlates with effect size for some statistical significance tests. However, that relationship completely breaks down when sample size changes. As described earlier, the ability of any significance test to detect a fixed effect depends entirely on the statistical power afforded by the size of the sample. Thus, for a set difference between two populations, simply increasing sample size may allow for easier rejection of the null hypothesis. Therefore, given enough observations to afford sufficient statistical power, any small difference between groups can be shown to be “significant” using a statistical significance test. The sensitivity of significance testing to sample size is an important reason why many researchers advocate reporting effect sizes and confidence intervals alongside test statistics and p values (Kirk, 1996; Thompson, 1996; Fan, 2001). Kotrlik and Williams (2003) highlight a particularly clear example in which statistical and practical significance differ. In their study, Williams (2003) was interested in comparing the percent time that faculty members spend teaching with the percent time that they would prefer to spend teaching. Despite the fact that the mean differences between actual and preferred teaching time were statistically significant (t 154 = 2.20, p = 0.03), the effect size (Cohen's d = 0.09) was extremely small (see Tables 1 and 2 for effect size metrics and interpretations). As a result, the author did not suggest that there were practically important differences between actual and preferred teaching time commitments (Williams, 2003). Reporting the confidence interval would have also illustrated the small effect in this study: while the confidence interval would not have contained zero, one of its end points would have been very close to zero, suggesting that the population mean difference could be quite small. Table 1. Common measures of effect size Table 2. Interpreting effect size valuesa Effect size measure Small effect size Medium effect size Large effect size Very large effect size Odds ratio 1.5 2.5 4 10 Cohen's d (or one of its variants) 0.20 0.50 0.80 1.30 r 0.10 0.30 0.50 0.70 Cohen's f 0.10 0.25 0.40 — Eta-squared 0.01 0.06 0.14 — a Cohen, 1992, 1988; Rosenthal, 1996. Although Williams (2003) presents a case in which a small “significant” p value could have led to an erroneous conclusion of practically meaningful difference, the converse also occurs. For example, Thomas and Juanes (1996) present an example from a study of juvenile rainbow trout willingness to forage under the risk of predation (Johnsson, 1993). An important part of the study tested the null hypothesis that large and small juveniles do not differ in their susceptibility to the predator, an adult trout. Using eight replicate survivorship trials, Johnsson (1993) found no significant difference in the distribution of risk between the two size classes (Wilcoxon signed-rank test: T + = 29, p = 0.15). However, the data suggest that there may in fact be a biologically significant effect: on average, 19 ± 4.9% (mean ± SE) of the large fish and 45 ± 7% of the small fish were killed by the predator (Johnsson, 1993). This difference likely represents a medium effect size (see Table 2; Thomas and Juanes, 1996). Not reporting effect size resulted in the researchers failing to reject the null hypothesis, possibly due to low statistical power (small sample size), and the potential to erroneously conclude that there were no differences in relative predation risk between size classes of juvenile trout. Thus, metrics of effect size and statistical significance provide complementary information: the effect size indicates the magnitude of the observed effect or relationship between variables, whereas the significance test indicates the likelihood that the effect or relationship is due to chance. Therefore, interpretations derived from statistical significance testing alone have the potential to be flawed, and inclusion of effect size reporting is essential to inform researchers about whether their findings are practically meaningful or important. Despite the fact that effect size metrics have been available since the 1960s (Huberty, 2002) and have been recognized as being a potentially useful aspect of analyses since the 1990s (e.g., Cohen, 1994; Thompson, 1996; Wilkinson and APA Task Force on Statistical Inference, 1999), the adoption of effect size as a complement to significance testing has been a slow process, even in high-impact research (Tressoldi et al., 2013). Nevertheless, many journals are beginning to develop editorial policies requiring some measure of effect size to be reported in quantitative studies (e.g., Royer, 2000). In response to this need for implementation, we next discuss the various methods used to calculate effect sizes and provide guidance regarding the interpretation of effect size indices. Measures of Effect Size: Two Categories We concentrate on parametric tests and group effect sizes into two main categories: those for 1) comparing two or more groups and 2) determining strength of associations between variables. The most frequently used statistical tests in these two categories are associated with specific effect size indices (see Table 1; Cohen, 1992), and we will discuss some of the more common methods used for each below. Refer to Figure 1 for a general guide to selecting the appropriate effect size measure for your data. Figure 1. A dichotomous key to selecting an appropriate measure of effect size. Because many quantitative researchers are already accustomed to employing statistical significance tests but may want to begin reporting effect sizes as well, we suggest effect size metrics that are appropriate for data analyzed using common significance tests. Although not intended to be a comprehensive guide to effect size indices, this key indicates many of the measures relevant for common quantitative analyses in educational research. Researchers are encouraged to gather more information about these metrics, including their assumptions and limitations. Comparing Two or More Groups. A common approach to both biological and educational research questions is to compare two or more groups, such as in our earlier examples comparing the effects of a treatment on plant growth or student performance. For these kinds of analyses, the appropriate measure of effect size will depend on the type of data collected and the type of statistical test used. We present here a sample of effect size metrics relevant to χ2, t, or F tests. When comparing the distribution of a dichotomous variable between two groups, for instance, when using a χ2 test of homogeneity, the odds ratio is a useful effect size measure that describes the likelihood of an outcome occurring in the treatment group compared with the likelihood of the outcome occurring in the control group (see Table 1; Cohen, 1994; Thompson, 1996). An odds ratio equal to 1 means that the odds of the outcome occurring is the same in the control and treatment groups. An odds ratio of 2 indicates that the outcome is two times more likely to occur in the treatment group when compared with the control group. Likewise, an odds ratio of 0.5 indicates that the outcome is two times less likely to occur in the treatment group when compared with the control group. Granger et al. (2012) provide an example of reporting odds ratios in educational research. In their study, the effectiveness of a new student-centered curriculum and aligned teacher professional development was compared with a control group. One of the instruments used to measure student outcomes produced dichotomous data, and the odds ratio provided a means for reporting the treatment's effect size on this student outcome. However, the odds ratio alone does not quantify treatment effect, as the magnitude of the effect depends not only on the odds ratio but also on the underlying value of one of the odds in the ratio. For example, if a new treatment for an advanced cancer increases the odds of survival by 50% compared with the existing treatment, then the odds ratio of survival is 1.5. However, if oddscontrol = 0.002 and oddstreatment = 0.003, the increase is most likely not practically meaningful. On the other hand, if an oddscontrol = 0.5 and the oddstreatment = 0.75, this could be interpreted as a substantial increase that one might find practically meaningful. When comparing means of continuous variables between two groups using a t test, Cohen's d is a useful effect size measure that describes the difference between the means normalized to the pooled standard deviation (SD) of the two groups (see Table 1; Cohen, 1988). This measure can be used only when the SDs of two populations represented by the two groups are the same, and the population distributions are close to normal. If the sample sizes between the two groups differ significantly, Hedges’ g is a variation of Cohen's d that can be used to weight the pooled SD based on sample sizes (see Table 1 for calculation; Hedges, 1981). If the SDs of the populations differ, then pooling the sample SDs is not appropriate, and other ways to normalize the mean difference should be used. Glass's Δ normalizes the difference between two means to the SD of the control sample (see Table 1). This method assumes that the control group's SD is most similar to the population SD, because no treatment is applied (Glass et al., 1981). There are many relevant examples in the educational research literature that employ variations on Cohen's d to report effect sizes. Abraham et al. (2012) used Cohen's d to show how an instructional treatment affected students’ post scores on a test of the acceptance of evolutionary theory. Similarly, Matthews et al. (2010) used Cohen's d to show the magnitude of change in student's beliefs about the role of mathematics in biology due to changes in course materials, delivery, and assessment between different years of the same course. Gottesman and Hoskins (2013) applied Cohen's d to compare pre/post means of data collected using an instrument measuring students’ critical thinking, experimental design ability, attitudes, and beliefs. When comparing means of three or more groups, for instance, when using an analysis of variance (ANOVA) test, Cohen's f is an appropriate effect size measure to report (Cohen, 1988). In this method, the sum of the deviations of the sample means from the combined sample mean is normalized to the combined sample SD (see Table 1). Note that this test does not distinguish which means differ, but rather just determines whether all means are the same. Other effect size measures commonly reported with ANOVA, multivariate analysis of covariance (MANCOVA), and analysis of covariance (ANCOVA) results are eta-squared and partial eta-squared. Eta-squared is calculated as the ratio of the between-groups sum of squares to the total sum of squares (see Table 1; Kerlinger, 1964). Alternatively, partial eta-squared is calculated as the ratio of the between-groups sum of squares to the sum of the between-groups sum of squares and the error sum of squares (Cohen, 1973). For example, Quitadamo and Kurtz (2007) reported partial eta-squared, along with ANCOVA/MANCOVA results, to show effect sizes of a writing treatment on student critical thinking. However, eta-squared is deemed by some as a better measure to report, because it describes the variance accounted for by the dependent measure (Levine and Hullett, 2002), which bears similarities to typical measures reported in correlational studies. Determining Strength of Association between Variables. Another common approach in both biological and educational research is to measure the strength of association between two or more variables, such as determining the factors that predict student performance on an exam. Many researchers using this type of analysis already report appropriate measures of effect size, perhaps without even realizing they are doing so. In most cases, the regression coefficient or analogous index provides information regarding the magnitude of the effect. The Pearson product-moment correlation coefficient (Pearson's r) measures the association between two continuous variables, such as in a linear regression (see Table 1). Squaring the r value when performing a simple linear regression results in the coefficient of determination (r 2), a measure that provides information about the amount of variance shared between the two variables. For multiple-regression analysis, the coefficient of multiple determination (R 2) is an appropriate effect size metric to report. If one of the study variables is dichotomous, for example, male versus female or pass versus fail, then the point-biserial correlation coefficient (r pb) is the appropriate metric of effect size. The point-biserial correlation coefficient is similar in nature to Pearson's r (see Table 1). An easy-to-use Web-based calculator to calculate r pb is located at www.vassarstats.net/pbcorr.html. Spearman's rank correlation coefficient (ρ) is a nonparametric association measure that can be used when both variables are measured on an ordinal or ranked scale or when variables on a continuous scale are not normally distributed. This measure can be used only after one applies a transformation to the data that ranks the values. Because this is a nonparametric measure, Spearman's ρ is not as sensitive to outliers as Pearson's r. Note that there are also variations of Spearman's ρ that handle different formats of data. Most statistical software packages can calculate all of these measures of variable association, as well as most of the measures comparing differences between groups. However, one must be careful to be sure that values provided by the software are indeed what they are claimed to be (Levine and Hullett, 2002). How to Interpret Effect Sizes Once you have calculated the effect size measure, how do you interpret the results? With Cohen's d and its variants, mean differences are normalized to SD units. This indicates that a d value of 0.5 can be interpreted as the group means differing by 0.5 SDs. Measures of association report the strength of the relationship between the independent and dependent variables. Additional manipulation of these association values, for example, r 2, can tell us the amount of shared variance between the variables. For the case of regression analysis, we can assume that an r 2 value of 0.3 means that 30% of the variance in the dependent variable can be explained by the independent variable. Additionally, McGraw and Wong (1992) developed a measure to report what they call “the common language effect size indicator,” which describes the probability that a random value sampled from one group will be greater than a random value sampled from a comparison group (McGraw and Wong, 1992). Statisticians have determined qualitative descriptors for specific values of each type of effect size measure (Cohen, 1988, 1992; Rosenthal, 1996). For more interpretation of these types of measures, see Table 2. These values can help guide a researcher to make some sort of statement about the qualitative nature of the effect size, which is useful for communicating the meaning of results. Additionally, effect size interpretations impact the use of data in meta-analyses. Please refer to Box 1 to see an example of how interpretations of the different types of effect size measures can be converted from one type to another for the purpose of meta-analysis. Box 1. Use of effect sizes in meta-analyses Effect size measures are an important tool used when performing meta-analyses because they provide a standardized method for comparing results across different studies with similar designs. Two of the more common measures are Pearson's r and Cohen's d. Cohen's d describes the difference between the means of two groups normalized to the pooled standard deviation of the two groups. Pearson's r measures the association between two continuous variables. A problem arises when comparing a study that reports an r value with one that reports a d value. To address this problem, statisticians have developed methods to convert r values into d values, and vice-versa. The equations are listed below: Many studies in the literature do not report effect sizes, and only report statistical significance results such as p values. Rosenthal and Rubin (2003) have developed a measure to account for this issue, r equivalent, which can determine effect size from experimental designs comparing the means of two groups on a normally distributed outcome variable (Rosenthal and Rubin, 2003). This measure allows meta-analysis researchers to derive apparent effect sizes from studies that only report p values and sample sizes. First, one determines a t value from a t-value table by using the associated sample size and one-tailed p value. Using this t value, one can calculate r equivalent using the following equation: , where df = degrees of freedom on which the p-value is based. Limitations of Effect Size We have built a justification for the reporting of effect sizes as a complement to standard statistical significance testing. However, we do not wish to mislead the reader to construe effect size as a panacea in quantitative analyses. Effect size indices should be used and interpreted just as judiciously as p values. Effect sizes are abstract statistics that experience biases from sampling effort and quality and do not differentiate among relationships of similar magnitude that may actually have more or less practical significance (Coe, 2002; Nakagawa and Cuthill, 2007; Ferguson, 2009). Rather, determination of what constitutes an effect of practical significance depends on the context of the research and the judgment of the researcher, and the values listed in Table 2 represent somewhat arbitrary cutoffs that are subject to interpretation. Just as researchers may have logical reasons to choose an alpha level other than p = 0.05 with which to interpret statistical significance, the interpretation of practical relationships based on effect size may be more or less conservative, depending on the context. For example, an r of 0.1 for a treatment improving survival of a fatal disease may be of large practical significance. Furthermore, as we mentioned earlier, one should always accompany the proper effect size measure with an appropriate confidence interval whenever possible (Cohen, 1994; Nakagawa and Cuthill, 2007; Ellis, 2010; Tressoldi et al., 2013). For example, Lauer et al. (2013) reported Cohen's d along with 95% confidence intervals to describe the effects of an administration of a values-affirmation exercise on achievement gaps between men and women in introductory science courses. CONCLUSION By highlighting the problems with relying on statistical significance testing alone to interpret quantitative research results, we hope to have convinced the reader that significance testing is, as Fan (2001) puts it, only one-half of the coin. Our intent is to emphasize that no single statistic is sufficient for describing the strength of relationships among variables or evaluating the practical significance of quantitative findings. Therefore, measures of effect size, including confidence interval reporting, should be used thoughtfully and in concert with significance testing to interpret findings. Already common in such fields as medical and psychological research due to the real-world ramifications of the findings, the inclusion of effect size reporting in results sections is similarly important in educational literature. The measures of effect size described here do not by any means represent the numerous possible indices, but rather are intended to provide an overview of some of the most common and applicable analyses for educational research and a starting point for their inclusion in the reporting of results. In addition to the references cited throughout this article, we recommend several informative and accessible authorities on the subject of effect sizes, summarized in Table 3. Table 3. Recommended references for learning more about and implementing effect size measures as a part of standard statistical analyses Introduction to effect sizes written for the nonstatistician and relevant to the educational researcher Coe R (2002). It's the effect size, stupid: what effect size is and why it is important. Paper presented at the Annual Conference of the British Educational Research Association, held 12–14 September 2002, at the University of Exeter, UK. www.leeds.ac.uk/educol/documents/00002182.htm. Theoretical explanation of effect size measures written for those with stronger statistical foundation Cohen J (1988). Statistical Power Analysis for the Behavioral Sciences, 2nd ed., Hillsdale, NJ: Lawrence Erlbaum. Accessible and relevant reference for the practical application of effect size in quantitative research; includes directions for calculating effect size in SPSS Ellis PD (2010). The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and the Interpretation of Research Results, Cambridge, UK: Cambridge University Press. A guide to implementing effect size analyses written for the researcher Nakagawa S, Cuthill IC (2007). Effect size, confidence interval and statistical significance: a practical guide for biologists. Biol Rev Camb Philos Soc 82, 591–605. American Psychological Association recommendation to report effect size analyses alongside statistical significance testing Wilkinson L, APA Task Force on Statistical Inference (1999). Statistical methods in psychology journals: guidelines and explanations. Am Psychol 54, 594–604.
- Record: found
- Abstract: found
- Article: found
Just the Facts? Introductory Undergraduate Biology Courses Focus on Low-Level Cognitive Skills
Jennifer Momsen, Tammy Long, Sara A. Wyse … (2010)
Introductory biology courses are widely criticized for overemphasizing details and rote memorization of facts. Data to support such claims, however, are surprisingly scarce. We sought to determine whether this claim was evidence-based. To do so we quantified the cognitive level of learning targeted by faculty in introductory-level biology courses. We used Bloom's Taxonomy of Educational Objectives to assign cognitive learning levels to course goals as articulated on syllabi and individual items on high-stakes assessments (i.e., exams and quizzes). Our investigation revealed the following: 1) assessment items overwhelmingly targeted lower cognitive levels, 2) the cognitive level of articulated course goals was not predictive of the cognitive level of assessment items, and 3) there was no influence of course size or institution type on the cognitive levels of assessments. These results support the claim that introductory biology courses emphasize facts more than higher-order thinking.
Author and article information
Journal
Journal ID (nlm-ta): Sci Adv
Journal ID (iso-abbrev): Sci Adv
Journal ID (publisher-id): SciAdv
Journal ID (hwp): advances
Title:
Science Advances
Publisher:
American Association for the Advancement of Science
ISSN
(Electronic):
2375-2548
Publication date Collection:
March
2016
Publication date
(Electronic):
18
March
2016
Volume: 2
Issue: 3
Electronic Location Identifier: e1501422
Affiliations
Author notes
Author information
Terry L. Derting
http://orcid.org/0000-0002-6974-5049
Diane Ebert-May
http://orcid.org/0000-0003-0189-597X
Timothy P. Henkel
http://orcid.org/0000-0002-2790-9535
Article
Publisher ID: 1501422DOI: 10.1126/sciadv.1501422
PMC ID: 4803486
PubMed ID: 27034985
SO-VID: af6e6517-17f0-40bb-9bb1-aea18f8537c2
Copyright © Copyright © 2016, The Authors
License:
This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial license, which permits use, distribution, and reproduction in any medium, so long as the resultant use is not for commercial advantage and provided the original work is properly cited.
History
Date
received
: 10
October
2015
Date
accepted
: 30
January
2016
Funding
Funded by: FundRef http://dx.doi.org/10.13039/100000001, National Science Foundation;
Award ID: ID0E6LAK5598
Award ID: 08172224
Award Recipient
:
Diane Ebert-May
Funded by: FundRef http://dx.doi.org/10.13039/100000001, National Science Foundation;
Award ID: ID0EFQAK5599
Award ID: 08172224
Award Recipient
:
Terry L. Derting
Categories
Subject:
Research Article
Subject:
Research Articles
Subject:
SciAdv r-articles
Subject:
Education Research
Custom metadata
Copyeditor Meann Ramirez
Data availability: