Teaching Psychology Research Methodology Across the Curriculum to Promote Undergraduate Publication: An Eight-Course Structure and Two Helpful Practices

Abstract

Goals for teaching research methods to undergraduates Teaching research methods is challenging because we not only wish to convey formal knowledge and encourage critical thinking, as with any course, but also to enable our students to dream up meaningful research projects, translate them into logical steps, conduct the research in a professional manner, analyze the data, and write a report in APA style. We also wish to spark interest in research, but in teaching undergraduates we have learned how elusive these goals can be (McKelvie, 1994). Even faculty have not mastered research design and writing. From serving as journal reviewers, we have found that many submissions show flaws such as elementary errors of logic (e.g., using a null control condition instead of a placebo treatment), tangled statistics, missing graphs, and ungrammatical, unclear writing that violates APA rules. Yet these manuscripts are sometimes written by people with doctorates and years of experience. Moreover, published papers may contain egregious faults (Standing and McKelvie, 1987). And although we have both published widely, we still hone our skills. It requires optimism to expect that a typical undergraduate will do better, after just a year or two of studies in psychology. In this paper, we describe a systematic set of methodology courses and two specific practices that we think can help. Methodology courses as the backbone of our psychology program How can methodology courses promote undergraduate involvement in publishable research? In our undergraduate liberal arts institution in Québec, where the Bachelor's degree normally takes 3 years following 2 years of college, we require more, rather than less: our psychology program has evolved since the 1960s to require a solid backbone of mandatory methods-related courses that is considerably more extensive than in most universities (McKelvie, 2000). Psychology majors take two consecutive introductory statistics courses in the first academic year, reaching the level of two-way ANOVA. Simultaneous with the second course, they take an introductory research methods course with lectures and discussion of important concepts, including theory, and involvement (participation, testing, writing) in instructor-planned projects that usually extend past research. In the second year, intending honors students take an advanced methods course that builds on the first one. It uses the same text, and continues active participation in project work. An unusual requirement (McKelvie, 2000) is a course in Psychometrics and Psychological Testing, reflecting our belief in the importance of measurement. In the third year, students with a program average of 80% or better, and a combined average of 75% in the advanced research methods course plus the second statistics course, may enter the honors program. They take a multivariate statistics course, and produce an idea for a data-based honors thesis under the direction of a main and a secondary supervisor. Students are encouraged to create their own research question on a topic of their choosing. Over two semesters, they discuss this project in a seminar course and write a formal proposal, and then in the thesis course, they conduct the research and write the report. These eight required methods-related courses produce well-grounded and motivated honors graduates, and gives them an opportunity to publish. Our students accept with good grace the challenge of this program, where only about one-fifth of them will obtain an honors degree rather than a major, and our departmental numbers have risen considerably over the years. Traditional solutions for teaching students to grapple with research methods A traditional solution in teaching research methods and exposing students to the research process has been first to lead them through a series of short, pre-packaged lab projects that demonstrate some well-established phenomena, and require brief write-ups, likely in APA format. This approach still has merit at the introductory stage, and the Online Psychology Laboratory experiments provided by the APA (https://opl.apa.org/) are valuable exercises for the neophyte. However, it appears that today more emphasis is commonly placed on original individual methods projects, perhaps to prepare students for an honors thesis, where they commonly choose the topic. The basic problem is that too many original student course projects lack a valid idea to test, coherent methods, and a valid formal design. Additionally, sample sizes are usually too low, yielding inadequate statistical power. More fundamentally, students habitually gravitate to correlational relationships rather than to randomized controlled experiments. They must learn to better justify proposals for non-experimental research. Two specific research methods teaching strategies Class discussion of published articles (introductory methods course) Critical class discussion of published articles can spark student interest in research papers and help them design better studies (McKelvie, 1994, 2013). Papers are carefully chosen to capture student attention and expose them to methodological issues. Study questions focus on important points in each reading. Students are also encouraged to generate their own critical comments and queries (McKelvie, 2013). This approach sits well with the typical case-oriented contents of leading methods texts (e.g., Morling, 2018). Five study questions are common to all papers (McKelvie, 2013): What type(s) of research method is (are) involved? In particular, is it a true experiment? What inferential statistics were used? Were they appropriate? What is (are) the sources of the problem (theory, past research, practical intervention, everyday life)? One example is Motley and Camden's (1985) study of sexual double entendres in lexical selection. It employed both experimental (manipulation of experimenter attractiveness) and non-experimental methods (sexual anxiety as a subject variable). Independent samples t-tests, chi square, and ANOVA were used. The study was based on theory and on everyday life. Another example is Milgram's (1963) seminal observational study of obedience. It is non-experimental, only contains descriptive statistics, and is based on everyday life. Students find it challenging to identify the research method. Realizing that “laboratory” does not mean “experiment” is a valuable lesson. A complete list of study questions and discussion papers is available (see link to McKelvie, 2013). Replication projects (advanced methods course) Although the traditional laboratory approach has merit, the projects may only be demonstrations, or suffer from inadequate sample size. Original individual student projects share this problem, and have other difficulties. One solution is for all the class to work on the same research project, created by the instructor in an area of their expertise. This study may be original as described in detail by LoSchiavo (2019). Advantages are that sample sizes will be healthy, increasing the likelihood of publication, and that students are motivated to create new knowledge. Alternatively, the instructor can plan a replication study, selecting a paper from the literature that is widely quoted and of manageable scope, and leading the class through either a conceptual or (better) an exact replication of the target study. Sample sizes will again be healthy, but other advantages are that planning is simplified, a rationale for the study exists, the method is pre-established, and students are educated about the replication debate (Maxwell et al., 2015). Their involvement may also be promoted by the realization that we can grapple together with the same issues as published authors, and that they are engaging in “real” research that is potentially publishable. The experience also prepares honors students in the seminar and thesis courses to create an original project, or one that replicates and extends previous work (e.g., Benmergui et al., 2017). The parallel teams approach A class may be divided into teams which each work on a different target article. Preferably, the teams can each try to replicate the same paper, which has the advantage of maximizing N, and we note that replications to be adequately powered should use more than the number of participants listed in the target article (Simonsohn, 2015). Additionally, the team results can be compared for consistency, before pooling the data to increase power. When the teams are of perhaps half a dozen members each, they can function most effectively as small, cohesive groups under the direction of an elected or designated team leader (for practical details see Standing, 2016). An example of a project using four parallel replication teams is described by Standing et al. (2016). This study successfully replicated Experiment 8 in the study by Gailliot et al. (2007), which made the controversial claim that self-control can be raised by consuming glucose rather than a placebo drink. As possible authorship is motivating to many students, the four team leaders here were included as coauthors of the instructor, with the remaining members of the class acknowledged in a footnote. Alternatively, all members of a class team may be listed as authors, as with a previous attempt focusing on the claim that priming a participant with a trait such as “intelligence,” or a stereotype such as “professor,” raises their cognitive performance in the form of Trivial Pursuit scores (Dijksterhuis and van Knippenberg, 1998). This replication attempt did not succeed (Roberts et al., 2013). Subsequently both a 9-experiment study (Shanks et al., 2013), and a preregistered replication study, involving 40 labs, have likewise failed to replicate the target study's results (O'Donnell et al., 2018). The results of student replication projects are most effectively communicated by posting them as summaries on the PsychFileDrawer.org website, which provides a refereed “Archive of Brief Reports of Replication Attempts in Experimental Psychology.” Challenges Major challenges in teaching research methods include students' limited ability to (a) build a study that makes a clear prediction with rival hypotheses, and (b) to think clearly and logically through key issues such as randomization, control conditions, double-blind testing, counterbalancing, power, sample size, experimenter effects, and demand characteristics (McKelvie, 1994). Another challenge is to have the student exert tighter controls in non-experimental studies [e.g., match groups on subject variables (Lemieux et al., 2002), and include a dependent variable on which no difference is expected]. Fundamentally, we ask honors students to propose a new study that is valid and interesting, with a clear connection to previous work. We confront these issues explicitly, in class and in personal interactions with students. Another major issue is that of obtaining prior approval from institutional research ethics boards, which requires careful time scheduling and attention to detail in the required documentation. To plan ahead of time, pilot testing is vitally needed, and this testing itself may require ethics approval, leading possibly to an infinite logical regress unless common sense is applied. The problems of writing skills, correct citation, and APA format are also pervasive. We find it useful to break reports into sections (staggered over time), to allow resubmission (after editing by the instructor), and to encourage students to consciously imitate the style of the APA model manuscript (American Psychological Association., 2010, pp. 41–53), rather than to memorize formal rules. Students also receive detailed handouts explaining these rules (https://www.ubishops.ca/wp-content/uploads/McKelvie_GuidetowritingreportsAPA6thedition-converted.pdf; http://www.ubishops.ca/wp-content/uploads/plagia04.pdf). Outcomes In addition to the replications published in PsychFileDrawer, the present approach, developed over several decades, has yielded a variety of PsycINFO-listed refereed articles with undergraduates: 71 and 50 papers for the present authors, respectively. Most papers are based on honors theses (e.g., McKelvie and Demers, 1979; Knight and McKelvie, 1986; Martel et al., 1987; Shackell and Standing, 2007; Standing et al., 2008, 2014; Sigal and McKelvie, 2012; Clohecy et al., 2015; Morin-Lessard and McKelvie, 2017), where students usually earn the right to primary authorship because they proposed and conducted the research, even if we led the conversion from thesis report to manuscript. In other cases (e.g., McKelvie et al., 2013), where students assisted in our projects, we were primary authors. Disputes can be avoided by clarifying authorship ground rules at the outset. As an example (Benmergui et al., 2017), one investigation attempted to replicate a report (Beauchamp, 2002) that false recall in the Deese-Roediger-McDermott-Read-Solso (DRMRS) procedure would be smaller when the materials were pictures rather than words. Items for remembering are constructed around a theme that is not on the list (e.g., thread, pin, sewing around needle). False memory occurs when the theme word (needle) is recalled. This replication was successful, and the experiment extended previous research with new materials, new conditions, and a measure of confidence. Conclusion Involving undergraduates in the publication process is not easy, but we believe that the present seeming success is related in part to our cumulative course structure, to the explicit identification and discussion of challenges, and to the two systematic exercises outlined here. Author contributions Both authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication. Conflict of interest statement The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.