how do you teach novice scientists to be thoughtful, critical researchers?
by cynthia brame
i recently had two experiences that spoke to this issue. first, i ran into my colleague carl johnson in the biosci department at vanderbilt. he said that he was finishing up getting ready to present at journal club, and that he needed to go take another look at the paper “to challenge them to be hypercritical.” i was intrigued by the notion of forcing the other readers of the paper into the uncomfortable position of being hypercritical and so i stopped to ask carl what he meant. he said, “we basically piss on any weak aspect of the paper. we’re trying to teach them to think like reviewers,” or to look at every piece of data with a very critical eye. i haven’t been to carl’s journal club, but i suspect that graduate students and post-docs quickly come to expect that readers will view their work with the same level of critique—and thus learn to apply a fairly rigorous lens to the work before submitting it for presentation or publication.
later the same day, i attended presentations by graduate students on teaching-as-research projects that they are completing as part of cirtl network programs. teaching-as-research, a core tenet of cirtl, bears a lot of resemblance to the scholarship of teaching and learning (sotl) and discipline-based education research (dber); i’m not sure i’m smart enough to parse out the differences, but all investigate questions of teaching and learning. thus these grad student presentations, which could be works-in-progress, investigated some facet of student learning, usually in response to an intervention (e.g., do active learning techniques improve learning in a graduate science class? does cognitive processing facilitated by think-pair-share improve student retention of content?). the tone during these presentations was notably different than the one that carl espoused earlier in the day, with facilitators (including myself) celebrating the fact that graduate students and post-docs were doing this type of research rather than casting a truly critical eye on the results. in this case, the goal seemed to be to support and applaud the effort rather than to critically review the result.
as all experienced mentors know, both approaches are important to the development of budding scientists. excessive early criticism drives people away from a field—be that a disciplinary research field or a discipline-based education research field. in the early stages of development, it’s vital to provide support and gentle, helpful criticism. in the case of discipline-based education research, however, how do we move from relatively uncritical support to carl’s training to “piss on any weak aspect”? for scientists to trust the results of the work, it’s vital that we be able to apply the latter lens: we have to be our excessively, exuberantly critical selves. and yet, few of us are trained in this sort of research. how do we move from the gentle support that the cirtl network is providing to more rigorous, critical support of the work?
cbe-life sciences education is providing one mechanism to make this transition with their recent implementation of a “research methods” feature in the journal. this feature seeks to make discipline-based education researchers (or sotl scholars…or teaching-as-researchers) more familiar with good practice in the sort of social science research that many of us seek but are not sure how to do. the first feature focused on the importance of calculating the effect size as well as the significance when characterizing the effect of an intervention on learning, while the most recent one addressed using linear regression to control for student characteristics in undergraduate stem education research.
there are also excellent local resources to help us be more rigorous as we investigate student learning. the cft’s sotl guide, authored by assistant director nancy chick, provides a fantastic introduction to the frameworks and approaches common in assessing teaching and learning, walking the reader through the process of identifying evidence, planning the project, and analyzing data.
finally, some of our science colleagues are leading by example. from richard hake’s classic meta-analysis comparing the effects of interactive engagement to traditional instruction on learning in physics, to carl wieman’s careful analysis of the efficacy of deliberate practice, to erin dolan’s careful construction of a tool to measure the effects of science inquiry experiences (and many more that i am omitting), we have excellent examples of rigorous analysis of student learning by scientists employing their inherently critical natures.
and so i think the answer is this: we must support novice scientists in their research through praise, example (from our own work or others’), thoughtful questioning…and the expectation of careful criticism. we owe them this support no less in discipline-based educational research (or sotl, or tar) than in their disciplinary research.
leave a response