{"id":1595,"date":"2010-07-19t19:04:57","date_gmt":"2010-07-19t19:04:57","guid":{"rendered":"\/\/www.imrbdigital.com\/"},"modified":"2010-07-28t14:36:32","modified_gmt":"2010-07-28t14:36:32","slug":"student-ratings-myths-vs-research-evidence","status":"publish","type":"page","link":"\/\/www.imrbdigital.com\/articles-and-essays\/the-teaching-forum\/student-ratings-myths-vs-research-evidence\/","title":{"rendered":"student ratings: myths vs. research evidence"},"content":{"rendered":"<p><strong><\/strong><a name=\"top\"><\/a><em>this article was originally published in the  fall 2003 issue of the cft&#8217;s newsletter, <em>teaching forum. <\/em><\/em><\/p>\n<p>micheal theall, ph.d.<\/p>\n<p><em>the following article is reprinted with permission of  the author and of <strong>focus on faculty <\/strong> (fall 2002), a  publication of the brigham young university faculty center, ed. d. lynn  sorenson. <\/em><\/p>\n<p><em>michael theall has twenty-six years of experience as a  faculty member and as a professional in instructional design,  development and evaluation. he has founded faculty centers for teaching,  learning and evaluation at three universities: the university of  illinois, the university of alabama, and youngstown state university  (oh). theall and colleague jennifer franklin recently received a career  achievement award from the american education research association  (aera). they are authors of &#8220;the student ratings debate,&#8221; a monograph  for <strong>new directions for institutional research <\/strong> (2001),  among numerous other research publications. <\/em><\/p>\n<p><a name=\"stud\"><\/a>student ratings of instruction are  hotly debated on many college campuses. unfortunately, these debates are  often uninformed by the extensive research on this topic. marsh&#8217;s  often-cited review of the research on student ratings shows that student  ratings data are: a) multidimensional; b) reliable and stable; c)  primarily a function of the instructor who teaches the course; d)  relatively valid against a variety of indicators of effective teaching;  e) relatively unaffected by a variety of variables hypothesized as  potential biases; and f) seen to be useful by faculty, students, and  administrators. <a href=\"#note1\">[1]<\/a><\/p>\n<p>the researchers who have synthesized all the major studies of  ratings have reached the same conclusions as marsh. but even when the  data are technically rigorous, one of the major problems is day-to-day  practice: student ratings are often misinterpreted, misused, and not  accompanied by other information which allows users to make sound  decisions. as a result, there is a great deal of suspicion, anxiety, and  even hostility toward ratings. several questions are commonly raised  with respect to student ratings. current research provides answers to  many of these questions.<\/p>\n<ul>\n<li><a href=\"#stud\">are  students qualified to rate their instructors and the instruction they  receive? <\/a><\/li>\n<li><a href=\"#ratings\">are  ratings based solely on popularity? <\/a><\/li>\n<li><a href=\"#learning\">are  ratings related to learning? <\/a><\/li>\n<li><a href=\"#situational\">are  ratings affected by situational variables? <\/a><\/li>\n<li><a href=\"#grades\">do  students rate teachers on the basis of expected (or received) grades? <\/a><\/li>\n<li><a href=\"#judgment\">can  students make accurate judgments while still involved in their  schooling? <\/a><\/li>\n<li><a href=\"#guidelines\">guidelines  for good evaluation practice <\/a><\/li>\n<li><a href=\"#notes\">notes  &amp; bibliography <\/a><br \/>\n<table cellspacing=\"0\" cellpadding=\"4\">\n<tbody>\n<tr>\n<td>are students qualified to rate their instructors  and the instruction they receive?<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>generally speaking, the answer is &#8220;yes.&#8221; students can  report the frequencies of teacher behaviors, the amount of work  required, how much they feel they have learned, and the difficulty of  the material. they can answer questions about the quality of lectures,  the value of readings and assignments, the clarity of the instructor&#8217;s  explanations, the instructor&#8217;s availability and helpfulness, and many  other aspects of the teaching and learning process. no one else is as  qualified to report what transpired during the semester, simply because  no one else is there for the entire semester. students are certainly  qualified to express their satisfaction or dissatisfaction with the  experience. they have a right to express their opinions in any case, and  no one else can report the extent to which the experience was useful,  productive, informative, satisfying, or worthwhile. while opinions on  these matters are not direct measures of the performance of the teacher  or the content learned, they are legitimate indicators of student  satisfaction; there is a substantial research base linking this  satisfaction to effective teaching and learning.<\/p>\n<p>but students are not necessarily qualified to report on all  issues. for example, beginning students cannot accurately rate the  instructor&#8217;s knowledge of the subject. a colleague&#8217;s rating is more  appropriate for this purpose. likewise, peers are better qualified to  judge content currency, curricular match, course design, or assessment  methods. both students and peers are in unique positions to provide  enlightening perspectives. for effective evaluation, remember to use  multiple sources of data and ask questions that respondents can  legitimately answer.<\/p>\n<p><a href=\"#top\">back  to top <\/a><\/p>\n<table cellspacing=\"0\" cellpadding=\"4\">\n<tbody>\n<tr>\n<td>are ratings based solely on popularity?<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><a name=\"ratings\"><\/a>there is no basis for this argument and no research to  substantiate it. when this topic arises, the term &#8220;popular&#8221; is never  defined. rather, it is left to imply that learning should somehow be  unpleasant, and the &#8220;popularity&#8221; statement is usually accompanied by an  anecdote suggesting that &#8220;the best teacher i ever had was the one i  hated most.&#8221; the assumption that popularity somehow means a lack of  substance, knowledge, or challenge is entirely without merit. in fact,  several studies show students learn more in courses in which teachers  demonstrate interest\/concern for the students and their learning; of  course these teachers also receive higher ratings.<\/p>\n<p><a href=\"#top\">back  to top <\/a><\/p>\n<table cellspacing=\"0\" cellpadding=\"4\">\n<tbody>\n<tr>\n<td>are ratings related to learning?<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><a name=\"learning\"><\/a>the most acceptable criterion  for good teaching is student learning.there are consistently high  correlations between student ratings of the &#8220;amount learned&#8221; in a course  and students&#8217; overall ratings of the teacher and the course. even more  telling are the studies in multi-section courses that employed a common  final exam. <a href=\"#note1\">[2]<\/a> in general, student ratings were the highest for instructors whose  students performed best on the exams. these studies are the strongest  evidence for the validity of student ratings because they connect  ratings with learning.<\/p>\n<p><a href=\"#top\">back  to top <\/a><\/p>\n<table cellspacing=\"0\" cellpadding=\"4\">\n<tbody>\n<tr>\n<td>are ratings affected by situational variables?<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><a name=\"situational\"><\/a>the research says that ratings are robust and not  greatly affected by situational variables. but we must keep in mind that  generalizations are not absolute statements. there will always be some  variations. for example, we know that required, large-enrollment,  out-of-major courses in the physical sciences get lower average ratings  than elective, upper-level, major courses in virtually all other  disciplines. does this mean that teaching quality varies? not  necessarily. what it does show is that effective teaching and learning  may be harder to achieve under certain sets of conditions. there is a  critical principle for evaluation practice embedded here: to be fair,  comparisons of faculty teaching performance based on ratings should use  sufficient amounts of data from similar situations. it would be grossly  unfair to compare the ratings of an experienced professor teaching a  graduate seminar of ten students to the one-time ratings of a new  instructor teaching an entry-level, required course with an enrollment  of 300.<\/p>\n<p><a href=\"#top\">back  to top <\/a><\/p>\n<table cellspacing=\"0\" cellpadding=\"4\">\n<tbody>\n<tr>\n<td>do students rate teachers on the basis of expected  (or received) grades?<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><a name=\"grades\"><\/a>this is currently the most contentious question in  ratings research. there is consistent evidence of a relationship between  grades and ratings: a modest correlation of about .20. the multisection  validity studies (mentioned in question 3) provide the most solid  evidence that ratings reflect learning (a correlation of about .43).  these findings lead to the conclusion reached by most researchers: that  there should be a relationship between ratings and grades because  effective teaching leads to learning which leads to student achievement  and satisfaction. ratings simply reflect this sequence.<\/p>\n<p><a href=\"#top\">back  to top <\/a><\/p>\n<table cellspacing=\"0\" cellpadding=\"4\">\n<tbody>\n<tr>\n<td>can students make accurate judgments while still  involved in their schooling?<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><a name=\"judgment\"><\/a>some argue that students  cannot discern real quality until years after leaving the classroom.  there is no research proving this statement: however several studies  compare in-class ratings to ratings by the same students the next  semester, the next year, immediately after graduation, and several years  later. <a href=\"#note1\">[3]<\/a> all these studies report the same results: although students may  realize later that a particular subject was more or less important that  they thought, student opinions about teachers change very little over  time. teachers rated highly in class are rated highly later on, and  those with poor ratings in class continue to get poor ratings later on.  this question is connected to the larger technical matter of overall  reliability of ratings. the research indicates that ratings are very  reliable. whether reliability is measured within classes, across  classes, over time, or in other ways, student ratings are remarkably  consistent.<\/p>\n<p><a href=\"#top\">back  to top <\/a><\/p>\n<table cellspacing=\"0\" cellpadding=\"4\">\n<tbody>\n<tr>\n<td>guidelines for good evaluation practice<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><a name=\"guidelines\"><\/a>in addition to emphasizing that student ratings are an  important part of evaluation, theall also suggests several rules for  improving the entire teaching evaluation process.<\/p>\n<ul>\n<li>establish the purpose&#8217;s of the evaluation and who the  users will be<\/li>\n<li>include stakeholders in decisions about evaluation  process and policy<\/li>\n<li>keep in mind a balance between individual and  institutional needs<\/li>\n<li>publicly present clear information about the  evaluation criteria, process, and procedures<\/li>\n<li>establish a legally defensible process, including a  system for grievances<\/li>\n<li>be sure to provide resources for improvement and  support of teaching and teachers<\/li>\n<li>build a coherent &#8220;system&#8221; for evaluation, rather  than a piecemeal process<\/li>\n<li>establish clear lines of responsibility\/reporting  for those who administer the system<\/li>\n<li>invest in the superior evaluation system and  evaluate it regularly<\/li>\n<li>use, adapt, or develop instrumentation suited to  institutional\/individual needs<\/li>\n<li>use multiple sources of information for evaluation  decisions<\/li>\n<li>collect data on ratings and validate the  instrument(s) used<\/li>\n<li>produce reports that can be easily and accurately  understood<\/li>\n<li>educate the users of rating results to avoid misuse  and misinterpretation<\/li>\n<li>keep formative evaluation confidential and separate  from summative decision making<\/li>\n<li>in summative decisions, compare teachers on the  basis of data from similar teaching situations<\/li>\n<li>consider the appropriate use of evaluation data for  assessment and other purposes<\/li>\n<li>seek expert, outside assistance when  necessary\/appropriate<\/li>\n<\/ul>\n<p><em>the bottom line is: good practice leads to good  decisions. <\/em><\/p>\n<p><a href=\"#top\">back  to top <\/a><\/p>\n<table cellspacing=\"0\" cellpadding=\"4\">\n<tbody>\n<tr>\n<td>notes &amp; bibliography<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><strong>notes <\/strong><\/p>\n<ol> <a name=\"notes\"><\/a><\/p>\n<li><a name=\"note1\"><\/a>marsh, h. w. &#8220;students&#8217; evaluations  of university teaching: research findings, methodological issues, and  directions for future research.&#8221; international journal of educational  research, 1987, 11, 253-388.<\/li>\n<li><a name=\"note2\"><\/a>cohen, p. a. &#8220;student ratings  of instruction and student achievement: a meta-analysis of multisection  validity studies.&#8221; review of educational research, 1981, 51, 281-309.<\/li>\n<li><a name=\"note3\"><\/a>centra, j. a. determining  faculty effectiveness. san francisco: jossey bass, 1979; and frey, p. w.  &#8220;validity of student instructional ratings. does timing matter?&#8221;  journal of higher education, 1976, 3, 327-336.<\/li>\n<\/ol>\n<p><strong>references and bibliography <\/strong><\/p>\n<ul> arreola, r. a. developing a comprehensive faculty evaluation  system. 2nd ed. bolton, ma: anker publishing company, 2000.<br \/>\nbraskamp, l. a. and j. c. ory. assessing faculty work. san  francisco: jossey bass, 1994.<br \/>\ncentra, j. a. reflective faculty evaluation. san francisco: jossey  bass, 1993.<br \/>\nknapper, c. and cranton, p., eds. &#8220;fresh approaches to the  evaluation of teaching.&#8221; new directions for teaching and learning 88  (winter 2001).<br \/>\ntheall, m., p. a. abrami, and l. mets, eds. &#8220;the student ratings  debate. are they valid? how can we best use them?&#8221; new directions for  institutional research 109 (2001).<br \/>\ntheall, m., and j. l. franklin. student ratings in the context of  complex evaluation systems. in m. theall and j. franklin, eds. &#8220;student  ratings of instruction: issues for improving practice.&#8221; new directions  for teaching and learning 43 (1990).<\/ul>\n<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>this article was originally published in the fall 2003 issue of the cft&#8217;s newsletter, teaching forum. micheal theall, ph.d. the following article is reprinted with permission of the author and of focus on faculty (fall 2002), a publication of the brigham young university faculty center, ed. d. lynn sorenson. michael theall has twenty-six years of&#8230;<\/p>\n","protected":false},"author":452,"featured_media":0,"parent":874,"menu_order":6,"comment_status":"open","ping_status":"closed","template":"","meta":{"spay_email":"","_links_to":"","_links_to_target":""},"tags":[],"acf":[],"_links":{"self":[{"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/pages\/1595"}],"collection":[{"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/users\/452"}],"replies":[{"embeddable":true,"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/comments?post=1595"}],"version-history":[{"count":0,"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/pages\/1595\/revisions"}],"up":[{"embeddable":true,"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/pages\/874"}],"wp:attachment":[{"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/media?parent=1595"}],"wp:term":[{"taxonomy":"post_tag","embeddable":true,"href":"\/\/www.imrbdigital.com\/wp-json\/wp\/v2\/tags?post=1595"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}