Monday, April 23, 2012

Evaluating evaluations: Low student response affects teacher evaluation statistics

Lindsey Treffry | Argonaut

It’s nearing midnight on Sunday before your final exam. Coffee in hand, you decide to go through your Spanish flashcards one more time. A three-week-old email from your Spanish instructor sits in your inbox, reminding you to fill out her instructor evaluation before the clock strikes 12 a.m. Your caffeine-surged hand shakes as you close your laptop, ignoring her request, and head to the kitchen to cook some Top Ramen. It’s going to be a long night.

“(The online student evaluations of teaching systems) are used a lot more than students realize,” said Archie George, director of University of Idaho Institutional Research and Assessment. “If students knew that, they’d be more inclined to fill it out.”

The IRA carries out policy and development of the online SET system, which replaced paper evaluations in the early 2000s. The digital format improved the reach of instructor surveys to off-campus students and cut a five-week process of sorting, scanning, compiling and distributing more than 32,000 forms. Issues with the current system include abnormal course dates, student log-ins and technology issues.

George said the biggest issue is not with the online system, but with student response rates.

“The logistics are great,” said Karen Humes, UI’s geography department chair. “Just the collection needs to be improved so we can have a genuine majority,”

In Humes’ department, she said there’s not a single class that has more than a 60 percent return rate of evaluations from students.

For Pamela Bathurst, associate professor of voice, the return rate of teaching evaluations has been meek as well. She said the lack of evaluations do not provide the feedback necessary for instructors.

“(Instructors) absolutely need to have something to look at in order to see progress and also … to be able to see where maybe they can tweak things so that they have a class that is the best class,” Bathurst said.

She said evaluations are even taken into consideration when reviewing faculty positions.

Instructors undergo an annual evaluation process that analyzes the extent and quality of their teaching. This process can lead to merit-based raises, promotions, tenured positions or firings.

“Students don’t realize how much these matter in people’s careers — especially the untenured,” Humes said.

When instructors are up for tenure, a report with a summary of student evaluations provides a summary score and is placed in every professor’s review packet. According to Institutional Research Analyst Chris Lighty, these are compared to department, college and university scores.

“If you’re denied tenure, you get one more year and then you’re out,” Humes said.

Humes said low student response rates also create a bias. For example, Humes said a small class of 20 students could receive a 50 percent response rate. With a few low scoring evaluations, she said this could really impact an instructor’s job.

“(The students who fill out the surveys are) people who really loved a professor or really hated a professor,” Humes said. “It is not in the middle.”

Kenneth Sprenke, professor of geophysics, environmental and planetary sciences, said evaluations either flatter you or they’re unfair.

“They either say something outrageous or that really annoys you,” Sprenke said. “You can lose sleep over that. Who wants to be told you’re doing a great job, when you’re not necessarily doing a great job?”

Bathurst said there are various reasons students in the middle sector don’t fill out evaluations.

“I think (students) get busy,” Bathurst said. “There are papers that are due, tests to study for. Things they feel are at the top of the list. (The evaluations) get shuffled to the bottom of the list. They can go online and do it and it doesn’t take very long, but it’s just one more thing.”

Although the majority of the university follows the same online evaluation process, there are exceptions. All law courses, for example, use paper evaluations of a narrative form. Elizabeth Barker Brandt, professor and associate dean of Faculty Affairs at the College of Law, said the evaluations are passed out in class. She said each professor is responsible for distributing evaluations during the last quarter of the semester and before a final exam.

“I’ll pass out the evaluations in my class and I go wait in my office,” Brandt said. “I designate a student to delegate. They take 15 or 20 minutes, collect them and bring them to the office and we return to class.”

Brandt said the whole university used to use a system similar to the College of Law.

“Then the university went to numerical and then online evaluations,” Brandt said. “Our faculty had really big concerns about that. We figured if we went to numerical, students would not write comments. We thought if we did not get the comments, it would not be as resourceful.”

Brandt reads all faculty evaluations and writes a summary, which is then submitted to the dean.

“We’re sacrificing the administrative efficiency because our faculty has really wanted to have the detail that a narrative provides,” Brandt said.

She said there is an 85 to 95 percent return rate per class for evaluations.

“Students don’t have a reason to forget to do it,” Brandt said. “They sit in class and do it. Students really feel like the narrative matters. They all know that I read them all, every semester.”

While it is unlikely that the entire university would return to a paper system, Bathurst said the system seemed to work well.

“The rate was much higher because students were already in the class, but I believe I understand the reason for the change — it makes it easier for the people going through all of them,” Bathurst said. “But I also saw (during the online switch) there was a drop in the amount of evaluations actually turned in. It became less mandatory and there was not time made for it. Going back to paper evaluations is not an option that would be looked at positively, but I think that’s it.”

Other than an unlikely return to a paper system, George offered other solutions to the response rates.

“The No. 1 recommendation I hear is for students to not be able to see grades for a period of time after grades have been posted, if they haven’t submitted evaluations,” George said.

Humes said some instructors give class incentives. While instructors cannot see who submitted a class evaluation, they can see the number of students who have. She said some instructors reward students with extra credit if they reach a certain percent of evaluation responses.

Bathurst suggested instructors require students to bring their laptops to class to fill out instructor evaluations.

“(Students) think their vote doesn’t count,” Bathurst said. “It’s the same thing with evaluations. It’s the idea of not feeling like they really count but that is not the case — they definitely count.”

The evaluation period for courses begins three weeks before a course end date. The period to evaluate full-semester courses ends the Sunday before final exam week — at 11:59 p.m. May 6.

“If (students) look at it as part of contributing to the overall excellence of the classes offered in the UI — if they see themselves being part of that — maybe they might take the time to give input,” Bathurst said. “It is not ignored. We all want everything to be the best we can. We are consciously looking to better, better, better our classes and our offerings and (student) contribution is viable.”

As seen in April 9 issue of The Argonaut.

No comments:

Post a Comment