There are a lot of good comments appended to my post on taking college teaching seriously, but I wanted to highlight one in particular. According to Kramer of Our Take,

One problem that consistently comes up when I talk with faculty who broadly agree is the difficulty in defining some easy metric to assess quality teaching.

Plenty of metrics exist for research (at least in the physical sciences where I spend most of my time) - things like number of publications, citation impact (somehow normalized for subdiscipline) while perhaps not including all good research clearly represent some subset.

I'd be curious to hear any suggestions as to effective ways to represent teaching quality those in university administration.

Every college teacher can point to other teachers who abuse the fact that no one sees them teaching but their own students, who abdicate their responsibility to shape their courses around the needs of students by using the classroom as a political soapbox or an intellectual hobbyhorse or a group therapy session, or similar. But college teachers know, too, that their development as teachers depends heavily on that much maligned and paradoxical phenomenon, "academic freedom." Great teachers are independent thinkers and creative actors; they have devised their own ways of inhabiting the classroom, of imparting information, of facilitating discussion and of provoking thought, and they were able to do so because they had the space in which to do so. How to balance the pressing imperative of assessment with the equally pressing imperative of giving teachers the room they need to develop and hone their craft? Thoughts are welcome.

While the standardized questionnaires currently in general use are indeed irresponsible, the survey methodology could be improved without becoming intrusive.

1. The questionnaires should be administered online, and students should be required to sign on to the course questionnaire in order to receive a grade in the course. Naturally, students cannot be required to answer any questions, but they can be required at least to abstain. The purpose of this policy would be to eliminate the gross sampling bias that occurs when questionnaires are answered by the students who happen to be in class on a particular day.

2. Students' responses should be stored with their ID's, protected by the same security measures that are used to protect their transcripts and financial accounts. This would allow the standardization of an individual student's scores across different courses (i.e., by re-expression in terms of standard deviations from that student's mean scores). It would also allow correlations between students' evaluation of the course and such things as: students' grades, students' year of study, students' major, whether students' satisfied the course prerequisites, etc.

3. Students should be asked for follow-up evaluations of their courses at graduation. Which courses helped them most in learning to handle college work? Which courses are most memorable? Which courses are they happiest/sorriest to have taken? These evaluations may differ significantly from the evaluations given at the end of the semester.

4. Evaluations should be made available in the form of a database on which various reports and statistical measures can be run.

I tend to agree that the student questionnaires are problematic. It's kind of like a company evaluating a salesman based primarily on customer feedback...some of the things the customer tells you may be very valuable ("really understands our business-always there when we need him")--but the results may be skewed by other factors ("gives us *really big discounts* even though we actually would still buy your product without them"). Ideally, the student evaluations should be used as one data point, in conjunction with other factors, but quantitative data seems to exert a magical power than causes other information sources to be ignored.

If a university has developed a culture in which teaching is taken seriously, then the assessments of other respected professors (based on direct observation and their interpretations of student comments) would be a partial solution. This doesn't work, though, if teaching is a poor stepchild to research..creating a bit of a chicken-and-egg problem.

The other thing that may be useful is the opinion of former students after a lag of 5-10 years...probably too late to be useful for many of the decisions that matter, though.

"The other thing that may be useful is the opinion of former students after a lag of 5-10 years."

As I read Erin's post, I was thinking that I wish I could thank my organic chemistry professor for being the wonderful teacher he was. I kind of knew it at the time, but comparing the preparation I had with that of people who went to other schools confirms it over and over. He's retired, of course. I wish my daughter could have his class when she does her pre-pharm.

The problem with student evaluations, at least in my experience, is that I often suffer for doing my job. If I actually hold certain students to a standard, they will slam me on evaluations and lower my overall "grade" for the course. Some of the people I've known with the highest evaluation scores are those whose courses are the easiest to receive an "A" in. My classes, through no conscious effort of my own, generally wind up with a typical bell curve.

When I encounter students years down the road, though, I am often thanked for my standards, and told that I actually taught them more abour writing and critical thinking than many of their other instructors.

I guess I just have no faith in the comments of 18 year old students on the quality of the education they are receiving. It's like asking the taxpayer whether or not the IRS is doing a good job.

I have two distinct lines of thought that I will attempt to describe:

1) Can students fairly evaluate a teacher who holds them to high standards? Yes. I truly believe that even college freshman are capable of deciding if you are a quality teacher. Students get a little cranky if you change the standards during the semester, fail to articulate standards for grading, or, lack some sort of rubric which describes what you are looking for in grading. I understand those concerns (I live on both sides of the desk as a grad student). If I understand how I will be graded on an assignment and can transparently see how I earned a grade, I have no problem. If I feel like I was given a grade rather than earning it, well, that's when I have a problem.

That is, you tell students what the contract is at the beginning of the semester, and you stick with it, no matter the difficulty of your course, and your students will evaluate you in a similar manner; fairly.

2) The evaluation of quality teaching: It's true, we have very little ability to talk about what it means to see quality teaching (that's true at all levels of education, not just college). I would suggest reading a piece by Fenstermacher and Richardson called "On determination of quality in teaching" (or something similar, available on Fenstermacher's web page).

But, here's a proposal that can be individualized:

1) Ask the professor what the learning goals are for the class at the beginning of the semester.
2) Jointly determine how one could evaluate those goals.
3) give a pre/post assessment.
4) ask the students to also characterize their study habit.
It's a fairly rough metric, but if you control for student study habits a pre/post metric on the teaching goals that the professor has identified as important could be reasonable if learning is one of our means of describing quality.

There are other, less quantitative suggestions I could make as well. Portfolios are an example. Check out www.ntbs.org, they are the national board for teaching standards-it's a k-12 organization, but devoted to teaching quality. It was started by Lee Shulman (now at the Carnegie Center at Stanford-looking at the scholarship of teaching at the college level-also worth checking out).

tim

In my department, we're about to start instituting peer reviews, which otherwise haven't been necessary, after tenure, unless one's going up for full professor, and after that aren't needed at all. I'm happy about this. There needs to be accountability -- we've got some wonderful teachers, we've got some adequate teachers, but we've also had our share of full professors who weren't teaching a damn thing. Or worse, teaching dreck.

The student reviews are partly useful, partly not. But even if we're able to tweak them successfully, we need to be watching each other. I need to not be the only professor in the department who knows what I'm doing in my classroom, despite the fact that the university gave me tenure.

And though certainly this can be abused -- while sitting on the college tenure review committee I've seen clear cases of peer reviews being misused -- it's also true that those cases were clear. If we've got more than one peer coming and reviewing us, various points of view will manifest.

We're also about to institute peer evaluations, which I suspect will be as prone to self-serving manipulation as the student scantron forms. At my institution, there's another factor that militates against real accountability in teaching--the "collegiality" factor. Where I work, any criticism, no matter how warranted or constructively offered, is routinely taken as an offense against collegiality. To imply that someone else isn't doing as good a job as they think they are always brings a retort of "that's not collegial." The peer evaluations will, if done by people in the same department, be exercises in cheerleading, by virtue of the organizational fallback of one hand washing the other. If done by others, any criticism--or anything but a rave--will be denounced as uncollegial.

I agree with Tim that student evaluations would be improved if they were based on goals and understandings established at the beginning of the semester. But I think it's unrealistic to think that most courses can include an agreed-upon method for evaluating achievement of those goals. In many disciplines, the goals will be too complex and subtle ("a better appreciation for ..." , "greater insight into ..."). But pre- and post- testing needn't be the point. The point can simply be to train the students' attention on the question whether they gained what they and the instructor projected when the course began. Just focusing their minds on that question would be a great improvement over the general popularity-contest questions ("This was an excellent course ... ."

You don't mention the problem with coverage. Suppose a prof only covers 2/3 of the topics. In other words, he waters down the course. If only student evaluations are used, the students think he's great. They mastered the course material better because there was less of it, compared to a course taught by a more conscientious prof. A student does not know, and is not qualified to judge a teacher, because he does not know what is expected in the course.

Over reliance on student evaluations, in my experience, leads to both grade inflation and, more importantly, watered-down courses. This is particularly important in my field, engineering, because a large number of courses are dependent on the prerequisite courses covering all of the material. However, the individual professor, who waters down courses, gets teaching awards (and raises) based on the student evaluations of his work. Meanwile, the curriculum suffers.