![[Critical Mass]](/archives/cmlogo.gif)
![[dot]](/archives/cmdot.gif){kind=small}
July 16, 2008
![[feather]](/archives/feather.gif){kind=small}
Getting testy
Colorado State is taking the idea of measuring learning outcomes--and of becoming accountable for them--seriously. It's launching a pilot testing program this fall that will measure what students know coming in, in order to be able to compare that to what they know when they leave. Of course this is raising all sorts of hackles among faculty members who object to the idea that testing can be a means of seeing what--and how well--college students learn.
George Leef, who writes regularly, intriguingly, and contrarianly about the value of college education, is all over it:
Students spend years and a great amount of money in college, yet we have to take it on faith that the pursuit of the degree is sensible.That was all right for most people in bygone days when college didn't cost so much and academic standards were solid enough to create a strong presumption that a student had gained in knowledge and skills from having earned a degree. But the cost of college has skyrocketed and academic standards have been plunging. Those facts have many people wondering if higher education is worth it.
One school, Colorado State University, is taking an initial step toward answering that question. According to this Denver Post story, before classes begin this fall, one hundred CSU freshmen will take a test to measure their writing and reasoning abilities.
"People want evidence that their tax dollars and tuition money are being well-used," says CSU vice provost Alan Lamborn. That's putting it mildly. With many college graduates ending up in the competition for jobs with low educational requirements and low pay – a point I have written about here -- proof that studying at a college or university demonstrably adds educational value for students could be the equivalent of the Underwriters Laboratories (UL) seal. It would give students and parents some assurance that its degrees are worth more than the paper they're printed on.
Colorado State's efforts hardly warrant a full-throated cheer, however. Incoming freshmen don’t have to take the test -- the Collegiate Learning Assessment -- and the inducement for them to do so is rather weak; they get to move into their dorms a day early and get a $10 voucher good in the university cafeteria. If you only test a rather small number of students who are probably the least test-averse, you won't learn much about educational value added. Still, it's a step in the right direction.
The Collegiate Learning Assessment is an interesting development. Designed in 2002 by a high-powered research group, the three-hour test is not your typical multiple choice or fill-in-the-blank exam. Instead, it requires students to write three essays that are meant to probe their thinking ability. Several dozen schools including Harvard and Duke are now using CLA, but only for internal evaluation. So far, no one is using CLA the way hotels and restaurants use Zagat ratings – to attract more business.
Perhaps the leading reason why colleges aren't racing to show how well they succeed in educating their students is that the faculty, which has huge clout at most schools, isn't enthusiastic about the idea testing to make comparisons. The Denver Post article quotes a professor at the University of Colorado: "But many faculty will be perturbed at the concept of a standardized test that could be used to blame them for inadequate teaching. As if that's the sole factor when a student doesn't succeed."
That defensiveness is revealing. A lot of professors know that their courses are short on content and give students credit for very little learning. They like things the way they are. If CLA or some other test were used to show the lack of student improvement, the hunt would be on to identify the weak links in the school. A frightening prospect to those with a vested interest in avoiding measurement of their efforts.
Higher education is ready for an entrepreneurial move by some college president who will get serious about the need to show that his institution adds to its students' foundation of skills and knowledge. Furthermore, the goal of such testing ought to be not only for internal use to show where the school is more or less effective, but as a metric by which individual students can demonstrate their accomplishments.
Suppose that a major state university adopted the Collegiate Learning Assessment or some other test that it devised to show how much each student improved in fundamental knowledge and skills (and also specific mastery in his major field) over the course of his studies. Its graduates would then be able to say to prospective employers or graduate schools, "I didn't just get a degree there, but as you can see from my scores, learned a lot."
Knowing that one university was testing in this manner would put pressure on others to follow suit.
There's more. And even as a thought experiment, it's worth thinking about. While it's certainly true that poorly devised tests can cramp educational effectiveness rather than enhance it, surely it ought also to be the case that well-devised tests would be revealing in ways that help, rather than hinder, professors' abilities to design courses, departments' abilities to structure majors, and faculties' abilities to set general curricular requirements. It all depends on how such tests are written, how they are evaluated, and how the results are used. If faculty worry that they tests be counterproductive if they are poorly designed, it might make more sense to get involved with designing and implementing them than to refuse to engage at all.
Trackback Pings:
TrackBack URL for this entry:
http://www.erinoconnor.org/movabletype/mt-tb.cgi/1500
Comments:
I'm skeptical of the CLA because there is scant independent evidence of what the CLA measures. Could the same test measure what a student learned in your English classes, in my history classes, or in someone else's anthropology? I suspect that if I argued that there was something called "critical thinking" absent a disciplinary curriculum, you'd bristle. So what makes the CLA a good measure, as opposed to the work my students produce in my classes?
Sherman...I suspect there *are* mental skills which are developed by English, history, and anthropology. The ability to read & understand complex documents, for example, should be improved by the study of each of these. Given appropriate test results, regression & factor analysis could be used to assess the degree to which this is correct.
My view would be that you learn to understand complex documents better by taking courses such as the above than by taking some "understanding complex documents" class, and you will likely learn inductive & deductive reasoning well by studying science and geometry than by taking somebody's idea of a "critical thinking" class.
not against testing or examining effectiveness of any program, but colorado taxpayers don't pay shit for their state universities. colorado ranks 16 in income level, but somewhere around 48 when it comes to funding education. college costs are skyrocketing, in part, because states continually cut the college's budgets.
and "a lot of professors know that their courses are short on content"--really? i don't buy this--were your courses short on content, erin? is that why you left penn? probably not. i'm sure there are some professors that fit this bill; however, they are most likely a minority. one reason that test may mean little: a student takes the typical two required comp courses in his freshman year, but is in a major that requires little writing. when the student takes the writing portion of the test and gets a low score the teacher receives the blame. the problem: the student could have very well worked hard and developed some writing skills (or calculus or whatever discipline), but the student could also have let them atrophy.
will faculty be able to engage in the testing progress? will they be asked? somehow, i doubt it.
My department does "outcomes assessment" on all our majors at the rising junior and graduating senior levels. The assessment is designed to measure how well we're doing in helping students achieve various departmental goals, and to be used by the department itself in tweaking its "delivery." It's not designed to be used to compare my program to other English programs, nor to help outsiders compare my institution to other institutions, nor to generate "evidence that...tax dollars and tuition money are being well-used."
Leef seems to be talking about assessing very broad skills in a way that would facilitate comparisons between institutions. I doubt that any such macro-level test would provide much micro-level help to individual department's trying to improve. If we don't want local faculties' curricular decisions to be driven by a national Mega Test, then we're talking about developing and implementing two separate projects here, one local, the other national (or designed to be articulable on a national scale). Anyway, at my institution, and I'm sure many others, there's already testing that meet's Erin's criteria, but nothing like what Leef is calling for.
I'd be happy to "get involved with designing and implementing" a second layer of testing--but not for free. Maybe the Center for College Affordability and Productivity could hire me as a consultant. (I know it helps "productivity" when faculty whose plates are already full take on extra work for free, but it's kinda not fair. Then there's the assumption that we should just suck it up and work for free on projects pushed by people who routinely insult us. Just explaining a couple of the sources of the defensiveness Leef mentions, Erin.)
Eveningsun -- I would *love* to know more about your department's internal testing model. Would you be willing to share some specifics about it? How does the test work? What kinds of goals is it keyed to? Are you testing knowledge of content? Theory? Skills? All of the above? Other? Has the testing led to constructive revisiting of the major curriculum, and if so, how has that worked?
Erin, my particular program's assessment is portfolio-based. All majors must turn in a "sophomore portfolio" and a "senior portfolio." The first consists of a freshman essay from Comp 1, a research paper from Comp 2, a formal essay from the required "study of lit" course, a sample of hand-written, unassisted writing (typically a bluebook from a development-of-civ exam or somesuch), and a cover letter summarizing each work and self-appraising its quality. The second is made up of a cover letter, four samples of upper-division work, and a resume. (The resume is not really part of the assessment; we just require it to make sure our seniors actually have a resume when they leave here.)
Each item in the portfolio is numbered to correspond to the scoring rubric, in order to ensure that evaluators are evaluating the right essays in the right evaluation categories, which are expository writing, critical thinking, inquiry and research, and analysis of literature. Using the rubric and scoresheet, faculty read through and evaluate each portfolio. The expository writing rubric is pretty detailed, with the usual stuff--content (equals significant topic, accurate statements, clear thesis, effective use of supporting evidence, etc.), organization, diction and style, grammar, and mechanics. Evaluation in the other areas (critical thinking, inquiry and research, analysis of literature) is basically holistic. We just read the essay and score it. (The entire faculty gets together for a norming and discussion session first.) Then the numbers are spreadsheeted and crunched and recorded.
How are the numbers used? Each program at my school does a program review on a five-year cycle. In my department, at least, much of the review concerns enrollment and budget matters, but the results of the evaluation described above form the heart of the curricular part of the review. In fact the program-review procedures explicitly require that we show how assessment results drive curricular revision. For us the perennial underperformer seems to be "inquiry and research." We're never particularly happy with any of the results, but lately we've been especially unsatisfied with our majors' ability to frame good questions ("inquiry") and then locate and use quality sources relevant to those questions ("research"). Our students seem to be much better at projects that require primary sources only (e.g., analyzing a plot). In response we've done a couple of things. We've dropped the oral communication component of our Comp 2 course, which leaves more time for teaching the "research" part of the research-paper component. And we've arranged with an instructional librarian to add a 1-credit lab component to our required "study of lit" course, which is a prerequisite for all upper-division lit courses. Starting (we hope) this spring, our majors will spend 15 contact hours in the library getting detailed instruction on using Academic Search Complete, LexisNexis, etc.
BTW, IIRC, our assessment process stems from an accreditation report. The accreditors didn't tell us how we had to assess-and-review, merely that we had to do so. I mention this because it seems to me lots of other schools accredited by the same agency must be performing similar kinds of assessments, which is why I think people are wrong to make blanket statements like "Higher ed never does assessment! There’s no accountability!" What's missing is not assessment per se, but inter-institutional assessments. And it seems to me that inter-institutional assessment of individual programs could be cobbled together without some kind of Grand National Test. Some of our English grads go on to grad school, and thus take the GRE; it wouldn't seem too hard to pull out their GRE scores and see how those scores are changing longitudinally and use them to compare our program to those at our peer institutions. Given the amount of testing already done, and the amount of data already floating around out there, it seems ludicrous for people to cry "We need more testing! There's no accountability! How can you tell how good you're doing if you don't measure your performance?"
Sure, it would be better to design a separate assessment specific to each question someone might want answered. But consider the tremendous number of questions that might legitimately be asked by so many different constituencies (state legislatures, state higher ed commissions, individual trustee boards, the Dept. of Ed, graduate and professional schools, accrediting agencies, and, of course, individual institutions and programs). By the time I finished my formal education, I had taken the California Test of Basic skills several times, the SAT, a college writing placement exam, and the GRE (plus, of course, innumerable tests in individual classes). A student in my home state following the same path today will have done all that, plus taken the Academic Profile exam and turned in a sophomore and senior assessment portfolio. And filled out a detailed exit survey. They'll also have filled out the NSSE survey several times. Not enough assessment? What the heck--just pile on some more! It's not like it takes time or costs money or anything.
Post a comment:
