About Critical Mass [dot] Writing [dot] Reviews [dot] Contact
« previous entry | return home | next entry »

September 29, 2006 [feather]
Measuring value added--or not

RIchard Vedder's College Affordability blog has a number of sharp and informative posts on the Spellings Commission's final report on higher education--just keep scrolling to get his take on where the report undermined itself by pulling punches, and on where Vedder sees the whole thing going.

Note, also, his post on a recent study issued by the Intercollegiate Studies Institute. Entitled The Coming Crisis in Citizenship: Higher Education's Failure to Teach America's History and Institutions, the ISI report actually attempted to do something that the Spellings Commission has--quite controversially--recommended: It attempted to measure how much knowledge of American history and civics college students gain over the course of their college educations. The results are telling:

The report says:

1. America's college students are woefully inadequate in their knowledge of American history, the operations of our political institutions and foreign relations, and economics.

2. The amount of learning about these core subjects occurring during the college years on average is appallingly little, approaching zero.

3. The "value added" (increase in learning) about these topics is typically much greater in the less well known, less expensive private institutions than in the elite yuppie universities like Harvard, Yale and Princeton.

Many will discredit the report by proclaiming that the Intercollegiate Studies Institute is a conservative organization. True, but that is irrelevant in this case. ISI hired the well-regarded polling group at the University of Connecticut to do the testing. A non-partisan, fact-based test of 60 multiple choice questions were designed to test knowledge of key events, persons, and ideas that relate to our civic institutions and our heritage. Several of the questions were taken directly from the highly regarded National Assessment of Economic Progress exams administered to 17 year old Americans. More than 14,000 students were surveyed on 50 campuses, roughly one-half of them freshman and the other half seniors (about 140 of each on average on each campus).

At 16 of the 50 schools, the senior scores averaged lower than the freshman ones, implying that little or no learning of these key topics was occurring at these schools. Some of those schools: Yale, Duke, Cornell, the University of Virginia, and the U. of California at Berkeley. The five schools showing the largest gains in knowledge were Rhodes College, Colorado State University, Calvin College, Grove City College, and the University of Colorado at Boulder --hardly viewed at top schools by the more popular magazine rankings.

The Spellings Commission and the Department of Education have been crying for measures of student performance, of the value added by attending colleges. ISI has actually done something to measure value added. Their findings are scary, exciting, and particularly humiliating for expensive elite private schools. Seniors at little known Grove City College did as good or better than students at more prestigious Williams College or Washington and Lee. George Mason students outperformed those at Michigan or Berkeley. This is the type of information parents and students need in evaluating schools, and this kind of test result is precisely why many schools, including the private elite ones, will likely fight this sort of testing tooth and nail.

Needed: Someone to give ISI (or CCAP) $1 or $2 million to replicate and expand this study -- perhaps to new subjects, to more universities, to bigger student samples. Let the light shine in.

Let it shine, indeed. Then maybe we can talk more honestly about why the academic establishment is so hostile to testing--even no-stakes testing that does not affect students' progress toward degree but that does allow schools' overall effectiveness to be measured. As Mark Bauerlein put it at Phi Beta Cons, the ISI study

provides precisely the kind of data that is essential in understanding the performance of higher education--the intellectual progress of students from first-year to graduation. Unfortunately, it is just the kind of assessment that universities will resist. Humanities folks don't like the idea of selecting certain knowledges as central.

Administrators will say that schools are too diverse in their missions to warrant comparisons. Top-tier schools don't want any new measurements that might compromise their standing. And lesser schools worry about findings that might reflect poorly on their benefits, angering legislators and parents who may be footing the bill.

But what could be more fair than to ask for some information on the acquisition of learning?

For a different sort of meditation on standardized testing for college students, see Tim Burke's post on the issue, as well as his commenters' responses.

posted on September 29, 2006 9:01 AM


There's a very good reason for college's to be hostile to the idea of stanardized tests for college students: colleges don't provide a standardized education.

Demanding standardized tests for universities is a backhanded way of demanding a standardized curriculum for college students.

This strategy worked for the Right at the K-12 level. Change the tests, and you force curricular changes. Your expressed ideology might be all about state rights and local control and small federal government, but by putting a certain kind of national testing and funding structure in place, you guarantee that each locality will have to reshape itself to fit.

While primary and secondary schooling should provide a standard curriculum, it's debatable whether such standardization is necessary for college students.

Any core curriculum will seriously reduce college students abilities to shape their training for future careers. The student who wants to study Latin American history as a major will need Spanish and Portugese classes, along with economics and political science and foreign policy classes (and perhaps some art history or comp lit courses). But if that student is also forced to take Intro to Western Civ, Intro to American History, Intro to European Literature, Intro to Philosophy, Calculus I, Biology, and various other high-school-replacement courses, the student's ability to shape his course of study is deeply affected.

Colleges have no responsibility to teach American history to all graduates. Given the increasing number of non-American students attracted to our universities, it's patently ridiculous.

Posted by: Jonah Goldberg at September 29, 2006 1:00 PM

That's "colleges" in the first sentence, of course.

Posted by: Jonah Goldberg at September 29, 2006 1:01 PM

I have to admit to ambivalence about the results of the ISI survey. I can't comment on their methodology -- were the freshman and senior samples really representative and comparable? -- but their results ring true.

The ignorance of the students is probably dismaying. But if I look at it from the vantage point of the students I teach in the natural sciences, or their future employers, I'm not sure I can see the case for forcing a change.

No employer, no graduate program has ever said that they wished our "products" -- our bachelor's graduates -- knew more American history, knew more about the federal budget, knew more about the Constitution, about Jefferson's letter in which he talked about the "wall of separation between church and state".

They may wish the students knew more about gas chromatography, or math, or how to write, or more about showing up for work on time, about not bringing lunch to the the boss's office. Though truth to tell, I've not heard many complaints about those things either.

But more about American history, civic matters? Never any complaints.

Posted by: chemistry prof at September 29, 2006 3:00 PM

One further point. Let's remember that "unacceptable" levels of knowledge do not mean that past students did better. As Berliner and Biddle argue regarding public school stats in *The Manufactured Crisis*, "Standardized tests provide /no evidence whatever/ that supports the myth of a recent decline in the school achievement of the average American student."

Regarding a Diane Ravitch poll about students' cultural literacy, B & B point out the research of Dale Whittington, which proved that on 43 comparable topics, 1980s students knew more than past students 1/3 of the time, knew as much 1/3 of the time, and knew less 1/3 of the time. Whittington also proved "that students seem /never/ to haved known as much social studies material as the test developers wanted them to know" (*Crisis* 34).

Posted by: Jonah Goldberg at September 29, 2006 5:49 PM

There's a very good reason for college's to be hostile to the idea of stanardized tests for college students: colleges don't provide a standardized education.

That is true but their should be a demonstrable correllation between time invested in higher education and one's capacity to answer questions that are not composed of disciplinary esoterica. (And it would appear that that with regard to Cornell University, that correllation may be inverse).

One might suggest we teach the liberal arts in revivified (and necessarily selective) secondary schools and have higher education consist of ten, twenty, thirty, and forty course programs in discrete subjects, academic or vocational. We reduce the mean number of (potentially productive) years expended in tertiary institutes by half, reduce the mean debt load of the youth of the nation by half, and reduce by half the number of adults disoriented by the excess of job security and deficit of supervision characteristic of employment as a professor. Win-win-win.

Posted by: Art Deco at September 29, 2006 8:07 PM

Academic administrators will probably argue that most of the stuff on this survey was traditionally taught in K-12 schools, not in colleges, and this is at least partly true. I think the survey must be regarded as a measurement of *combined* effectiveness of K-12 and university education, rather than university education alone. The primary university contribution to the problem lies in their continued complicity with the ridiculous "education" programs that systematically keep good people out of K-12 teaching.

Posted by: david foster at September 29, 2006 9:32 PM

I think that the commenters are responding as if the student came out of college and immediately went to work productively.

I submit that that proposition is definitely not true. Whenever someone comes from college and goes to work, that person has been trained in a classroom situation which is set up to give pre-determined solutions. When that student gets into a real life situation, that solution is probably not applicable at all or does not fit the methodology of the place of employment. Whenever a graduate goes to work, there is a period while that graduate is trained in the work methods of the place of employment. At that point it is more important that the graduate be trained in thinking rather than the major emphasis being on the subject matter studied in college. For that reason I think that there is a definite place for testing. The test questions to be used to determine the quality of the education received are very much up for grabs at this point but there is a need for checking the thinking processes and not just the subject matter. How this will be done is something I am not sure of but I do think there is a need for it.

Posted by: dick at September 30, 2006 9:30 PM

I don't buy the notion that studying the subject is not so important. When you hire an electrical engineer, there undoubtedly is a period of training that has to go on, but you hire the electrical engineer rather than somebody else for real reasons. I don't believe that one generally learns to think by studying how to think. One learns to think by studying and thinking about a body of knowledge about which others have studied and thought long and hard. I believe that programs that purport to teach "thinking skills" apart from a subject matter are probably bogus.

Devising a test to measure thinking skills a hard, hard task. What is likely to come of this testing craze is simply more testing of rote learning, like this ISI test. Not completely without value, perhaps, but in no way a test of thinking capacity.

I have to say, I'm kind of amused at the sudden enthusiasm of ISI for standardized testing. They and some of their conservative allies (apparently not Jonah Goldberg) seem to want to turn the students into little lab animals to be constantly tested, if not prodded and poked, in the name of social-science/business type "accountability" and "productivity". I have always thought of ISI as a quirky outfit with a very much outside-the-business-model idea of education.

Maybe they should be careful what they wish for. If the testing mania catches on, then after the Bushies are long gone, the academic left, perhaps with some help from their semi-allies in the corporate personnel departments, will be ready to impose their own agenda in the testing regime. I suspect it's not going to be much to the liking of groups like ISI.

Posted by: chemistry prof at October 1, 2006 1:55 AM

"Learning how to think" is often used by educators as an excuse for failing to transmit any actual knowledge. Seems to me there are at least two components to "learning how to think."

First, there are elements that are domain-specific: for example, learning how to think like an electrical engineer (and this involves much more than merely memorizing formulae)

Second, there are elements that are cross-domain. These include things like the ability to read and understand complex documents, to reason deductively, to craft a persuasive argument, to know the elements of probability & statistics, to understand and be able to use the scientific method.

Interestingly, the items in the second category were well-covered by traditional liberal arts education. They are not covered at all by the kind of things that today's educators include under "learning to think."

Related: see my post Thinking and Memorizing.

Posted by: david foster at October 1, 2006 9:35 AM

Reasons why the US spends more per child and per teacher than any other country and yet obtains such a poor return on its "investment" are legion. And for a nation of "bottom-line" pragmatists this must indeed be a stinging irony.

Among reasons for this national waste of time and money called "American education", here is one: Too many Americans--not all, of course, for many Americans do extol cutural self-improvement and wish their children to have opportunities they hadn't--(including its teachers) simply may be too unlearned and uncultured themselves and want to stay so; and further, they may wish to require or at least to allow their schools to reflect their own lamentable prejudices and misplaced egalitarianism. So it may simply be a case of invincible ignorance and anti-class prejudice proud of itself. Case in point: convinced perhaps by "educationist" whingers and excuse-mongers, Americans in general and teachers in particular seem unjustifiably prejudiced against so-called "rote learning"--which may range from protests against requiring pupils to learn by heart multiplication tables, mathematical formulae, definitions of scientific terms and procedures, paradigms of grammar and standard usage in English and foreign languages (American children are generally deprived of ANY meaningful instruction in modern foreign, let alone classical languages--and nearly all admit that languages are best taught early-on in school), literary and rhetorical terms and examples, vocabulary-building, historical time-lines of personages and events, geographical lore and, in general, the wise, beautiful and moving thoughts and expressions of great inividuals often encapsulated in aphorisms, bons mots and the like. Resistance to so-called "rote learning" in the above examples exhibits one regrettable feature of an attitude inimical to instruction and correction through education and acculturation. Et repetitio mater memoriae.

It's best to define or describe "culture" here: it is not anything people like and do, especially in groups; rather, it is and ought to be (in its educational context) the development and refinement of intellectual, aesthetic and moral life. Thus, "[c]ulture", as Matthew Arnold has it, is "acquainting ourselves with the best that has been known and said in the world, and thus the history of the human spirit". "Acquainting", i.e., greeting and welcoming and taking into our own intellectual house--not "being (indifferently) aware of, as many now would have it.

But, give me a specific solution, not just a restatement of the problem, you say. I've one, to start: the reintroduction of classical languages in schools. A mon avis, Latin instruction to begin at eight and Greek at eleven for all children educable and not merely trainable. Festina lente.

Posted by: Jacques Albert at October 3, 2006 10:06 AM

close quotation after " . . .aware of"/

add words "of the indivudual" after "moral life" (omitted words)


Posted by: Jacques Albert at October 3, 2006 11:35 AM

Jacques--you might be interested in the thoughts of management consultant Michael Hammer on the value of a classical education for aspiring businesspeople.

Posted by: david foster at October 3, 2006 12:12 PM

Thanks for the tip, df--Hammer's views take the best of the new and old in an innovative combo. I also like his mode of expression--jaunty, but repectful of the worth of a classical education. The left, or just the hundreds of thousands of profscammers dragged up from the river in metastasing non-disciplines like education, social work, all "gender and ethnic" studies (social and political advocacy masquerading as academic disciplines), disability studies, peace studies (mostly frightened ex-draft-dodging passive-aggressives whose bellicosity is reserved for OUR govt, business and military) "leisure (takes at least a little Greek to sort out this etymology and its intellectual connotations), recreation "studies", physical education (clear oxymoron), most sosh and psych, business marketing (AKA hawking and sales), environmental studies (do the hard sciences and math without the whinging!), critical theory (i.e., "theorrhea"--poseurs and poseueses who think they're philosophers--pipsqueak Hegels, Nietzsches, and Sartres all (avec ou sans beret et pipe)--next thing will be departments and experts devoted to remedial, or "basic" ed--vous y etes, ca se voit!--"study studies". Plus three-quarters of the ed--admin drones feeding on taxpayer honey can go, along with the entire campus criminal underworld AKA college sports (no more semi-literate droolers [coaches] raking in far more per annum than our nation's president). For a start, then we rest a bit--for we don't want to rock the boat over-much (in deference to the wisdom of the ancient Seven Sages'--medev agan, or ne quid nimis or "nothing too much".
Thanks again, df. Dr JA

Posted by: Jacques Albert at October 3, 2006 2:07 PM

Read: "leisure . . .and recreation studies"

Also: Sorry for the Fraudian slip--"lit-crets" and lit-crits (esp, film studies profs--i.e., pseudo-intellectuals who read pictures only) are often interchangeables left out in my list of expendables hanging around colleges and universities with their snouts in the public and donors' troughs.

Posted by: Jacques Albert at October 3, 2006 2:14 PM

Read: "meden" (incomplete transliteration)

Posted by: Jacques Albert at October 3, 2006 2:21 PM

one last oxymoronic pseudo-discipline to show to the door--"pop culture" studies--and the earth-shoe experts (mums and pops) who teach them--pick up a Wheelock and a Chase and Phillips and get a real discipline. . . . Let's have our culture neat, i.e., without the "pop"!

Posted by: Jacques Albert at October 4, 2006 11:17 AM

Jacques: E. D. Hirsch and Allen Bloom called. They said they want their schtick back.

America doesn't spend more money per student than any other nation. This point was already destroyed by Berliner and Biddle in *The Manufactured Crisis*, along with the "nation at risk" "literacy crisis." Lawrence Cremin and Larry Cuban also deflated the claims that teachers today only pass on skills without content. See Richard Ognibene's "Social Foundations and School Reform Networks: The Case Against E. D. Hirsch" in /Educational Foundations/, Fall 1998, 5-25.

Evidence from psychology points out that learning Latin teaches you, well, Latin. it has no proven effect on "general cognitive competency" in other areas. Skills and background knowledge are domain-specific (which even Hirsch realized). Classics are fine for "cultural literacy," but why have students slog through disproven science, faulty logic, magical Platonism, primitive religious superstitutions, and Augustus' imperial apologists? Better to read analytic philosophy in English than Plato in Greek.

Posted by: Jonah Goldberg at October 4, 2006 8:09 PM

Jonah: Thanks for your blog on my (admittedly) provocative blogs. I'll read the B&B and C&C studies (I'm willing to revise my claim to, "the US WASTES more money per student . . .", but only if your B&B evidence holds), though with a sceptical eye--for remember that I simply don't believe psych--after Plato, Augustine, Pascal, Kierkegaard and Nietzsche, at least, has proved itself very instructive (and I said so above). And since you vaunt analytical philosophy vis a vis classics, remember Wittgenstein's dicta on this supposedly "young science" (the rouge of supposed "novelty" cannot hide its woefully decrepit face): "What we have in psychology is experimental methods and conceptual confusion". Likewise, I can't help but mention LW's rejection of the claim its age exempts it from serious challenges to its fundamental soundness as a science. And then there's the mighty lion (ena alla leonta--"one, yes, but a lion") Thomas Szasz to deal with on this score. The ed-psych schtick on Latin you refer to does not persuade in the least--I've read it over and over again when I was qualifying as a schoolteacher multi anni fa--it's the same anti-intellectual, egalitarian, instrumentalist, Deweyist rubbish that conveys deep and unjustified prejudice against so called "rote" learning.

Better to read some of your own camp's writings on the value of the classics as centre of the humanistic curriculum--I mean Tracy Lee Simmons, Victor Davis Hanson, John Heath, Bruce Thornton, Roger Scruton, etc., for a start. Then Bill Bennett, Chester Finn, Diane Ravitch et multi alii on the anti-humanist, hard-left emotive antinomianism of "instruction" in the humanities today that has replaced fact-based competencies. Better Bloom than doom. And know that I didn't spend hard (and embarrassing) years catching myself up on the classics I was denied in the States' public schools (cheer for vouchers here!)in order to justify later claims I'd done the right thing, but knew that sans Latin and at least some Greek, I could never claim to be an educated person. As I've said on other blogs, when I first really began to read the Greeks (while a young soldier in Vietnam), I thrilled to encounter the words of another soldier, Charles Peguy, who wrote from the trenches just before his death in 1914 that Homer was fresh that morning but there was nothing as old as yesterday's newspaper. Fads and fashions in the ed-psych, "more research is indicated" industry are Peguy's and my yesterday's NYTs. Better put on our classic finery like Machiavelli did to read his beloved Livy than wallow in the ephemera of yesterday's news and pop "culture". By the way, Jonah, do your views on the classics come from experience--for I've 'fessed to my experience with them--or lack of it? This is not a disingenuous question, for I shake your hand warmly for the many spot-on articles you've written in the past. And doubtless, in future. Cheers,


Posted by: Jacques Albert at October 5, 2006 7:44 AM

Evidence from psychology points out that learning Latin teaches you, well, Latin. it has no proven effect on "general cognitive competency" in other areas.

Err, learning a different language generally improves IQ test scores a bit. Learning a second one doesn't have the same effect. Does not need to be Latin, mind you, but learning Latin if you have no other second language does have a proven effect on general cognitive competency.

Just an FYI.

Posted by: Stephen M (Ethesis) at October 5, 2006 9:35 PM

Stephen M (Ethesis):

I regret to have lost (peccato!) a while ago a longer blog with a number of questions pertinent to your rather bald assertions above. I'll try to recap a few points here. First, what sources support your claims that the second language one learns, e.g., Dutch (which I've at present begun--obviously some German helps here with vocab, world order, declensions, syntax, etc.) or Estonian or Romansch as opposed to Greek and Latin (think of the values of cognate vocab and etymology to those sitting IQ tests)is an indifferent matter, and further, that no advantage accrues to those learning several languages? Is there a difference, too, reflected in IQ scores for a native speaker of Spanish successfully learning Russian (obviously from a different Indo-European language branch) and one learning Catalan or Portuguese? And do the credentials of those researching the issue tend to rely on linguistic competencies or merely on the "recognised experimental procedures" and "stats" in the field of ed-psych? And then, what constitutes "learning" a language? And how is competency or fluency measured? (A hopeless vulgarian of a bank manager once told me in a cafe that his three-year old daughter spoke "fluent Latin!"--whereupon, lng my throatin disgust, I quickly excused myself to take a non-existent emergency "call" from my girl-friend . . .) My experience as a state-certified teacher (including some grad school course work in ed-psych) and later college prof in the humanities (with a keen interest in what I regard as the quasi-pseudo-science of psych) has not given me overmuch confidence in the foreign language (and especially classical language) abilities of most ed-psych "researchers". And I've come to agree with Donald Lehman Clark (in Rhetoric in Greco-Roman Education) that Latin masters and mistresses make the best English teachers.

But aside from the relatively minor question of the relationship of language acquisition and raw IQ scores is the far more important one, a mon avis, of cultural competency. Greek and Latin remain, and will remain, crucial to understanding Western scholarly discourse for more than 2000 years--less so for the major Western vernaculars like English, Spanish, Portuguese, Italian, German, French, and
Russian--and much less so for the rest. Better for one's true cultural competency to wrestle with Dante's Divina commedia with one's trusty Petrocchi at one's side than to con an equivalent number of expressions in commercial traveler's technical or business jargon? Or better to understand the great influence of Greek on Latin (whence Horace's "Graeca capta ferum victorem cepit"--"and captive Greece took its rude conqueror") than to deliver a speech in a foreign language at the next international agricultural manuring exposition (though a Flaubert might be able to cultivate something rich in this field at once eminently useful and fertile for satire)? "Better mendacities/Than the classics in paraphrase", as Pound has it.
And what better way to live anew than to learn a new (or old) language and literature, which are the true portals to understanding another "culture", rather than to make vague and emotive appeals to so-called "diversity" and "multiculturalism", or con a splash of "toga Latin"? And what better way to understand the great authors of the past than to cultivate the languages and literatures they themselves knew and employed in their works (glosses and cribs help, but there's nothing more exhilarating than discovering literary allusions for oneself, and especially in foreign languages)? But in the West, first get classical languages, which, as I've contended in blogs de temps en temps, tend to support even the most up-to-date attitudes in evolutionary biology, for what's a backside for but(t) to facilitate the learning of Latin and Greek? So when I come to the next Pythagorean fork in the road where one way leads to another lang/lit competency and the other to the latest sheaf of ed-psych studies concluding that "further research [and grant money!] is indicated", I'm afraid . . . the choice isn't difficult.

Posted by: Jacques Albert at October 6, 2006 10:15 AM

correction above: "clutching my throat in disgust" (garbled transmission--someone must been clearing his or her throat while posting my comments)

Posted by: Jacques Albert at October 6, 2006 10:18 AM

addition: "crucial to understanding scholarly AND LITERARY discourse" (omission from lost original)

Posted by: Jacques Albert at October 6, 2006 10:31 AM