Wednesday, December 10, 2008

 

Kevin Drum on the kids these days

THE KIDS THESE DAYS....PART 476....Here's the latest international report card on how American kids are doing in science and math. Short answer: not badly, really. According to the latest TIMSS report, eighth grade Asian kids outscore everyone, and American kids outscore nearly everyone who's not Asian (only Hungary, England, and Russia do slightly better). The story in science is about the same. TIMSS doesn't conduct their tests in every country, but among the countries that do worse than the U.S. are Australia, Sweden, Italy, Norway, and Israel.

Full results here.

I comment.

I'm delighted.

However, I do have to note that the TIMSS test is mostly a multiple choice. Students in the USA have practice with the format. I teach in Italy and I can assure you that Italian students just don't know how to deal with multiple choice questions. It is a specific skill and not really related to knowledge about or understanding of math and science.

Also, in the past, the USA had extraordinary inequality in TIMSS scores. Not surprising given decentralized budgets and curricula, but a huge national problem.

Stil great news.

As to the incomprehensible graphs, I can assure you that about the most intellectually difficult task I ever attempted was figuring out how the TIMSS people processed the data.

update: background on TIMSS and other international studies of student achievement here.

The word I hear about the kids these days is that they are just frighteningly knowledgeable, hard working, polite, self confident and well rounded (and must sleep 4 hours a night).

As to our generation (I am 48) I think we were totally like wasted by the aftershocks of the 50s. I personally enjoyed a good bit of experimental progressive education (I mean in school) which might braden the mind but is not good for TIMSS scores. Also marijuana. I once discussed SAT scores with a fellow high school student. I argued that the decline was meaningless (I forget my argument). He said "I think it is real and happened for 2 reasons marijuana and TV." Maybe with cable, parents can get kids to watch the less stupid channels (some parents not me).

update: Kevin Drum requested more details on Italian tests

There is no Italian equivalent of the SAT. Students are admitted to a degree program (say economics or philosophy) not to the University so entrance exams are field specific. More importantly, they didn't exist until realtively recently (there was open admission for students with a high school degree).

I have two daughters and they never took multiple choice tests. I also have examined University students using multiple choice tests (not my idea). They were totally hopeless about, for example, checking for "d) all of the above" and not marking a) because it was true. I mean it was grim.

The difference is more extreme than you can imagine or believe. The tradition in Italy is that most exams are oral (I am not kidding). Also students seem to have been taught to recite the 5 pages from the textbook which are most related to the question they are asked.

On US vs Italian high schools, obviously Italian high schools are more rigorous (I mean US high school is very exceptional as is the fact that most people in the US completed high school way back in the 20s). However, there was a comment by Italian students who were in the US on an exchange program that with multiple choice tests one has to think. Compared to learning by rote and reciting that is really true.

So what do I mean by "rigorous". It is very simple ... total hours of work required to pass (in school plus homework). Rigor is not reduced if the hours are totally wasted memorizing facts which will be forgotten rather than actively using facts to test theories (that way students learn how to reason and also remember the facts).

I have more information from an interview of an actual Italian student, my younger daughter. She is sure she has answered multiple choice questions on tests (I thought otherwise)and regularly does them on homework (although these are mostly true false questions). However, she also had only one written test per semester per subject !?!? the rest of evaluation was based on oral exams (called interrogazioni). The word "test" in Italian means "entrance exam". Such "tests" are by multiple choice, because no matter how much one detests multiple choice exams, if one has thousands of exams to grade one choses a multiple choice test.


Now rigorous also refers to how advanced the curriculum is. My 11 year old daughter is solving problems with 2 linear equations and two variables in 6th grade. Not her father's basic math education.

Now the problem Italian students have with multiple choice questions might very well be that you can't BS your way through them *and* it might be that some thought is required and parroting memorized text doesn't fill in the right oval. That is, I think that my students have been taught in the wrong wrong wrong way (for one thing I have to re-teach university students studying economics how to solve 2 linear equations with 2 variables). However, the TIMSS tests might underestimate their knowledge, because they include a lot of multiple choice questions (about 2 out of 3 questions IIRC).

update: Pulled back from comments

I wonder if there are not other factors at work here
1. drop out rates
2. number of subjects (this test was just on maths and science).
# posted by Blogger reason : 10:04 AM

OK first TIMSS aims to avoid sample selection due to dropping out. They test students who are young enough that they are required to be in school. Another testing effort -- PISA -- tests people age 16, just before they are allowed to drop out.

This raises another concern. My impression is that the real relatively low study effort in the US is in high school. In other countries (except maybe Canada) high school attendance was very far from universal until relatively recently. Thus the concept was academic education for the elite. Now in most countries attendance is roughly up to US levels (so that not getting a higher seconday degree is the exception not the rule) but the programs remain demanding.

In Italy, the fraction of current cohorts who complete upper secondary education is still around 50% (it's up around 80% in France IIRC). This is just not at all like graduating from high school. There is a very serious final exam with outside examiners -- teachers in an Italian high school equivalent are not the final judges of whether they have taught their students.

On fields, this is a problem. However it is very very hard to make an internationally comparable test in humanities -- OK lets not be pretentious -- reading comprehension. Basically translating the passage to be read, the multiple choice question and the proposed answers is impossible. It is not enough that the original and the translation have the same meaning -- it is necessary that it is equally difficult to understand this meaning.

Translating a reading comprehension test is roughly as difficult as translating poetry -- that is roughly impossible.

Now any international test on history or literature or whatever is just impossible -- different countries have different views on which literature is great and which history is important (also opinions differ on e.g. who won the war of 1812 and why it was fought). the M and S in TIMSS are based mostly on the fact that people agree on math and science.

posted by Robert permalink and comments2:07 AM

Comments:

I wonder if there are not other factors at work here
1. drop out rates
2. number of subjects (this test was just on maths and science).

# posted by  reason : 10:04 AM

 

Very enjoyable posting, and one to which I very much relate. I studied in Italy from 1st grade to college (Physics at Roma 1), and then in the US for my MS and PhD (Statistics and Operations Research at Stanford). I like to think and read about education because I believe it's the Great Equalizer of opportunities, and because I agree with Dante ("fatti non foste a viver come bruti/ ma per seguir virtute e conoscenza"), so education is its own reward.

My theses below, with short arguments:

1. I disagree that large national variance is a "huge national problem". Large variance in TIMMS and PISA can be largely explained by higher percentages of foreign-born students.
There is obviously a positive correlation between immigration status, income and educational attainment. But we should be careful in understanding the causal direction. There is a weak relationship (at the margin) between per-student education expenditures and educational attainment. As for flexibility in curriculum choice, I believe it's a positive feature of the US system, as it allows experimentation and helps meet the demand for education on a local basis.

2. Multiple choice tests are NOT uncommon, to the point that they cannot explain the difference.

2a. [This regards PISA only] I do not see why a cross-country language test regarding essential comprehension and inferences would be hard to craft. We are not discussing understanding Plato or Ezra Pound here.


3. There are two causes of the relatively low italian performance. First, italian education has always been biased against science. Croce and Gramsci are our unofficial saints, and both knew zilch about science. When I recall some pages from "Quaderni dal carcere" critical of technical education, my balls are still fuming. Going to the "liceo classico" still signals status in Italy. Conversely, the US seem to have a more pragmatic, hands-on approach to learning. They are also more open to scientific investigation. I have seen excellent research projects by high school students here. They would be unthinkable in Italy.

The second reason should be obvious to Waldmann. Teachers' quality has suffered from excessive centralization and employment-targeted governmental policies. The results are poor selection and screening.

# posted by  rara avis : 10:29 PM

 

Thanks for the thoughtful comments rara avis.

I have responses

1) While the high rate of immigration is a factor, I don't think it is the only or even the main explanation of the huge national variance of scores in the USA. Variance in the US is really really extreme and much higher than other high immigration countries. I haven't checked but one check with just native born students. I'm sure the USA would still show huge inequality.

The effects of the school on individual scores are very large in the US and everywhere. A very powerful explanatory variable is the average educational background of all parents of students in teh school (this helps explain scores totally aside from the own student's parents' educational attainment). Per pupil spending has a lot to do with do you count capital spending and are they building new schools. I think it is clear that class segregated schools imply high inequality in scores. I think this should be avoided.

I too like the idea of flexible curricula. However, there are costs.

2. I think this depends on what you mean by "not uncommon". How many did you take per year when you were in school ? My daughter took 2 written tests per subject in elementrary school (the rest of evaluation was based on interrogazioni sp?).

This is not remotely comparable to the freequency with which US students take multiple choice tests. Note 2 written tests. She's pretty sure some had multiple choice questions. I don't have current info on the US, but from my recollection and well guesses, I would say that US students take maybe 5 times as many written tests and that those tests are overwhelmingly multiple choice tests). Anyway, my students were totally unable to deal with multiple choice questions even if they could solve little math problems.

2a Look it it were easy for the students being tested they would all get the same score. To test reading comprehension, you have to have passages which challenge the students' reading comprehension.

The problem in translating such tests is that, while it is possible to translate the meanings, it is impossible to know if it is equally difficult to understand the meaning.

I do mean impossible. One can tell that passage A is as difficult as passage A and that question 1 about passage A is as difficult as question 1 about passage A. Aside from that, you can't know. The only way to prove that passage A in English is more difficult than passage B is to find that students have more trouble comprehending it. The only way to check a translation is to find perfectly bilingual subjects who are challenged by PISA level reading tests. Oh and you have to prove that they are *perfectly* bilingual which is impossible.

3a. I think (hope) that kids these days are choosing liceo scientifico over liceo classico -- I mean I think that for this generation things are changing.

3b) Thank you very much indeed. I have been selected to teach in Italy, but I'll try not to take it personally.

# posted by  Robert : 3:32 AM

 

A correction. TIMSS and PISA are separate studies. The analyses of the US PISA and TIMSS results are both published by the National Center for Education Statistics. The data I mentioned are from the 2006 PISA study.

# posted by  rara avis : 7:49 AM

 

Post a Comment