Saturday, June 18, 2011

The problem with grandfathered-in metrics

Consider this, admittedly preposterous, discussion:
I contend that my class is a better deal than that of Professor Moura, since my textbook costs $\$125$, while Prof Moura textbook costs $\$210$.

Professor Moura then retorts that my textbook has a paltry 320 pages, while his textbook has 850 pages, making it a much better deal at 25 cents per page against 40 cents per page for mine.

But, I say, my textbook is 8 by 11 inches, while Prof Moura textbook is 5 by 7, so correcting for page area, mine is a better deal at 0.45 cents per square inch against his at 0.71 cents per square inch.

Then Professor Moura remarks that there are many photos in my book, which take up space, and I remark that my book has more formulas, which condense a lot of text, and he remarks that his book has technical diagrams which condense a lot of formulas...
A person coming at the tail end of this discussion will get caught up in the details of correcting the metric for precision and how to adjust differences in typesetting, etc. A very technical person might even use a Markov chain prediction model to measure the entropy of the text and therefore approximate the information content of the books better.

Missing the main problem: the metric is inappropriate to begin with.

People taking my class need to get my textbook; the value of the class comes from the material covered, not the density of words per square inch of paper in the textbook.

The example used here is clearly ridiculous, but in many cases the metric is adopted and grandfathered in long before a given person comes in contact with it, and is used blindly. Many people then accept it and create all sorts of structures around it, so that anyone questioning the metric is seen as threatening their structures.

A few examples:
  • Body Mass Index (did you know that most sprinters, strength athletes, and gymnasts are obese? Packing muscle is a way to become obese in BMI terms);
  • Food calories (measured by burning -- yes, with fire -- food, as if the metabolic processes inside the body were that simple; a hydrazine and nitrogen tetroxide cocktail must be really nutritious); 
  • Student evaluations as measures of teaching effectiveness (these measure whether the students liked the teacher, which might be useful information for predicting alumni donations; to measure whether the students learn, i.e. if the teacher is effective, there's this old metric called an independently administered and blind-graded exam).
The problem is not the numbers or the processing done to the data. The problem is measuring the wrong thing. Which is much harder to solve if the metric has been grandfathered in.