Saturday, December 19, 2009

My measurement mistake of this morning.

Oh, my! How could I make such a beginner's mistake in such a public way?

Earlier today I tweeted:
Revealed preference matches stated preference: my favorite jazz tune is Take 5; my iTunes library has 27 versions of Take 5. (=max in jazz)
This is all true: Paul Desmond's Take Five is my favorite jazz tune. I own multiple arrangements of it, 27 of which are on my iTunes library. There is no other jazz tune for which I have more than 27 versions.

But the number of different versions one has on iTunes is not a good measure of preference. What's wrong with it? (And along the way, with other measures of preference that we might consider in alternative?)

First, there's the books by the foot effect. Some people like to have many impressively-titled books in their shelves, but have no interest in what the books say: they own, but never read, the books. Similarly, I could own 27 versions of Take Five and never listen to them.

Second, there's a supply effect. There are many versions of Take Five to choose from, but few jazz arrangements of Marin Marais's Alcione. My favorite tune might indeed be a jazz arrangement of Alcione, of which I have only the one existing version, while the 27 Take Five versions count as 27.

Maybe I should use play counts as the measure of preference. Which bring us to the next problem with being careless about measures:

If I have more play counts of Take Five than of Rachmaninov's Third Piano Concerto (aka Rach 3), that doesn't mean I like Take Five better than Rach 3. (I don't.) The longest version of Take Five I have takes about 9 minutes; the shortest performance of Rach 3 I have takes about 33. If the play count for Rach 3 is at least 28% of Take Five's, I spend more time listening to Rach 3 than to Take Five. Given the different durations and the discreteness of the play count (if I listen to 32 minutes of Rach 3, its play count is unchanged), play counts are not good measures.

Luckily there's one good measure: choice under scarcity. But, you say, there's clearly no scarcity of variations on Take Five, and no shortage of them on my library either. True, but there's limited space on the iPods, and therefore the music that I carry in them reveals my preference. And indeed my main iPod carries six versions of Take Five.

Well, this was an educational mistake. It taught me not to go shooting off tweets with technical terms while my brain is still asleep. But at least I got a post to send to people who ask me about measurement, in lieu of addressing their specific concerns.

Eternal vigilance is the price of appropriate measurement.

Post Scriptum: Some people believe, incorrectly, that Dave Brubeck wrote Take Five. He didn't, but Paul Desmond, his long-time saxophonist, did write it for the Dave Brubeck quartet to play, so the misconception is understandable.