Saturday, November 21, 2009

Statisticians and marketers: two professions separated by a common dataset.

Some days ago, in response to a friend's question, I wrote a tweet about nested choice models:
Someone else saw the tweet and sent me a question (edited):
Why choose order logit for quantity instead of a count model, say a Poisson regression?
This is a reasonable question: the Poisson "regression" (aka, the estimation of the Poisson lambda parameter as a linear combination of independent variables) is widely used to model physical count processes and makes writing the likelihood function easier. (In the old days, we wrote likelihood functions by hand! Kids these days, I tell you, with the clothes and the music and the hair, I mean, the Mathematica and the R libraries and the powerful home computers...)

The question illustrates an important difference between the statistical view of purchase data and the marketing analysis view of purchase data.

Marketing research, both of the commercial measurement type and of the experimental academic type, has produced a lot of consumer behavior knowledge. Within such knowledge we find that a count process is unlikely to be a good description of buying products in diverse quantities.

Instead, the number of cans of tuna purchased by customer Olivia is likely to fall into a handful of ordinal categories: zero, lower than normal inter-purchase interval consumption, normal inter-purchase interval consumption, higher than normal inter-purchase interval consumption, and very high. It's this categorical variable that should be regressed (order-logited, methinks!) on the variables thought to change Olivia's behavior. For Olivia the categories might be
{0}, {1,2,3}, {4,5}, {6,7,8,9}, {n | n>9}
These categories can be identified before the estimation of the nested choice model; coding them into a separate DV for the ologit is trivial. (The {0} category is never coded, as it is subsumed by the inter-purchase timing part of the nested model.)

Using the categories captures the behavior that marketers really want to understand: how does a marketing action M make Olivia change her quantity consumption in the Olivia scale, rather than the shared scale of natural numbers?

And this is one of the reasons why people analyzing marketing data need to know the basics of marketing model-building: because thinking like a marketer is different from building models based on physical processes.

Post-Bloggum: Yes, I know about the identification problems and how sensitive the estimation is to numeric issues. Those really important details are part of the -- unacknowledged and mostly misunderstood -- barriers to entry to the profession.

Wednesday, November 18, 2009

Online learning can teach us a lot.

Online learning is teaching us a lot. Mostly about reasoning fallacies: of those who like it and of those who don't.

Let us first dispose of what is clearly a strawman argument: no reasonable person believes that watching Stanford computer science lectures on YouTube is the same as being a Stanford CS student. The experience might be similar to watching those lectures in the classroom, especially in large classes with limited interaction, but lectures are a small part of the educational experience.

A rule of thumb for learning technical subjects: it's 1% lecture (if that); 9% studying on your own, which includes reading the textbook, working through the exercises therein, and researching background materials; and 90% solving the problem sets. Yes, studying makes a small contribution to learning compared to applying the material.

Good online course materials help because they select and organize topics for the students. By checking what they teach at Stanford CS, a student in Lagutrop (a fictional country) can bypass his country's terrible education system and figure out what to study by himself.

Textbooks may be expensive, but that's changing too: some authors are posting comprehensive notes and even their textbooks. Also, Lagutropian students may access certain libraries in other countries, which accidentally on purpose make their online textbooks freely accessible. And there's something called, I think, deluge? Barrage? Outpouring? Apparently you can find textbooks in there. Kids these days!

CS has a vibrant online community of practitioners and hackers willing to help you realize the errors of your "problem sets," which are in fact parts of open software development. So, for a student who wants to learn programming in Python there's a repository of broad and deep knowledge, guidance from universities, discussion forums and support groups, plenty of exercises to be done. All for free. (These things exist in varying degrees depending on the person's chosen field -- at least for now.)

And, by working hard and creating things, a Lagutropian student shows his ability to prospective employers, clients, and post-graduate institutions in a better country, hence bypassing the certification step of going to a good school. As long as the student has motivation and ability, the online learning environment presents many opportunities.

But herein lies the problem! Our hypothetical Lagutropian student is highly self-motivated, with a desire to learn and a love of the field. This does not describe the totality of college students. (On an related statistical note, Mickey D's has served more than 50 hamburgers.)

The Dean of Old Mizzou's journalism school noticed that students who downloaded (and presumably listened to) podcasts of lectures retained almost twice as much as students in the same classes who did not download the lectures. As a result, he decreed that henceforth all journalism students at Old Mizzou would be required to get an iPod, iPhone, or similar device for school use.

Can you say "ignoring the selection effect"?

Students who download lectures are different from those who don't: they choose to listen to the lectures on their iPod. Choose. A verb that indicates motivation to do something. No technology can make up for unmotivated students. (Motivating students is part of education, and academics disagree over how said motivation should arise. N.B.: "education" is not just educators.)

Certainly a few students who didn't download lectures wanted to but didn't own iPods; those will benefit from the policy. (Making an iPod required means that cash-strapped students may use financial aid monies to buy it.) The others chose not to download the lectures; requiring they have an iPod (which most own anyway) is unlikely to change their lecture retention.

This iPod case scales to most new technology initiatives in education: administrators see some people using a technology to enhance learning, attribute that enhanced learning to the technology, and make policies to generalize its use. All the while failing to consider that the learning enhancement resulted from the interaction between the technology and the self-selected people.

This is not to say that there aren't significant gains to be made with judicious use of information technologies in education. But in the end learning doesn't happen on the iPod, on YouTube, on Twitter, on Internet forums, or even in the classroom.

Learning happens inside the learner's head; technology may add opportunities, but, by itself, doesn't change abilities or motivations.

Tuesday, November 17, 2009

Professional amateurs

It's funny when professionals in one field make amateur mistakes in another.

When economists without any training in software design or large-scale programming start writing large computer programs, for example elaborate econometrics or complicated simulations, they tend to make what programmers consider rookie mistakes. Not programming errors; just missing out on several decades of wisdom on how to set up large programming endeavors: creating reusable code, working in modules, sharing data structures across problems, appropriate documentation -- the basics, ignored.

Of course, when the roles are reversed, that is when engineers and scientists start butting into economics and business problems, a similar situation arises. A good physicist I know makes a complete fool of himself every time he tries to write about economics, making basic mistakes that students of Econ-101 are taught to avoid.

The main difference seems to be that, while many economists and business modelers will appreciate the advice of programmers on how to handle large scale projects better, most non-economists and non-business researchers are unwilling to consider the possibility that there's actual knowledge behind the pronouncements of economists (and some business researchers).

Which is why I find this tweet so funny:
It's funny because it's true: most of the time technocratic pronouncements by technologists and scientists are either examples of the ceteris paribus fallacy (also known as the one-step-lookahead-problem) or would only work within a thorough command-and-control economy.

The ceteris paribus fallacy is the assumption that when we change something in a complex system, the only effects are local. (Ceteris paribus is Latin for "all the rest unchanged.") A common example is taxes: suppose that a group of people make $1M each and their tax rate is 30%. A one-step thinker might believe that increasing that tax rate to 90% would net $600k per person. That assumes that nothing else changes (other than the tax rate). In reality, it's likely that the increase in the tax rate would give the people in the group an incentive to shift time from paid production to leisure, which would reduce the pool of money to be taxed.

The need for a command-and-control economy to implement many of the quick-fix solutions of technologists and scientists comes from the law of unintended consequences. Essentially an elaboration on the ceteris paribus fallacy, the law says that the creativity of 15 year old boys looking for pictures of naked women online cannot be matched by the designers of adult filters: for any attempt at filtering done purely in the internet domain (i.e. without using physical force in the real world or its threat, aka without the police and court system), there'll be work-arounds popping up almost immediately.

Consider the case of subsidies for mixing biodiesel-like fuels with oil in industrial furnaces. Designed to lower the consumption of oil, it led to the opposite outcome when paper companies started adding oil to their hitherto wood-chip-byproducts only furnaces in order to get the subsidy. Pundits from the right and the left jumped on International Paper and others and screeched for legislative punishment; but the companies were just following the law -- a law which did not consider all its consequences.

Because people game rule systems to fit their own purposes (the purposes of the people living under the rule system, not the purposes of the people who make the rules), mechanism design in the real world is very difficult, prone to error, and almost never works as intended. Therefore, in most cases the only way a one-step based solution can work is by mandating the outcome: by using force to impose the outcome rather than by changing the incentives so that the outcome is desirable to the people. *

So, it is funny to see some people we know to be smart and knowledgeable about their field make rookie mistakes when talking about economics and business; but we should keep in mind that many others take the mantle of "science" or "technology" to assert power over us in matters for which they have no real authority or competence.

That's why that tweet is both funny and sad. Because it's true.

* Sunstein and Thaler's book Nudge suggests using psychology to solve the problems of mechanism design. My main objection to Nudge is that I don't trust that those who would change our behavior would give up if the nudges didn't work. In the words of Andy Stern of SEIU: "first we try the power of persuasion, then, if that doesn't work, we use the persuasion of power."

I fear that the Nudge argument would be used to sell the outcome to people ("sure we want people to eat veggies, but we are just making it a little more work to get the chocolate mousse, don't worry") and, once the outcome was sold, the velvet glove would come off of the iron fist ("tax on chocolate," "ban chocolate," and eventually the "war on chocolate dealers").

David Friedman wrote a much better critique of Nudge and its connection with slippery slopes here.

Tuesday, November 3, 2009

More online [work-related] books.

It seems I cannot stop work-related topics from seeping into this personal blog... Here are two more work books I like whose authors have generously posted the full text online:
Eric von Hippel's Democratizing Innovation.

Yochai Benkler's The Wealth of Networks.
And Cory "every author should put their books online for free because I make a good living even though my books are available for free online, but never mention that said good living is from giving talks, not selling books" Doctorow put his new novel Makers online as well. Haven't read it yet, but his earlier works were ok and this one seems to have engineering as the hero, so I'll give it a read when I have free time, around 2020 or so.