A statistical analysis of reviews of L.A. Finest: audience vs. critics

"If numbers are available, let's use the numbers. If all we have are opinions, let's go with mine." -- variously attributed to a number of bosses.

There's a new police procedural this season, L.A. Finest, and Rotten Tomatoes has done it again: critics and audience appear to be at loggerheads. Like with The Orville, Star Trek Discovery, and the last season of Doctor Who.

But "appear to be" is a dequantified statement. And Rotten Tomatoes has numbers; so, what can these numbers tell us?

Before they can tell us anything, we need to write our question: first in words, then as a math problem. Then we can solve the math problem and that solution gets translated into a "words" answer, but now a quantified "words" answer.

The question, which is suggested by the above numbers is:

Do the critics and the audience use similar or opposite criteria to rate this show?

One way to answer this question, which would have been feasible in the past when Rotten Tomatoes had user reviews, would be to do text analytics on the reviews themselves. But now the user reviews are gone so that's no longer possible.

Another way, a simpler and cleaner way, is to use the data above.

To simplify we'll assume that all ratings are either positive or negative, 0 or 1; there are some unobservable random factors that make some people like a show more or less, so these ratings are random variables. For a given person $i$, the probability that that person likes L.A. Finest is captured in some parameter $\theta_i$ (we don't observe that, of course), which is the probability of that person giving a positive rating.

So, our question above is whether the $\theta_i$ of the critics and the $\theta_i$ of the audience are the same or "opposed." And what is "opposed"? If $i$ and $j$ use opposite criteria, the probability that $i$ gives a 1 is the probability that $j$ gives a 0, so $\theta_i = 1-\theta_j$.

We don't have the individual parameters $\theta_i$ but we can simplify again by assuming that all variation within each group (critics or audience) is random, so we really only need two $\theta$.

We are comparing two situations, call them: hypothesis zero, $H_0$, meaning the critics and the audience use the same criteria, that is they have the same $\theta$, call it $\theta_0$; and hypothesis one, $H_1$, meaning the critics use criteria opposite to those of the audience, so if the critics $\theta$ is $\theta_1$, the audience $\theta$ is $(1-\theta_1)$.

Yes, I know, we don't have $\theta_0$ or $\theta_1$. We'll get there.

Our "words" question now becomes the following math problem: how much more likely is it that the data we observe is created by $H_1$ versus created by $H_0$, or in a formula: what is the likelihood ratio

$LR = \frac{\Pr(\mathrm{Data}| H_1)}{\Pr(\mathrm{Data}| H_0)} $?

Observation: This is different from the usual statistics test: the usual test is whether the two distributions are different; we are testing for a specific type of difference, opposition. So there are in fact three states of the world: same, opposite, and different but not opposite; we want to compare the likelihood of the first two. If same is much more likely than opposite, then we conclude 'same.' If opposite is much more likely than same, we conclude 'opposite.' If same and opposite have similar likelihoods (for some notion of 'similar' we'd have to investigate), then we conclude 'different but not opposite.'

Our data is four numbers: number of critics $N_C = 10$, number of positive reviews by critics $k_C = 1$, number of audience members $N_A = 40$, number of positive reviews by audience members $k_A = 30$.

But what about the $\theta_0$ and $\theta_1$?

This is where the lofty field of mathematics gives way to the down and dirty world of estimation. We estimate $\theta$ by maximum likelihood, and the maximum likelihood estimator for the probability of a positive outcome of a binary random variable (called a Bernoulli variable) is the sample mean.

Yep, all those words to say "use the share of 1s as the $\theta$."

Not so fast. True, for $H_0$, we use the share of ones

$\theta_0 = (k_C + k_A)/(N_C + N_A) = 31/50 = 0.62$;

but for $H_1$, we need to address the audience's $1-\theta_1$ by reverse coding the zeros and ones, in other words,

$\theta_1 = (k_C + (N_A - k_A))/(N_C + N_A) = 11/50 = 0.22$.

Yes, those two fractions are "estimation." Maximum likelihood estimation, at that.

Now that we are done with the dirty statistics, we come back to the shiny world of math, by using our estimates to solve the math problem. That requires a small bit of combinatorics and probability theory, all in a single sentence:

If each individual data point is an independent and identically distributed Bernoulli variable, the sum of these data points follows the binomial distribution.

Therefore the desired probabilities, which are joint probabilities of two binomial distributions, one for the critics, one for the audience, are

$\Pr(\mathrm{Data}| H_0) = c(N_C,k_C) (\theta_0)^{k_C} (1- \theta_0)^{N_C- k_C} \times c(N_A,k_A) (\theta_0)^{k_A} (1- \theta_0)^{N_A- k_A}$

and

$\Pr(\mathrm{Data}| H_1) = c(N_C,k_C) (\theta_1)^{k_C} (1- \theta_1)^{N_C- k_C} \times c(N_A,k_A) (1 -\theta_1)^{k_A} (\theta_1)^{N_A- k_A}$.

Replacing the symbols with the estimates and the data we get

$\Pr(\mathrm{Data}| H_0) = 3.222\times 10^{-5}$;
$\Pr(\mathrm{Data}| H_1) = 3.066\times 10^{-2}$.

We can now compute the likelihood ratio,

$LR = \frac{\Pr(\mathrm{Data}| H_1)}{\Pr(\mathrm{Data}| H_0)} = 915$,

and translate that into words to make the statement

It's 915 times more likely that critics are using criteria opposite to those of the audience than the same criteria.

Isn't that a lot more satisfying than saying they "appear to be at loggerheads"?

Numbers as props vs numbers as information

Once you learn to tell the difference, you'll know whom to trust.

A very long time ago, in 2018, Elon Musk announced that Tesla would be ramping up production to 6000 vehicles per week. An anchor for a business program played the video, then addressed their co-host with:

"That's like four full parking structures a week. Wow!"
Co-host makes assenting noises.

That statement is true for parking structures that have 1500 spots, which most in San Francisco (where the show is produced) don't. Typical numbers here are closer to 500 than 1500. But that's not the important part.

The important part is that the number was used as a prop, not information.

More precisely, the anchor first bought into the idea that 6000 is a large number for a car company's weekly production, then looked for a way to make that number look big to the show's audience; parking structures are big buildings and are related to cars, so that was a good way to create the perception of "bigness." [1]

In other words, the process of using a number as a prop is:

1. Make a decision based on something other than the number
2. Look for a number to support that decision
3. Choose context to present the number that molds perception in favor of the decision.

The alternative to using numbers as props is using them as information.

The metric '6000 per week' is just data. It becomes information when it answers a question. A few of these questions that come to mind, considering that this is a business program focussing on technology for a mostly finance and finance-adjacent audience would be:

a. How does this production level compare to that of the competitors that Musk repeatedly states he's going to put out of business?
b. How does this production level compare to that of Toyota when it was running the factory that is now Tesla's?
c. How does this production level compare to the demand for electric vehicles in general, possibly by geographical area and brand of vehicle?

Note that these questions extract information from the number 6000, by comparing it to other numbers that are of business interest. This illustrates a very important principle of data-processing for decision-making:

What is informative about data depends on what decision is to be made.

Choosing question a for illustration, and using Wikipedia data for 2016, because it's publicly available so anyone can check this computation without having to pay financial information service fees, here are the production rates for the top 15 car companies by number of vehicles produced:

Those numbers put Tesla's production in context; they suggest that Tesla, relative to the competitors that Musk repeatedly taunts as "dinosaurs" and "on their way out," is a niche player and not a serious business threat. [2]

Note the process for using numbers as information:

1. Determine what decisions are to be informed by the number
2. Find the context that is relevant for that decision
3. Compare number with the numbers from that context

Using numbers as information is important primarily for decision-makers. Realizing when others are using numbers as props, not information, is important for everyone. Especially regarding whether you can trust the numbers -- and the other person.

Just because someone uses numbers as props, that doesn't necessarily mean their intent is to deceive you. Our society, particularly our news and edutainment, are full of prop-use of numbers for non-nefarious reasons: ignorance, desire to connect abstract numbers to concrete objects, laziness.

But there are people whose intent is to deceive, and often you can tell who they are by calling them on their use of numbers as props. [3]

When faced with the above table, many Tesla fans on twitter, some of whom manage third-party money, either resorted to ad hominem ("how big is your short position?" is a common one, even used by Musk) or changing the subject ("these cars will save the planet").

This is how you identify someone who's not making a good-faith mistake of using numbers as props, but rather someone who deliberately avoids using the appropriate context for the numbers to use them as props: they never address the relevant comparison.

Because most people don't process numbers as they hear or read them, but are still influenced by the perceived authority of the number, this behavior (deliberately using numbers as props to deceive, that is) is usually effective as a persuasion tool. And people who deliberately use numbers as props know about that effectiveness and that's why they do it. Which brings us to an important insight about people we get from their use of numbers:

People who deliberately use numbers as props are not to be trusted.



-- -- -- --  FOOTNOTES -- -- -- --

[1] More likely the choice was made by a writer or a producer, not the anchor; but the anchor is the face of the show, so we'll keep referring to them.

[2] Or, if we want to apply strategic thinking, Tesla should build itself by market expansion starting from its niche, instead of a frontal assault on the much larger companies (its current strategy)

[3] For what it's worth, I don't think the anchor, or the TV channel, were trying to deceive their audience. They were just caught in Musk's Reality Distortion Field, which in 2018 was much stronger than Steve Jobs's ever was.


-- -- -- -- ADDENDUM -- -- -- --

Later that year, numbers-as-props sophistry continued unimpeded by any sense of shame on the part of Tesla fans:

Blogging again?

For the last few months I've been using Twitter more and more like a blogging-in-threads platform. That makes no sense, so I've decided that perhaps a repurposing of the old blog as something of an escape valve for the innumeracy, numberphobia, and acalculia I see might be in order.

Also, I get to use mathjax  for pretty math instead of post-it notes: \(e^{i \pi}+1=0\).

(In keeping with the new purpose, some purging of the past will occur.)

Thinking hiatus

As I consider some possible major changes to life, this blog is in hiatus (has been for a while).

That's a hiatus for thinking, not a hiatus in thinking.

Reality vs nonsensical products (part 688 of Aleph-null)

Via Thunderf00t, I found this Waterseer-wannabe, which is about as feasible as the original Waterseer, that is not at all.

Obviously it's very important that the product is 3D-printed, rather than CNC-machined or heat-molded. 3D-printers, like the Internet Of Things, are magical incantations that can get around the laws of Physics. Or so one would think, given how credulous people become at the sound of these incantations.

Alas, as is usual with engineering, ugly numbers murder beautiful illusions:

Since the battery voltage is 12V, a 12kW Peltier effect cooler will require a 1000A current, which is likely to make Li-ion battery a bit... well, just watch what happens:

Engineering rule: when an electronic device starts outgassing, that's generally not a good thing.

The power of "equations"

If a picture is worth a thousand words, an equation is worth a thousand pages of text.

This was inspired by a livestream about free trade based on criticism of "original texts." (Basically Ricardo and Schumpeter.) The quotes aren't a diss on the texts themselves, but rather a way to emphasize that this is a type of scholarly pursuit in itself, though not the type used in modern economics, STEM, or pragmatic professional fields like business analytics or medicine.

What's the problem with the argumentation from these original texts? Simply put, the texts are long and convoluted, with many unnecessary diversions and some logical problems in the presentation. The valid arguments in these texts can be condensed in about one page of stated assumptions and two results about specialization.

It's not just that math's an efficient way to communicate, math has precise meaning and an inference process. It brings discipline and clarity to the texts and the inference process isn't open to debate. (Checks and corrections, yes; debate, no.)

Unfortunately, without math, the speaker's argument was essentially a sequence of variations on "Schumpeter points out that this assumption of Ricardo doesn't hold true," without the extra step of determining whether those assumptions are important to the final result or not. (We'll come back to this problem.)

Word-thinking about quantitative fields is generally to be avoided.

That was the inspiration, and this post isn't about free trade or the particular mode of thought of that speaker, but rather about the power of mathematical modeling, which I'm calling "equations" in the title.

Here's a reasonably robust statement: when the price of a commodity goes up, people buy less of that commodity. (Sometimes this is put as "demand goes down," which is incorrect, it's the demand quantity that goes down. Changes in demand are movements of an entire function.)

So, quantity is a decreasing function of price (and first-time readers of economics textbooks get confused because the charts have quantity in the $x$ axis and price in the $y$ axis). This has been known for a long time; what's the problem with that formulation, simplified to "when price rises, quantity falls"?

The problem, of course, is that there are many different types of decreasing function. Here are a few, for example (click for bigger):

Functions 1 to 4 represent four common behaviors of decreasing functions: the linear function has similar changes leading to similar effects; the convex function has decreasing effect of similar change (like most natural decay processes); the concave function has increasing effect of similar change (like the accelerating effect of a bank run on bank reserves); and the s-shaped function shows up in many diffusion processes (and is a commonly used price response function in marketing).

Functions 5 to 8 are variations on the convex function, showing increasing curvature. (Function 2 would fit between 5 and 6.) They're here to make the point that even knowing the general shape isn't enough: one must know the parameters of that shape.

That figure does have 2000 data points, since each function has 250 points plotted. (When talking about math, some people use drawing tools to make their "functions," I prefer to plot them from the mathematical formula; it's a habit of mine, not lying to the audience.) To describe them in text would take a long time (unless the text is a description of mathematical formulation), while they can be written simply as formulas; for example, the convex functions are all exponentials:

$\qquad y = 100 \, \exp(-\kappa \, x) $

with different values of $\kappa$. They are the type of exponential decay found in many processes, for example, where $x$ is time and $y(x) = \alpha \, y(x-1)$ with $y(0)>0$ models a process of decay with discrete-time rate $0 < \alpha < 1$. In case it's not obvious, $\kappa = -\log_{e}(\alpha)$.*

So, what does this have to do with reasoning?

Here we go back to the problem with arguments like "Schumpeter showed that Ricardo's assumption X was wrong." When a model is written out in equations, we have a sequence of steps leading to the result, each step tagged with either a know result, rules of math inference (say "$a \times b = a \times c$ simplifies to $b = c$ unless $a = 0$"), or an assumption of the model. This allows a reader to quickly see where a failed assumption will lead to problems and determine whether the assumption can be replaced with something true (or, as is the case with many of the assumptions made by Ricardo, is unnecessary for the result).

The main power, however, is that mathematical notation forces the speaker to be precise, and inferences from mathematical models can be checked independently of subject matter expertise. A mathematician may not understand any of the economics involved, but will merrily check that a decay process of the kind $y(n)= \alpha \, y(n-1)$ can be described by an equation $y(n) = y(0) \, \exp(-\kappa \, n)$ and determine the relationship between $\kappa$ and $\alpha$.

From those precise models, one can make inferences that take into account details hidden by language. Consider the "price rises, quantity falls" text and compare it with the different decreasing functions in the figure above. The shape of the function, its slope and its curvature have different implications for how price changes affect a market, differences that are lost in the "price rises, quantity falls" formulation.

It bears repeating the first mentioned advantage: that hundreds of pages can be condensed in one page of equations. Once one's mind is used to processing equations, this is a very efficient way to learn new things. Stories about Port wineries in Portugal and textile factories in England may be entertaining, but they aren't necessary to understand specialization (which is what comparative advantage really is).

Math. It's a superpower mostly anyone can acquire. Sadly, most opt not to.


- - - - - Addendum - - - - -

No self-respecting economist would use the Ricardo comparative advantage argument for international trade now, particularly because it's so simple it can be understood by anyone. Most likely they'd use some variation of the magic factory example:

"Let's say a new technology that converts corn into cars is discovered and a factory is built in Iowa that can take ~ $\$20,000$ of corn and convert it into a car that costs $\$30,000$ to make in Michigan. Can we agree that this technology makes the US richer?

Now, move the factory to Long Beach, CA. Maybe there's a little more cost in moving the corn there, but we're still making the US richer, right?

Now, someone goes into the magic factory and discovers that it's really a depot: stores grain until it's sent to China on bulk carriers and receives cars made in China from RoRos during the night. The effect is the same as the magic factory, so it makes the US richer, right?"

There are many cons to this example, but it does make one issue clear: trade is in many respects just like a different technology.


- - - - - Footnote - - - - -

* It's obvious to me, because after decades of playing around with mathematical models, I grok most of these simple things. There are some people who mistake this well-developed and highly available knowledge (from practice) for ultra-high intelligence (rather than regular very high intelligence), a mistake I elaborate upon in this post. 😎

Deep understanding and problem solving

There's value in deep understanding.

Nope, I don't mean the difference between word thinkers and quantitative thinkers. Been there, done that. Nor the difference between different levels of expertise on technical matters; again, been there, done that.

No, we're talking the crème de la crème, experts that can adapt to changing situations or comprehend complexity across different fields, by being deep understanders.

Because any opportunity to mock those who purport to educate the masses by passing along material they don't understand, let us talk about Igon Values... ahem, eigenvalues and eigenvectors.

Taught in AP math classes or freshman linear algebra, the eigenvectors $\mathbf{x}_{i}$ and associated eigenvalues $\lambda_{i}$ of a square matrix $\mathbf{A}$ are defined as the solutions to $\mathbf{A} \, \mathbf{x}_{i} = \lambda_{i} \, \mathbf{x}_{i}$.

Undergrads learn that these represent something about the structure of the matrix, learn that the matrix can be diagonalized using them, how they appear in other places (principal components analysis and network centrality, for example).

But those who get to use these and other math concepts on a day-to-day basis, who get to really understand them, develop a deeper understanding of the meaning of the concepts. There's something important about how these objects relate to each other.

After a while, one realizes that there are structures and meta-structures that repeat across different problems, even across different fields. Someone said that after a lot of experience in one engineering (say, electrical), adapting to another (say, mechanical) revealed that while the nouns changed, the verbs were very similar.

This is what deep understanding affords: a quasi-intuitive grokking of a field, based on the regularities of knowledge across different fields.

For example: while many who have taken a linear algebra in college may vaguely recall what an eigenvalue is, those who understand the meaning of eigenvalues and eigenvectors for matrices will have a much easier time understanding the eigenfunctions of linear operators:

The structure [something that operates] [something operated upon] = [constant] [something operated upon] is common, and what it means is that the [something operated upon] is in some sense invariant with the [something that operates], other than the proportionality constant. That suggests that there's a hidden meaning or structure to the [something that operates] that can be elicited by studying the [something operated upon].

And this structure, mathematical as it might be, has a lot of applications outside of mathematics (and not just as a mathematical tool for formalizing technical problems). It's a basic principle of undestanding: what is invariant to a transformation tells us something deep about that transformation. (Again, invariant in "direction," so to speak, possibly a change of size or even sign.)

And this is itself a meta-principle: that the study of what changes and what's invariant in a particular set of problems gives some indications about latent structure to that set of problems. That latent structure may be a good point to start when trying to solve problems from this set.

Yep, really dumbing down this blog, pandering to the public...