The world changes; some people don't get that

I read something by a smart person that basically said: "90% of attempted X have failed, therefore X won't be viable."

The problem is that what that person really should have said was: "90% of the attempted X up to now have failed, therefore unless the underlying conditions have changed, X won't be viable."

Here's an example, adapted from a real situation in the past, with disguised (and simplified) numbers; smart person would have spoken up in year 20 and said: "people have been trying that for 20 years now, and trying in ever increasing numbers of attempts, and over 90% of the attempts have failed."

(click for bigger)

And while that's true, by the year 25, sixty-eight percent of all attempts made that year succeed! The secret is a change in underlying conditions (could be infrastructure, social acceptance, technology, economics, or other change).

The details of the model are the following:

1. Number of attempts in year $N$ is $A[N] = 2^{N-1}$.

2. Given a probability of success $p[N]$ in year $N$, the number of successes is the expected number (so we get rid of any stochasticity, for simplicity only):

$S[N] = p[N] \, A[N]$.

3. The probability of success in year $N$ is given by a deterministic difference equation,

$p[N+1] - p[N] = (a + b\, p[N]) (1 - p[N])$,

where the parameters $a$ (innovation rate) and $b$ (imitation rate) are between zero and one, with the specific values for the model above being $a=0.00001, b=0.6$, and $p[0] = 0$.

In other words, very slow innovation, but a large imitation rate (which leads to slow adoption at first then a cascading effect at some point, before saturation sets in). 

[The general form of this process is called a Bass model of evolution (used to analyze product diffusion in the past); I don't like it for data analysis or decision support (for a variety of technical reasons), but it's a convenient model for simulation when we take the parameters as given.]

Here's a real-world example of things changing, possibly one of the reasons of the current interest (institutional as well as individual) in cryptocurrencies:

But some people have this blindness: that just because something has been tried before and hasn't worked then, it must never work. And that something could be SpaceX landing its boosters, or the feasibility of Bitcoin, electric vehicles, small nuclear reactors, etc.

Fear is the mind killer. And not just for scions of House Atreides.

How to recommend a book one hasn't read

(Yes, we could just write a blind positive recommendation. That's apparently a common approach in traditional publishing circles. This post is about something else.)

Phil Tetlock recommends Tim Harford's book "The data detective" in the tweet above. Having read some Tim Harford books in the past, and knowing Phil Tetlock, I can second that recommendation using Bayes's rule, even though I haven't read the book (I'm not in the demographic for it).

How?

In three steps:

STEP 1: Prior probabilities. As far as I recall, the Tim Harford books I read were good in the two dimensions I care about for popularization of technical material: they didn't have any glaring errors (I would remember that) and they were well written without falling into the "everything ancillary but none of the technical detail" trap of so much popularization. So, the probability that this book is good, in the absence of information, (the prior probability) is high.

Note that in a simple world where a book is either good or bad, we have Pr(not good) = 1 – Pr(good); so we can plot the informativeness of that prior distribution using the odds ratio (where informativeness increases with difference to 1; note the log scale):

STEP 2: Conditional probabilities. To integrate the information that Phil Tetlock recommends this book, we need to know how likely he is to recommend any book when it's good and how likely he is to recommend any book when it's bad. Note that these are not complementary probabilities: there are some people who recommend all books, regardless of quality, so for those people these two probabilities would both be 1; observing a tweet from one of these people would be completely uninformative: the posterior probability would be the same as the prior (check that if you don't believe me*).

Having known Phil Tetlock for some years now, I'm fairly certain that his recommendation is informative, i.e. Pr(recommend | good) is much larger than Pr(recommend | not good).

STEP 3: Posterior probabilities Putting the prior and conditional probabilities together, we can use Bayes's rule (below) to determine that the probability that the book is good given the tweet is high.

As with all Bayesian models of beliefs (that is, not calibrated on measurements or actuarial statistics), these are subjective probabilities. Still, I stand by my seconding of Phil Tetlock's recommendation.

- - - -

* If you're the trusting type that believes without checking, I have a lovely oceanside villa in Kansas City, MO to sell. Trust but verify, as the Committee for State Security used to say.