There was actually a real financial context, but I don’t recall it clearly enough to post about it in this blog. It was somehwere along the lines of taking an average yield based on a collection of yields but ignoring those that lie outside one standard deviation.

Anyway, my initial reaction to the problem was to use Chebyshev’s inequality, which I vaguely remember from school (stated here for your convenience and because in blogs looks so cool):

By definition, the inequality is valid for any real number k > 0. For k = 2, no more than 25% of the values are more than two standard deviations away from the mean. Stated another way, at least 75% of the values are within two standard deviations. However, for k = 1, the inequality states that at least 0% of the values are within one . What the hell? Does this statement really tell us anything? Or is it really allowing for exactly zero values within one ?

Since Chebyshev’s was of no help (or rather I’m too inept to understand it), let’s try to think about this from a more intuitive standpoint. Suppose you have the trivial case of one point. It lies within one . Then, suppose another point is added to the distribution to give two points. The standard deviation is just the distance of one point from the mean, so both points are good. Now, try to add a third point. To minimize the standard deviation, the new point will have to be placed at the mean of the first two values. Since the standard deviation is smaller than what it was before, only the new point satisfies the condition. Continuing with this experiment, no matter where any additional points are added, there will always be some that lie within a standard deviation.

It would seem like we have come to the conclusion that all distributions must have at least one point within one standard deviation. However, due to the serious degree of handwaving *whoosh* in the above argument, who knows? This is why sometimes I wish I had majored in math instead of electrical engineering in college. Intuition is great, but more often than not, it makes me want to relearn all the math that I think I understand. Bleh…

 He raises an interesting question:

The point is mathematically simple but does not register easily. I’ve enjoyed giving math students the following quiz (to be answered intuitively, on the spot). In a Gaussian world, the probability of exceeding one standard deviations is ~16%. What are the odds of exceeding it under a distribution of fatter tails (with same mean and variance)?

When I hear the term “fat tail” I imagine an elevated normal distribution with more mass residing further away from the mean. So, at first thought, the answer would be higher odds of landing outside one standard deviation. However, this is wrong and misses Taleb’s point completely. In order to keep the same variance (and same standard deviation), fatter tails also mean more probability mass must be concentrated at the mean. Thus, more points lie within one standard deviation, and the probability of exceeding it becomes smaller.

The point is that extreme events happen less frequently, but the ones that do happen tend to matter a lot more.

EDIT 2/14

That last sentence seems kind of off. Extreme events should happen more frequently in a fat-tailed setting. However, the danger is in the perceived distribution of your typical, run of the mill data. The hypothetical distribution will appear to have a smaller variance most of time. More points will fall within one standard deviation causing some re-calibration of risk models to fit the lower volatility, and that’s when Russia defaults on its bonds, or portfolio-insurance-minded algorithms all decide to sell at the same time, or…