Inadequate Equilibria - Eliezer Yudkowsky

Equilibrium: the condition of a system in which competing influences are balanced

I can’t predict a 5% move in Microsoft stock in the next two months, and neither can you. If your uncle tells an anecdote about how he tripled his investment in last year and he attributes this to his skill rather than luck, we know immediately and out of hand that he is wrong. Warren Buffett at the peak of his form couldn’t reliably triple his money every year. If there is a strategy so simple that your uncle can understand it, which has apparently made him money—then we guess that there were just hidden risks built into the strategy, and that in another year or with less favorable events he would have lost half as much as he gained. Any other possibility would be the equivalent of a $20 bill staying on the floor of Grand Central Station for ten years while a horde of physics PhDs searched for it using naked eyes, microscopes, and machine learning.

You can’t go against the system’s default without going against the forces that underpin that default. A doctor who gives a baby a nutrition formula that isn’t FDA-approved will lose their job. A hospital that doesn’t fire that kind of doctor will be sued. A scientist that writes proposals for a big, expensive, definitive study won’t get a grant, and while they were busy writing those failed grant proposals, they’ll have lost their momentum toward tenure.