To add this, it may be that e.g. deep knowledge of the heuristics and biases literature helped Luke become a better thinker, but most people I know from SingInst became good thinkers either before or without reading much social psychology or decision science. Empirically it seems that microeconomics and Hofstadter are the biggest influences on good thinkers in this social sphere, but I wouldn’t put too much weight on the importance of either. Bayesian probability theory is also a big theme, but note that Eliezer’s devotion to it seems to have stemmed from a misconception about how fundamental it is, ’cuz at the time of his optimization enlightenment he doesn’t seem to have known about key problems in decision theory that Bayes isn’t obviously equipped for. In general it seems wise to be skeptical of claims that we know very much about why various people have had whatever success we think they’ve had; it’s very easy to unknowingly fall into cargo cult “rationality”.
To add this, it may be that e.g. deep knowledge of the heuristics and biases literature helped Luke become a better thinker, but most people I know from SingInst became good thinkers either before or without reading much social psychology or decision science. Empirically it seems that microeconomics and Hofstadter are the biggest influences on good thinkers in this social sphere, but I wouldn’t put too much weight on the importance of either. Bayesian probability theory is also a big theme, but note that Eliezer’s devotion to it seems to have stemmed from a misconception about how fundamental it is, ’cuz at the time of his optimization enlightenment he doesn’t seem to have known about key problems in decision theory that Bayes isn’t obviously equipped for. In general it seems wise to be skeptical of claims that we know very much about why various people have had whatever success we think they’ve had; it’s very easy to unknowingly fall into cargo cult “rationality”.