I think proponents of the instrumental convergence thesis would expect a consequentialist chess program to exhibit instrumental convergence in the domain of chess. So if there were some (chess-related) subplan that was useful in lots of other (chess-related) plans, we would see the program execute that subplan a lot. The important difference would be that the chess program uses an ontology of chess while unsafe programs use an ontology of nature.
Gram Stone
Karma: 160
It seems like a good idea to collect self-reports about why LessWrongers didn’t invest in Bitcoin. For my own part, off the top of my head I would cite:
less visible endorsement of the benefits than e.g. cryonics;
vaguely sensing that cryptocurrency is controversial and might make my taxes confusing to file, etc.;
and reflexively writing off articles that give me investment advice because most investment advice is bad and I generally assume I lack some minimal amount of wealth needed to exploit the opportunity.
So something like We Agree: Get Froze,might have helped. I also could have actually evaluated the difficulty of filing taxes given that I’d purchased bitcoins, instead of flinching away and deciding it was too difficult on the spot. I could pay more attention to figures that are relatively small even for someone with little wealth.
If surveys are still being done, that seems like important data to collect on the next survey. I would want to know if most people knew but didn’t invest, or simply didn’t know at all, and so on.
Other things equal, choose the reversible alternative.
In particular, thank you for pointing out that in social experiments, phenomenal difficulty is not much Bayesian evidence for ultimate failure.
I’d be very interested in reading about EverQuest as an exemplar of Fun Theory, if you’re willing to share.