I think this is an excellent post. I really liked the insight about the mechanisms (and mistakes) shared by the counting arguments behind AI doom and behind “deep learning surely won’t generalize.” Thank you for writing this; these kinds of loose claims have roamed freely for far too long.
EDIT: Actually this post is weaker than a draft I’d read. I still think it’s good, but missing some of the key points I liked the most. And I’m not on board with all of the philosophical claims about e.g. generalized objections to the principle of indifference (in part because I don’t understand them).
I think this is an excellent post. I really liked the insight about the mechanisms (and mistakes) shared by the counting arguments behind AI doom and behind “deep learning surely won’t generalize.” Thank you for writing this; these kinds of loose claims have roamed freely for far too long.
EDIT: Actually this post is weaker than a draft I’d read. I still think it’s good, but missing some of the key points I liked the most. And I’m not on board with all of the philosophical claims about e.g. generalized objections to the principle of indifference (in part because I don’t understand them).