A question for the folks who voted this up: on a scale from “enjoyed reading this even though didn’t feel like I really learned anything” to “fantastic, now I understand everything”, how useful did this post feel to you?
Assigning the former 0, the latter 10, I felt somewhere around 4. While all the points and arguments felt reasonable enough, I’m only somewhat persuaded that they’re actually correct (so I picked up a bunch of new beliefs at like 40% confidence levels). The main shortcoming of this post in my view was that it felt like it lacked direction (consistent with your observation that you figured out some of the important insights while writing it) - the list of clues did not take me by the hand and lead me along a straight and narrow path to the conclusion. Instead, they meandered around, and then Clue 4 seemingly became the primary seed for the “summing up” section, despite not being foreshadowed very much before.
These are mostly writing structure complaints, but I think the main reason the post isn’t higher scoring/more discussed is the writing structure, so that seems appropriate.
Speaking about the substance, I’m not persuaded that the model of reinforcement based learner with abstract model stuff is accurate. I find it hard to explain why exactly (which is part of the reason I haven’t commented to say as much), but if I had to pick a reason, it would be that I don’t think the messy evolved human reasoning can be meaningfully broken down into such categories. I would be more persuaded if the explanation was something like “but then it turned out that reinforcement learning was pretty good, but could be improved by imagining what reinforcements might come later and improving on those, but doing so well required imagining yourself in the future, which required understanding your current behaviour and identity”. Which now that I read it is not so different from the two models thing, but is framed in a just-so story that’s more appealing to me. (The approach I would personally use to dissolve personal identity is to try to figure out what exactly it is and what it does. What processes are improved by its existence and which ones could be carried on without it. I recall thinking at one point that it’s probably there to help with thinking about thinking, but I haven’t though it through at any length, so I’m very far from confident in that.)
TL;DR: if you rewrote it with better structure, it would score higher and may persuade me (and probably others) better, even though maybe I should be persuaded already and am being silly.
Speaking about the substance, I’m not persuaded that the model of reinforcement based learner with abstract model stuff is accurate. I find it hard to explain why exactly (which is part of the reason I haven’t commented to say as much), but if I had to pick a reason, it would be that I don’t think the messy evolved human reasoning can be meaningfully broken down into such categories.
Oh, I don’t think that the underlying implementation would actually be anywhere near as clear-cut as the post described: I just gave a simplified version for the sake of clarity. The actual architecture is going to be a lot messier and the systems more overlapping.
Assigning the former 0, the latter 10, I felt somewhere around 4. While all the points and arguments felt reasonable enough, I’m only somewhat persuaded that they’re actually correct (so I picked up a bunch of new beliefs at like 40% confidence levels). The main shortcoming of this post in my view was that it felt like it lacked direction (consistent with your observation that you figured out some of the important insights while writing it) - the list of clues did not take me by the hand and lead me along a straight and narrow path to the conclusion. Instead, they meandered around, and then Clue 4 seemingly became the primary seed for the “summing up” section, despite not being foreshadowed very much before.
These are mostly writing structure complaints, but I think the main reason the post isn’t higher scoring/more discussed is the writing structure, so that seems appropriate.
Speaking about the substance, I’m not persuaded that the model of reinforcement based learner with abstract model stuff is accurate. I find it hard to explain why exactly (which is part of the reason I haven’t commented to say as much), but if I had to pick a reason, it would be that I don’t think the messy evolved human reasoning can be meaningfully broken down into such categories. I would be more persuaded if the explanation was something like “but then it turned out that reinforcement learning was pretty good, but could be improved by imagining what reinforcements might come later and improving on those, but doing so well required imagining yourself in the future, which required understanding your current behaviour and identity”. Which now that I read it is not so different from the two models thing, but is framed in a just-so story that’s more appealing to me. (The approach I would personally use to dissolve personal identity is to try to figure out what exactly it is and what it does. What processes are improved by its existence and which ones could be carried on without it. I recall thinking at one point that it’s probably there to help with thinking about thinking, but I haven’t though it through at any length, so I’m very far from confident in that.)
TL;DR: if you rewrote it with better structure, it would score higher and may persuade me (and probably others) better, even though maybe I should be persuaded already and am being silly.
Thanks for your feedback.
Oh, I don’t think that the underlying implementation would actually be anywhere near as clear-cut as the post described: I just gave a simplified version for the sake of clarity. The actual architecture is going to be a lot messier and the systems more overlapping.