A question for the folks who voted this up: on a scale from “enjoyed reading this even though didn’t feel like I really learned anything” to “fantastic, now I understand everything”, how useful did this post feel to you?
Personally I felt this had several very important insights that only clicked properly together while I was writing it, such as the way how it’s almost impossible to even imagine certain kind of decision-making if we literally had no concept of personal identity, as well as the way that anticipated experience is treated separately from more abstract modeling in our brains. But judging from the relatively low score of the post and the fact that there’s very little discussion of those insights in the comments, it looks like most folks didn’t come off as feeling that they were important? (Or maybe didn’t agree with them, but in that case I would’ve expected more criticism.)
It also slightly brought into focus for me the distinction between “theoretical decision processes I can fantasize about implementing” and “decision processes I can implement in practice by making minor tweaks to my brain’s software”. The first set can include self-less models such as paperclip maximization or optimizing those branches where I win the lottery and ignoring the rest. It’s possible that in the second set a notion of self just keeps bubbling up whatever you do.
One and a half insights is pretty good going, especially on a tough topic like this one. Because of inferential distance, what feels like 10 insights to you will feel like 1 insight to me—it’s like you’re supplying some of the missing pieces to your own jigsaw puzzle, but in my puzzle the pieces are a different shape.
A question for the folks who voted this up: on a scale from “enjoyed reading this even though didn’t feel like I really learned anything” to “fantastic, now I understand everything”, how useful did this post feel to you?
Assigning the former 0, the latter 10, I felt somewhere around 4. While all the points and arguments felt reasonable enough, I’m only somewhat persuaded that they’re actually correct (so I picked up a bunch of new beliefs at like 40% confidence levels). The main shortcoming of this post in my view was that it felt like it lacked direction (consistent with your observation that you figured out some of the important insights while writing it) - the list of clues did not take me by the hand and lead me along a straight and narrow path to the conclusion. Instead, they meandered around, and then Clue 4 seemingly became the primary seed for the “summing up” section, despite not being foreshadowed very much before.
These are mostly writing structure complaints, but I think the main reason the post isn’t higher scoring/more discussed is the writing structure, so that seems appropriate.
Speaking about the substance, I’m not persuaded that the model of reinforcement based learner with abstract model stuff is accurate. I find it hard to explain why exactly (which is part of the reason I haven’t commented to say as much), but if I had to pick a reason, it would be that I don’t think the messy evolved human reasoning can be meaningfully broken down into such categories. I would be more persuaded if the explanation was something like “but then it turned out that reinforcement learning was pretty good, but could be improved by imagining what reinforcements might come later and improving on those, but doing so well required imagining yourself in the future, which required understanding your current behaviour and identity”. Which now that I read it is not so different from the two models thing, but is framed in a just-so story that’s more appealing to me. (The approach I would personally use to dissolve personal identity is to try to figure out what exactly it is and what it does. What processes are improved by its existence and which ones could be carried on without it. I recall thinking at one point that it’s probably there to help with thinking about thinking, but I haven’t though it through at any length, so I’m very far from confident in that.)
TL;DR: if you rewrote it with better structure, it would score higher and may persuade me (and probably others) better, even though maybe I should be persuaded already and am being silly.
Speaking about the substance, I’m not persuaded that the model of reinforcement based learner with abstract model stuff is accurate. I find it hard to explain why exactly (which is part of the reason I haven’t commented to say as much), but if I had to pick a reason, it would be that I don’t think the messy evolved human reasoning can be meaningfully broken down into such categories.
Oh, I don’t think that the underlying implementation would actually be anywhere near as clear-cut as the post described: I just gave a simplified version for the sake of clarity. The actual architecture is going to be a lot messier and the systems more overlapping.
A question for the folks who voted this up: on a scale from “enjoyed reading this even though didn’t feel like I really learned anything” to “fantastic, now I understand everything”, how useful did this post feel to you?
Personally I felt this had several very important insights that only clicked properly together while I was writing it, such as the way how it’s almost impossible to even imagine certain kind of decision-making if we literally had no concept of personal identity, as well as the way that anticipated experience is treated separately from more abstract modeling in our brains. But judging from the relatively low score of the post and the fact that there’s very little discussion of those insights in the comments, it looks like most folks didn’t come off as feeling that they were important? (Or maybe didn’t agree with them, but in that case I would’ve expected more criticism.)
I’m unsatisfied with it as a finished product, but I like it as a start, and it got me thinking along interesting directions.
I felt like I gained one insight, which I attempted to summarize in my own words in this comment.
It also slightly brought into focus for me the distinction between “theoretical decision processes I can fantasize about implementing” and “decision processes I can implement in practice by making minor tweaks to my brain’s software”. The first set can include self-less models such as paperclip maximization or optimizing those branches where I win the lottery and ignoring the rest. It’s possible that in the second set a notion of self just keeps bubbling up whatever you do.
One and a half insights is pretty good going, especially on a tough topic like this one. Because of inferential distance, what feels like 10 insights to you will feel like 1 insight to me—it’s like you’re supplying some of the missing pieces to your own jigsaw puzzle, but in my puzzle the pieces are a different shape.
So yeah, keep hacking away at the edges!
Assigning the former 0, the latter 10, I felt somewhere around 4. While all the points and arguments felt reasonable enough, I’m only somewhat persuaded that they’re actually correct (so I picked up a bunch of new beliefs at like 40% confidence levels). The main shortcoming of this post in my view was that it felt like it lacked direction (consistent with your observation that you figured out some of the important insights while writing it) - the list of clues did not take me by the hand and lead me along a straight and narrow path to the conclusion. Instead, they meandered around, and then Clue 4 seemingly became the primary seed for the “summing up” section, despite not being foreshadowed very much before.
These are mostly writing structure complaints, but I think the main reason the post isn’t higher scoring/more discussed is the writing structure, so that seems appropriate.
Speaking about the substance, I’m not persuaded that the model of reinforcement based learner with abstract model stuff is accurate. I find it hard to explain why exactly (which is part of the reason I haven’t commented to say as much), but if I had to pick a reason, it would be that I don’t think the messy evolved human reasoning can be meaningfully broken down into such categories. I would be more persuaded if the explanation was something like “but then it turned out that reinforcement learning was pretty good, but could be improved by imagining what reinforcements might come later and improving on those, but doing so well required imagining yourself in the future, which required understanding your current behaviour and identity”. Which now that I read it is not so different from the two models thing, but is framed in a just-so story that’s more appealing to me. (The approach I would personally use to dissolve personal identity is to try to figure out what exactly it is and what it does. What processes are improved by its existence and which ones could be carried on without it. I recall thinking at one point that it’s probably there to help with thinking about thinking, but I haven’t though it through at any length, so I’m very far from confident in that.)
TL;DR: if you rewrote it with better structure, it would score higher and may persuade me (and probably others) better, even though maybe I should be persuaded already and am being silly.
Thanks for your feedback.
Oh, I don’t think that the underlying implementation would actually be anywhere near as clear-cut as the post described: I just gave a simplified version for the sake of clarity. The actual architecture is going to be a lot messier and the systems more overlapping.