On the review: I don’t think this post should be in the Alignment section of the review, without a significant rewrite / addition clarifying why exactly coherence arguments are useful or important for AI alignment.
Assuming that one accepts the arguments against coherence arguments being important for alignment (as I tentatively do), I don’t see why that means this shouldn’t be included in the Alignment section.
The motivation for this post was its relevance to alignment. People think about it in the context of alignment. If subsequent arguments indicate that it’s misguided, I don’t see why that means it shouldn’t be considered (from a historical perspective) to have been in the alignment stream of work (along with the arguments against it).
(Though, I suppose if there’s another category that seems like a more exact match, that seems like a fine reason to put it in that section rather than the Alignment section.)
Does that make sense? Is your concern that people will see this in the Alignment section, and not see the arguments against the connection, and continue to be misled?
I actually think it shouldn’t be in the alignment section, though for different reasons than Rohin. There’s lots of things which can be applied to AI, but are a lot more general, and I think it’s usually better to separate the “here’s the general idea” presentation from the “here’s how it applies to AI” presentation. That way, people working on other interesting things can come along and notice the idea and try to apply it in their own area rather than getting scared off by the label.
For instance, I think there’s probably gains to be had from applying coherence theorems to biological systems. I would love it if some rationalist biologist came along, read Yudkowsky’s post, and said “wait a minute, cells need to make efficient use of energy/limited molecules/etc, can I apply that?”. That sort of thing becomes less likely if this sort of post is hiding in “the alignment section”.
Zooming out further… today, alignment is the only technical research area with a lot of discussion on LW, and I think it would be a near-pareto improvement if more such fields were drawn in. Taking things which are alignment-relevant-but-not-just-alignment and lumping them all under the alignment heading makes that less likely.
It seems weird to include a post in the book if we believe that it is misguided, just because people historically believed it. If I were making this book, I would not include such posts; I’d want an “LW Review” to focus on things that are true and useful, rather than historically interesting.
That being said, I haven’t thought much about the goals of the book, and if we want to include posts for the sake of history, then sure, include the post. That was just not my impression about the goal.
Is your concern that people will see this in the Alignment section, and not see the arguments against the connection, and continue to be misled?
I would have this concern, yes, but I’m happy to defer (in the sense of “not pushing”, rather than the sense of “adopting their beliefs as my own”) to the opinions of the people who have thought way more than me about the purpose of this review and the book, and have caused it to happen. If they are interested in including historically important essays that we now think are misguided, I wouldn’t object. I predict that they would prefer not to include such essays but of course I could be wrong about that.
Assuming that one accepts the arguments against coherence arguments being important for alignment (as I tentatively do), I don’t see why that means this shouldn’t be included in the Alignment section.
The motivation for this post was its relevance to alignment. People think about it in the context of alignment. If subsequent arguments indicate that it’s misguided, I don’t see why that means it shouldn’t be considered (from a historical perspective) to have been in the alignment stream of work (along with the arguments against it).
(Though, I suppose if there’s another category that seems like a more exact match, that seems like a fine reason to put it in that section rather than the Alignment section.)
Does that make sense? Is your concern that people will see this in the Alignment section, and not see the arguments against the connection, and continue to be misled?
I actually think it shouldn’t be in the alignment section, though for different reasons than Rohin. There’s lots of things which can be applied to AI, but are a lot more general, and I think it’s usually better to separate the “here’s the general idea” presentation from the “here’s how it applies to AI” presentation. That way, people working on other interesting things can come along and notice the idea and try to apply it in their own area rather than getting scared off by the label.
For instance, I think there’s probably gains to be had from applying coherence theorems to biological systems. I would love it if some rationalist biologist came along, read Yudkowsky’s post, and said “wait a minute, cells need to make efficient use of energy/limited molecules/etc, can I apply that?”. That sort of thing becomes less likely if this sort of post is hiding in “the alignment section”.
Zooming out further… today, alignment is the only technical research area with a lot of discussion on LW, and I think it would be a near-pareto improvement if more such fields were drawn in. Taking things which are alignment-relevant-but-not-just-alignment and lumping them all under the alignment heading makes that less likely.
That makes a lot of sense to me. Good points!
It seems weird to include a post in the book if we believe that it is misguided, just because people historically believed it. If I were making this book, I would not include such posts; I’d want an “LW Review” to focus on things that are true and useful, rather than historically interesting.
That being said, I haven’t thought much about the goals of the book, and if we want to include posts for the sake of history, then sure, include the post. That was just not my impression about the goal.
I would have this concern, yes, but I’m happy to defer (in the sense of “not pushing”, rather than the sense of “adopting their beliefs as my own”) to the opinions of the people who have thought way more than me about the purpose of this review and the book, and have caused it to happen. If they are interested in including historically important essays that we now think are misguided, I wouldn’t object. I predict that they would prefer not to include such essays but of course I could be wrong about that.