I really like this post and found it very interesting, particularly because I’m generally interested in the relationship between the rationality side of the AI Alignment community and academia, and I wanted to register some thoughts. Sorry for the long comment on an old post and I hope this doesn’t come across as pernickety. If anything I sort of feel like TurnTrout is being hard on himself.
I think the tl;dr for my comment is sort of that to me the social dynamics “mistakes” don’t really seem like mistakes—or at least not ones that were actually made by the author.
Broadly speaking, these “mistakes” seem to me like mostly normal ways of learning and doing a PhD that happen for mostly good reasons and my reaction to the fact that these “mistakes” were “figured out” towards the end of the PhD is that this is a predictable part of the transition from being primarily a student to primarily an independent researcher (the fast-tracking of which would be more difficult than a lot of rationalists would like to believe).
I also worry that emphasizing these things as “mistakes” might actually lead people to infer that they should ‘do the opposite’ from the start, which to me would sound like weird/bad advice: e.g Don’t try to catch up with people who are more knowledgeable than you; don’t try to seem smart and defensible; don’t defer, you can do just as good by thinking everything through for yourself.
I broadly agree that
rationality is not about the bag of facts you know.
but AI alignment/safety/x-risk isn’t synonymous with rationality (Or is it? I realise TurnTrout does not directly claim that it is, which is why I’m maybe more cautioning against a misreading than disagreeing with him head on, but maybe he or others think there is a much closer relationship between rationality and alignment work than I do?).
Is there not, by this point, something at least a little bit like “a bag of facts” that one should know in AI Alignment? People have been thinking about AI alignment for at least a little while now. And so like, what have they achieved? Do we or do we not actually have someknowledge about the alignment problem? It seems to me that it would be weird if we didn’t have any knowledge—like if there was basically nothing that we should count as established and useful enough to be codified and recorded as part of the foundations of the subject. It’s worth wondering whether this has perhaps changed significantly in the last 5-10 years though, i.e. during TurnTrout’s PhD. That is, perhaps—during that time—the subject has grown a lot and at least some things have been sufficiently ‘deconfused’ to have become more established concepts etc. But generally, if there are now indeed such things, then these are probably things that people entering the field should learn about. And it would seem likely that a lot of the more established ‘big names’/productive people actually know a lot of these things and that “catching up with them” is a pretty good instrumental/proxy way to get relevant knowledge that will help you do alignment work. (I almost want to say: I know it’s not fashionable in rationality to think this, but wanting to impress the teacher really does work pretty well in practice when starting out!)
Focussing on seeming smart and defensible probably can ultimately lead to a bad mistake. But when framed more as “It’s important to come across as credible” or “It’s not enough to be smart or even right; you actually do need to think about how others view you and interact with you”, it’s not at all clear that it’s a bad thing; and certainly it more clearly touches on a regular topic of discussion in EA/rationality about how much to focus on how one is seen or how ‘we’ are viewed by outsiders. Fwiw I don’t see any real “mistake” being actually described in this part of the post. In my opinion, when starting out, probably it is kinda important to build up your credibility more carefully. Then when Quintin came to TurnTrout, he writes that it took “a few days” to realize that Quintin’s ideas could be important and worth pursuing. Maybe the expectation in hindsight would be that he should have had the ‘few days’ old reaction immediately?? But my gut reaction is that that would be way too critical of oneself and actually my thought is more like ‘woah he realised that after thinking about it for only a few days; that’s great’. Can the whole episode not be read as a straightforward win: “Early on, it is important to build your own credibility by being careful about your arguments and being able to back up claims that you make in formal, public ways. Then as you gain respect for the right reasons, you can choose when and where to ‘spend’ your credibility… here’s a great example of that...”
And then re: deference, certainly it was true for me that when I was starting out in my PhD, if I got confused reading a paper or listening to talk, I was likely to be the one who was wrong. Later on or after my PhD, then, yeah, when I got confused by someone else’s presentation, I was less likely to be wrong and it was more likely I was spotting an error in someone else’s thinking. To me this seems like a completely normal product of the education and sort of the correct thing to be happening. i.e. Maybe the correct thing to do is to defer more when you have less experience and to gradually defer less as you gain knowledge and experience? I’m thinking that under the simple model that when one is confused about something, either you’re misunderstanding or the other person is wrong, one starts out in the regime where your confusion is much more often better explained by the fact you have misunderstood and you end up in the regime where you actually just have way more experience thinking about these things and so are now more reliably spotting other people’s errors. The rational response to the feeling of confusion changes because once fully accounted for the fact you just know way more stuff and are a way more experienced thinker about alignment. (One also naturally gains a huge boost to confidence as it becomes clear you will get your PhD and have good postdoc prospects etc… so it becomes easier to question ‘authority’ for that reason too, but it’s not a fake confidence boost; this is mostly a good/useful effect because you really do now have experience of doing research yourself, so you actually are more likely to be better at spotting these things).
I really like this post and found it very interesting, particularly because I’m generally interested in the relationship between the rationality side of the AI Alignment community and academia, and I wanted to register some thoughts. Sorry for the long comment on an old post and I hope this doesn’t come across as pernickety. If anything I sort of feel like TurnTrout is being hard on himself.
I think the tl;dr for my comment is sort of that to me the social dynamics “mistakes” don’t really seem like mistakes—or at least not ones that were actually made by the author.
Broadly speaking, these “mistakes” seem to me like mostly normal ways of learning and doing a PhD that happen for mostly good reasons and my reaction to the fact that these “mistakes” were “figured out” towards the end of the PhD is that this is a predictable part of the transition from being primarily a student to primarily an independent researcher (the fast-tracking of which would be more difficult than a lot of rationalists would like to believe).
I also worry that emphasizing these things as “mistakes” might actually lead people to infer that they should ‘do the opposite’ from the start, which to me would sound like weird/bad advice: e.g Don’t try to catch up with people who are more knowledgeable than you; don’t try to seem smart and defensible; don’t defer, you can do just as good by thinking everything through for yourself.
I broadly agree that
but AI alignment/safety/x-risk isn’t synonymous with rationality (Or is it? I realise TurnTrout does not directly claim that it is, which is why I’m maybe more cautioning against a misreading than disagreeing with him head on, but maybe he or others think there is a much closer relationship between rationality and alignment work than I do?).
Is there not, by this point, something at least a little bit like “a bag of facts” that one should know in AI Alignment? People have been thinking about AI alignment for at least a little while now. And so like, what have they achieved? Do we or do we not actually have some knowledge about the alignment problem? It seems to me that it would be weird if we didn’t have any knowledge—like if there was basically nothing that we should count as established and useful enough to be codified and recorded as part of the foundations of the subject. It’s worth wondering whether this has perhaps changed significantly in the last 5-10 years though, i.e. during TurnTrout’s PhD. That is, perhaps—during that time—the subject has grown a lot and at least some things have been sufficiently ‘deconfused’ to have become more established concepts etc. But generally, if there are now indeed such things, then these are probably things that people entering the field should learn about. And it would seem likely that a lot of the more established ‘big names’/productive people actually know a lot of these things and that “catching up with them” is a pretty good instrumental/proxy way to get relevant knowledge that will help you do alignment work. (I almost want to say: I know it’s not fashionable in rationality to think this, but wanting to impress the teacher really does work pretty well in practice when starting out!)
Focussing on seeming smart and defensible probably can ultimately lead to a bad mistake. But when framed more as “It’s important to come across as credible” or “It’s not enough to be smart or even right; you actually do need to think about how others view you and interact with you”, it’s not at all clear that it’s a bad thing; and certainly it more clearly touches on a regular topic of discussion in EA/rationality about how much to focus on how one is seen or how ‘we’ are viewed by outsiders. Fwiw I don’t see any real “mistake” being actually described in this part of the post. In my opinion, when starting out, probably it is kinda important to build up your credibility more carefully. Then when Quintin came to TurnTrout, he writes that it took “a few days” to realize that Quintin’s ideas could be important and worth pursuing. Maybe the expectation in hindsight would be that he should have had the ‘few days’ old reaction immediately?? But my gut reaction is that that would be way too critical of oneself and actually my thought is more like ‘woah he realised that after thinking about it for only a few days; that’s great’. Can the whole episode not be read as a straightforward win: “Early on, it is important to build your own credibility by being careful about your arguments and being able to back up claims that you make in formal, public ways. Then as you gain respect for the right reasons, you can choose when and where to ‘spend’ your credibility… here’s a great example of that...”
And then re: deference, certainly it was true for me that when I was starting out in my PhD, if I got confused reading a paper or listening to talk, I was likely to be the one who was wrong. Later on or after my PhD, then, yeah, when I got confused by someone else’s presentation, I was less likely to be wrong and it was more likely I was spotting an error in someone else’s thinking. To me this seems like a completely normal product of the education and sort of the correct thing to be happening. i.e. Maybe the correct thing to do is to defer more when you have less experience and to gradually defer less as you gain knowledge and experience? I’m thinking that under the simple model that when one is confused about something, either you’re misunderstanding or the other person is wrong, one starts out in the regime where your confusion is much more often better explained by the fact you have misunderstood and you end up in the regime where you actually just have way more experience thinking about these things and so are now more reliably spotting other people’s errors. The rational response to the feeling of confusion changes because once fully accounted for the fact you just know way more stuff and are a way more experienced thinker about alignment. (One also naturally gains a huge boost to confidence as it becomes clear you will get your PhD and have good postdoc prospects etc… so it becomes easier to question ‘authority’ for that reason too, but it’s not a fake confidence boost; this is mostly a good/useful effect because you really do now have experience of doing research yourself, so you actually are more likely to be better at spotting these things).