I agree with you that framing is important; I just deleted the old ETA. (For anyone interested, it used to read:
ETA: Having talked just now to people at our open house, I would like to clarify:
Even though our aim is explicitly AI Safety... CFAR does still need an art of rationality, and a community of rationality geeks that support that. We will still be investing at least some in that community. We will also still be running some “explore” workshops of different sorts aiming at patching gaps in the art (funding permitting), not all of which will be deliberately and explicitly backchained form AI Safety (although some will). Play is generative of a full rationality art.
(In addition to sometimes targeting things more narrowly at particular high-impact groups, and otherwise more directly backchaining.)
(More in subsequent posts.)
I’m curious where our two new docs leave you; I think they make clearer that we will still be doing some rationality qua rationality.
Will comment later re: separate organizations; I agree this is an interesting idea; my guess is that there isn’t enough money and staff firepower to run a good standalone rationality organization in CFAR’s stead, and also that CFAR retains quite an interest in a standalone rationality community and should therefore support it… but I’m definitely interested in thoughts on this.
Julia will be launching a small spinoff organization called Convergence, facilitating double crux conversations between EAs and EA-adjacent people in, e.g., tech and academia. It’ll be under the auspices of CFAR for now but will not have opinions on AI. I’m not sure if that hits any of what you’re after.
I agree with you that framing is important; I just deleted the old ETA.
For what it’s worth I think even without saying that your aim is explicitly AI safety, a lot of people reading this post will take that away unless you do more to cancel the implicature. Even the title does this! It’s a slightly odd grammatical construction which looks an awful lot like CFAR’s new focus: AI Safety; I think without being more up-front about alternative interpretation it will sometimes be read that way.
I’m curious where our two new docs leave you
Me too! (I assume that these have not been posted yet, but if I’m just failing to find them please let me know.)
I think they make clearer that we will still be doing some rationality qua rationality.
Great. Just to highlight that I think there are two important aspects of doing rationality qua rationality:
Have the people pursuing the activity have this as their goal. (I’m less worried about you failing on this one.)
Have external perceptions be that this is what you’re doing. I have some concern that rationality-qua-rationality activities pursued by an AI safety org will be perceived as having an underlying agenda relating to that. And that this could e.g. make some people less inclined to engage, even relative to if they’re run by a rationality org which has a significant project on AI safety.
my guess is that there isn’t enough money and staff firepower to run a good standalone rationality organization in CFAR’s stead
I feel pretty uncertain about this, but my guess goes the other way. Also, I think if there are two separate orgs, the standalone rationality one should probably retain the CFAR brand! (as it seems more valuable there)
I do worry about transition costs and losing synergies of working together from splitting off a new org. Though these might be cheaper earlier than later, and even if it’s borderline right now whether there’s enough money and staff to do both I think it won’t be borderline within a small number of years.
Julia will be launching a small spinoff organization called Convergence
This sounds interesting! That’s a specialised enough remit that it (mostly) doesn’t negate my above concerns, but I’m happy to hear about it anyway.
Even the title does this! It’s a slightly odd grammatical construction which looks an awful lot like CFAR’s new focus: AI Safety; I think without being more up-front about alternative interpretation it will sometimes be read that way.
Datapoint: it wasn’t until reading your comment that I realized that the title actually doesn’t read “CFAR’s new focus: AI safety”.
Thanks. I’ll dwell more on these. Quick thoughts from a first read:
I generally liked the “further discussion” doc.
I do think it’s important to strongly signal the aspects of cause neutrality that you do intend to pursue (as well as pursuing them). These are unusual and important.
I found the mission statement generally opaque and extremely jargony. I think I could follow what you were saying, but in some cases this required a bit of work and in some cases I felt like it was perhaps only because I’d had conversations with you. (The FAQ at the top was relatively clear, but an odd thing to lead with.)
I was bemused by the fact that there didn’t appear to be a clear mission statement highlighted anywhere on the page!
ETA: Added some more in depth comments on the relevant comment threads: here on “further thoughts”, and here and here on the mission statement.
To get a better idea of your model of what you expect the new focus to do, here’s a hypothetical. Say we have a rationality-qua-rationality CFAR (CFAR-1) and an AI-Safety CFAR (CFAR-2). Each starts with the same team, works independently of each other, and they can’t share work. Two years later, we ask them to write a curriculum for the other organization, to the best of their abilities. This is along the lines of having them do an Ideological Turing Test on each other. How well do they match? In addition, is the newly written version better in any case? Is CFAR-1′s CFAR-2 curriculum better than CFAR-2′s CFAR-2 curriculum?
I’m treating curriculum quality as a proxy for research progress, and somewhat ignoring things like funding and operations quality. The question is only meant to address worries of research slowdowns.
Thanks for the thoughts; I appreciate it.
I agree with you that framing is important; I just deleted the old ETA. (For anyone interested, it used to read:
I’m curious where our two new docs leave you; I think they make clearer that we will still be doing some rationality qua rationality.
Will comment later re: separate organizations; I agree this is an interesting idea; my guess is that there isn’t enough money and staff firepower to run a good standalone rationality organization in CFAR’s stead, and also that CFAR retains quite an interest in a standalone rationality community and should therefore support it… but I’m definitely interested in thoughts on this.
Julia will be launching a small spinoff organization called Convergence, facilitating double crux conversations between EAs and EA-adjacent people in, e.g., tech and academia. It’ll be under the auspices of CFAR for now but will not have opinions on AI. I’m not sure if that hits any of what you’re after.
Thanks for engaging. Further thoughts:
For what it’s worth I think even without saying that your aim is explicitly AI safety, a lot of people reading this post will take that away unless you do more to cancel the implicature. Even the title does this! It’s a slightly odd grammatical construction which looks an awful lot like CFAR’s new focus: AI Safety; I think without being more up-front about alternative interpretation it will sometimes be read that way.
Me too! (I assume that these have not been posted yet, but if I’m just failing to find them please let me know.)
Great. Just to highlight that I think there are two important aspects of doing rationality qua rationality:
Have the people pursuing the activity have this as their goal. (I’m less worried about you failing on this one.)
Have external perceptions be that this is what you’re doing. I have some concern that rationality-qua-rationality activities pursued by an AI safety org will be perceived as having an underlying agenda relating to that. And that this could e.g. make some people less inclined to engage, even relative to if they’re run by a rationality org which has a significant project on AI safety.
I feel pretty uncertain about this, but my guess goes the other way. Also, I think if there are two separate orgs, the standalone rationality one should probably retain the CFAR brand! (as it seems more valuable there)
I do worry about transition costs and losing synergies of working together from splitting off a new org. Though these might be cheaper earlier than later, and even if it’s borderline right now whether there’s enough money and staff to do both I think it won’t be borderline within a small number of years.
This sounds interesting! That’s a specialised enough remit that it (mostly) doesn’t negate my above concerns, but I’m happy to hear about it anyway.
Datapoint: it wasn’t until reading your comment that I realized that the title actually doesn’t read “CFAR’s new focus: AI safety”.
+1
Oh, sorry, the two new docs are posted and were in the new ETA:
http://lesswrong.com/lw/o9h/further_discussion_of_cfars_focus_on_ai_safety/ and http://lesswrong.com/r/discussion/lw/o9j/cfars_new_mission_statement_on_our_website/
Thanks. I’ll dwell more on these. Quick thoughts from a first read:
I generally liked the “further discussion” doc.
I do think it’s important to strongly signal the aspects of cause neutrality that you do intend to pursue (as well as pursuing them). These are unusual and important.
I found the mission statement generally opaque and extremely jargony. I think I could follow what you were saying, but in some cases this required a bit of work and in some cases I felt like it was perhaps only because I’d had conversations with you. (The FAQ at the top was relatively clear, but an odd thing to lead with.)
I was bemused by the fact that there didn’t appear to be a clear mission statement highlighted anywhere on the page!
ETA: Added some more in depth comments on the relevant comment threads: here on “further thoughts”, and here and here on the mission statement.
To get a better idea of your model of what you expect the new focus to do, here’s a hypothetical. Say we have a rationality-qua-rationality CFAR (CFAR-1) and an AI-Safety CFAR (CFAR-2). Each starts with the same team, works independently of each other, and they can’t share work. Two years later, we ask them to write a curriculum for the other organization, to the best of their abilities. This is along the lines of having them do an Ideological Turing Test on each other. How well do they match? In addition, is the newly written version better in any case? Is CFAR-1′s CFAR-2 curriculum better than CFAR-2′s CFAR-2 curriculum?
I’m treating curriculum quality as a proxy for research progress, and somewhat ignoring things like funding and operations quality. The question is only meant to address worries of research slowdowns.