Scott Aaronson recently wrote something relevant to these issues:
Max Ra: What would change your mind to explore research on the AI alignment problem? For a week? A month? A semester?
Scott: The central thing would be finding an actual potentially-answerable technical question around AI alignment, even just a small one, that piqued my interest and that I felt like I had an unusual angle on. In general, I have an absolutely terrible track record at working on topics because I abstractly feel like I “should” work on them. My entire scientific career has basically just been letting myself get nerd-sniped by one puzzle after the next.
Matt Putz: [...] do you think that money could ever motivate you to work on AI Alignment. If it was enough money? Can you imagine any amount that would make you say “okay, at this point I’ll switch, I’ll make a full-hearted effort to actually think about this for a year, I’d be crazy to do anything else”. If so, do you feel comfortable sharing that amount (even if it’s astronomically high)?
Scott: For me personally, it’s not about money. For my family, I think a mere, say, $500k could be enough for me to justify to them why I was going on leave from UT Austin for a year to work on AI alignment problems, if there were some team that actually had interesting problems to which I could contribute something.
Shmi: I’d guess that to get attention of someone like Scott, one would have to ask a question that sound like (but make more sense than) “what is the separation of complexity classes between aligned and unaligned AI in a particular well defined setup?” or “A potential isomorphism between Eliciting Latent Knowledge and termination of string rewriting” or “Calculating SmartVault action sequences with matrix permanent”
Scott: LOL, yes, that’s precisely the sort of thing it would take to get me interested, as opposed to feeling like I really ought to be interested.
I wonder how valuable it would be to have a high quality post or sequence on open problems in AI alignment that is substantially optimized for nerd sniping. Is it even possible to make something like this?
Extremely valuable I’d guess, but the whole problem is that alignment is still preparadigmatic. We don’t actually know yet what the well-defined nerd snipe questions we should be asking are.
I think that preparadigmatic research and paradigmatic research are two different skill sets, and most Highly Impressive People in mainstream STEM are masters at the later, not the former.
I do think we’re more paradigmatic than we were a year ago, and that we might transition fully some time soon. I’ve got a list of concrete experiments on modularity in ML systems I’d like run for example, and I think any ML savvy person could probably do those, no skill at thinking about fuzzy far mode things required.
So I’m not sure a sequence like this could be written today, but maybe in six months?
Scott Aaronson recently wrote something relevant to these issues:
There is also a question on EA Forum about the same issue: What are the coolest topics in AI safety, to a hopelessly pure mathematician?
I wonder how valuable it would be to have a high quality post or sequence on open problems in AI alignment that is substantially optimized for nerd sniping. Is it even possible to make something like this?
Extremely valuable I’d guess, but the whole problem is that alignment is still preparadigmatic. We don’t actually know yet what the well-defined nerd snipe questions we should be asking are.
I think that preparadigmatic research and paradigmatic research are two different skill sets, and most Highly Impressive People in mainstream STEM are masters at the later, not the former.
I do think we’re more paradigmatic than we were a year ago, and that we might transition fully some time soon. I’ve got a list of concrete experiments on modularity in ML systems I’d like run for example, and I think any ML savvy person could probably do those, no skill at thinking about fuzzy far mode things required.
So I’m not sure a sequence like this could be written today, but maybe in six months?