On what basis do you think it’s the ‘best shot’? I used to think it was a good idea, a few years ago, but in retrospect I think that was just a computer scientist’s love of recursion. I don’t think that at present conditions are good for automating R&D. On the one hand, we have a lot of very smart people working on AI safety R&D, with very slow progress, indicating it is a hard problem. On the other hand, present-day LLMs are stupid at long-term planning, and acquiring new knowledge, which are things you need to be good at to do R&D.
What advantage do you see AIs having over humans in this area?
I think there will be a period in the future where AI systems (models and their scaffolding) exist which are sufficiently capable that they will be able to speed up many aspects of computer-based R&D. Including recursive-self-improvement, Alignment research and Control research. Obviously, such a time period will not be likely to last long given that surely some greedy actor will pursue RSI. So personally, that’s why I’m not putting a lot of faith in getting to that period [edit: resulting in safety].
I think that if you build the scaffolding which would make current models able to be substantially helpful at research (which would be impressively strong scaffolding indeed!), then you have built dual-use scaffolding which could also be useful for RSI. So any plans to do this must take appropriate security measures or they will be net harmful.
On what basis do you think it’s the ‘best shot’? I used to think it was a good idea, a few years ago, but in retrospect I think that was just a computer scientist’s love of recursion. I don’t think that at present conditions are good for automating R&D. On the one hand, we have a lot of very smart people working on AI safety R&D, with very slow progress, indicating it is a hard problem. On the other hand, present-day LLMs are stupid at long-term planning, and acquiring new knowledge, which are things you need to be good at to do R&D.
What advantage do you see AIs having over humans in this area?
I think there will be a period in the future where AI systems (models and their scaffolding) exist which are sufficiently capable that they will be able to speed up many aspects of computer-based R&D. Including recursive-self-improvement, Alignment research and Control research. Obviously, such a time period will not be likely to last long given that surely some greedy actor will pursue RSI. So personally, that’s why I’m not putting a lot of faith in getting to that period [edit: resulting in safety].
I think that if you build the scaffolding which would make current models able to be substantially helpful at research (which would be impressively strong scaffolding indeed!), then you have built dual-use scaffolding which could also be useful for RSI. So any plans to do this must take appropriate security measures or they will be net harmful.