One thing to note about RSI, we know mindless processes like gradient descent and evolution can improve performance of a model/organism enormously despite their stupidity. And so it’s not clear to me that the RSI loop has to be very smart or reliable to start making fast progress. We are approaching a point where the crystallized intelligence and programming and mathematics ability of existing models strike me as being very close to being in extremely dangerous territory. And though reliability probably needs to improve before doom—perhaps not as much as one would think.
One thing to note about RSI, we know mindless processes like gradient descent and evolution can improve performance of a model/organism enormously despite their stupidity. And so it’s not clear to me that the RSI loop has to be very smart or reliable to start making fast progress. We are approaching a point where the crystallized intelligence and programming and mathematics ability of existing models strike me as being very close to being in extremely dangerous territory. And though reliability probably needs to improve before doom—perhaps not as much as one would think.