If you haven’t already, you might want to see my answer to Steve’s comment, on why translation to low-level structure is the right problem to think about even if the AI is using higher-level models.
I did see that answer and pretty strongly agree with it, the “low-level structure” part of my summary was meant to be an example, not a central case. To make this clearer, I changed
which could potentially be detailed accurate low-level simulations
to
which could be very alien to us (e.g. perhaps the AI system uses detailed low-level simulations)
Endorsed; that definitely captures the key ideas.
If you haven’t already, you might want to see my answer to Steve’s comment, on why translation to low-level structure is the right problem to think about even if the AI is using higher-level models.
I did see that answer and pretty strongly agree with it, the “low-level structure” part of my summary was meant to be an example, not a central case. To make this clearer, I changed
to