Hmm, might still be something gleaned for attempting to steelman this or work in different related directions.
Edit; maybe something with an AI not being able to tolerate things it can’t make certain proofs about? Problem is it’d have to be able to make those proofs about humans if they are included in its environment, and if they are not it might make UFAI there (Intuition pump; a system that consists of a program it can prove everything about, and humans that program asks questions to). Yea this doesn’t seem very useful.
Hmm, might still be something gleaned for attempting to steelman this or work in different related directions.
Edit; maybe something with an AI not being able to tolerate things it can’t make certain proofs about? Problem is it’d have to be able to make those proofs about humans if they are included in its environment, and if they are not it might make UFAI there (Intuition pump; a system that consists of a program it can prove everything about, and humans that program asks questions to). Yea this doesn’t seem very useful.