Expected evidence is conserved if almost anything you could observe would update you slightly in the same direction, with the much rarer cases updating you correspondingly much further in the opposite direction.
For what it’s worth, I also read this as Eliezer reporting on a case where he later realised that his views were violating CoEE. This is equivalent to observing evidence, realising that your prior should have given it more credence, and then updating in the same direction you moved your prior. Sounds awful but can be reasonable for computationally bounded agents!
Expected evidence is conserved if almost anything you could observe would update you slightly in the same direction, with the much rarer cases updating you correspondingly much further in the opposite direction.
For what it’s worth, I also read this as Eliezer reporting on a case where he later realised that his views were violating CoEE. This is equivalent to observing evidence, realising that your prior should have given it more credence, and then updating in the same direction you moved your prior. Sounds awful but can be reasonable for computationally bounded agents!