As well as various self modification and coherence arguments, there is also the “frictionless sphere in a vacuum”.
Sometimes we assume an object is a sphere, when its actually slightly aspherical, just because spheres are so easy to reason about and the approximation is good enough.
There is also an argument that goes “we aren’t claiming that all AI’s will optimize for fixed goals, it just happens that most of the research is on such AI’s”. And finally, if an AI is too incoherent, it may sit there fighting itself, ie doing nothing.
Remember that in some sense, time consistent agents are a small subset of all programs. You have to make a cart out of something specific, you can’t make a cart out of “nonwood materials”. https://www.lesswrong.com/posts/2mLZiWxWKZyaRgcn7/selling-nonapples If anyone can describe a specific design that isn’t a time consistent outer optimizer, and that has interesting safety properties, that would be valuable. If you are telling us that nonconsistant AI’s should be in our search space, yes they should. They might be not too high in the search ordering, because when you find a smaller, easier to understand subset of a space that has nice mathematical properties, you usually want to look there first.
As well as various self modification and coherence arguments, there is also the “frictionless sphere in a vacuum”.
Sometimes we assume an object is a sphere, when its actually slightly aspherical, just because spheres are so easy to reason about and the approximation is good enough.
There is also an argument that goes “we aren’t claiming that all AI’s will optimize for fixed goals, it just happens that most of the research is on such AI’s”. And finally, if an AI is too incoherent, it may sit there fighting itself, ie doing nothing.
Remember that in some sense, time consistent agents are a small subset of all programs. You have to make a cart out of something specific, you can’t make a cart out of “nonwood materials”. https://www.lesswrong.com/posts/2mLZiWxWKZyaRgcn7/selling-nonapples If anyone can describe a specific design that isn’t a time consistent outer optimizer, and that has interesting safety properties, that would be valuable. If you are telling us that nonconsistant AI’s should be in our search space, yes they should. They might be not too high in the search ordering, because when you find a smaller, easier to understand subset of a space that has nice mathematical properties, you usually want to look there first.