I don’t see how this relates to the Orthogonality Thesis.
It relates to it because it’s an explicit component of it, no? The point being that if there is only one way of general cognition to work, perhaps that way by default involves self-reflection, which brings us to the second point...
Do you believe that an agent which terminally values tiny molecular squiggles would “question its goals and motivations” and conclude that creating squiggles is somehow “unethical”?
Yes, that’s what I’m suggesting; not saying it’s definitely true; but it’s not obviously wrong, either. Haven’t read the sequence, but I’ll try to find the time to do so—but basically I question the wording ‘terminally values’. I think that perhaps general intelligence tends to avoid valuing anything terminally (what do we humans value terminally?)
I think reflective stability, as it is usually used on LW, means something more narrow than how you’re interpreting it
Possibly, but I’m responding to its definition in the OT post I linked to, in which it’s used to mean that agents will avoid making changes that may affect their dedication to their goals.
It relates to it because it’s an explicit component of it, no? The point being that if there is only one way of general cognition to work, perhaps that way by default involves self-reflection, which brings us to the second point...
Yes, that’s what I’m suggesting; not saying it’s definitely true; but it’s not obviously wrong, either. Haven’t read the sequence, but I’ll try to find the time to do so—but basically I question the wording ‘terminally values’. I think that perhaps general intelligence tends to avoid valuing anything terminally (what do we humans value terminally?)
Possibly, but I’m responding to its definition in the OT post I linked to, in which it’s used to mean that agents will avoid making changes that may affect their dedication to their goals.