What would an unfriendly superintelligence that wanted to hack your brain say to you? Does knowing the answer to that have positive value in your utility function?
That said, I do think information is a terminal value, at least in my utility function; but I think an exception must be made for mind-damaging truths, if such truths exist.
I don’t think the idea of a conditional terminal value is very useful. If information is a terminal value for me I’d want to know what the unfriendly superintelligence would say, but unless it’s the only terminal value and I don’t think the result would have any influence on other information gathering there would be other considerations speaking against learning that particular piece of information and probably outweighing it. No need to make any exceptions for mind damaging truths because to the extent mind damage is a bad thing according to my terminal values they will already be accounted for anyway.
What would an unfriendly superintelligence that wanted to hack your brain say to you? Does knowing the answer to that have positive value in your utility function?
That said, I do think information is a terminal value, at least in my utility function; but I think an exception must be made for mind-damaging truths, if such truths exist.
I don’t think the idea of a conditional terminal value is very useful. If information is a terminal value for me I’d want to know what the unfriendly superintelligence would say, but unless it’s the only terminal value and I don’t think the result would have any influence on other information gathering there would be other considerations speaking against learning that particular piece of information and probably outweighing it. No need to make any exceptions for mind damaging truths because to the extent mind damage is a bad thing according to my terminal values they will already be accounted for anyway.