It depends on the nature of the analysis I’m doing.
I mean, sure, if the scam artist is smart enough to, for example, completely encapsulate my sensorium and provide me with an entirely simulated world that it updates in real time and perfect detail, then all bets are off… it can make me believe anything by manipulating the evidence I observe. (Similarly, if the scam artist is smart enough to directly manipulate my brain/mind.)
But if my reasoning is reliable and I actually have access to evidence about the real world, then the better I am at evaluating that evidence, the harder I am to scam about things relating to that evidence, even by a scam artist far smarter than me.
I disagree. All the scam artist has to know is your method of coming to your conclusions. Once he knows that then he can probably exploit you depending on his cleverness (and then it becomes an arms race). If anything, trying to defend yourself from being manipulated in that way would probably be extremely difficult in of itself. Either way, my initial guess is that your methodology would still be superficial pattern-matching, but it would just be a deeper, more complex level of it.
This seems to be what Eliezer is doing with all the various scenarios. He’s testing his methodology against different attacks and different scenarios. I’m just suggesting is to change your viewpoint to the Bad Guy. Rather than talk about your reliable reasoning, talk about the bad guy and how he can exploit your reasoning.
It depends on the nature of the analysis I’m doing.
I mean, sure, if the scam artist is smart enough to, for example, completely encapsulate my sensorium and provide me with an entirely simulated world that it updates in real time and perfect detail, then all bets are off… it can make me believe anything by manipulating the evidence I observe. (Similarly, if the scam artist is smart enough to directly manipulate my brain/mind.)
But if my reasoning is reliable and I actually have access to evidence about the real world, then the better I am at evaluating that evidence, the harder I am to scam about things relating to that evidence, even by a scam artist far smarter than me.
I disagree. All the scam artist has to know is your method of coming to your conclusions. Once he knows that then he can probably exploit you depending on his cleverness (and then it becomes an arms race). If anything, trying to defend yourself from being manipulated in that way would probably be extremely difficult in of itself. Either way, my initial guess is that your methodology would still be superficial pattern-matching, but it would just be a deeper, more complex level of it.
This seems to be what Eliezer is doing with all the various scenarios. He’s testing his methodology against different attacks and different scenarios. I’m just suggesting is to change your viewpoint to the Bad Guy. Rather than talk about your reliable reasoning, talk about the bad guy and how he can exploit your reasoning.
Fair enough. If I accept that guess as true, I agree with your conclusion.
I also agree that adopting the enemy’s perspective is an important—for humans, indispensible—part of strategic thinking.