The concern here is that if I reject all offers that superficially pattern-match to this sort of scam, I run the risk of turning down valuable offers as well. (I’m reminded of a TV show decades ago where they had some guy dress like a bum and wander down the street offering people $20, and everyone ignored him.)
Of course, if I’m not smart enough to actually evaluate the situation, or don’t feel like spending the energy, then superficial pattern-matching and rejection is my safest strategy, as you suggest.
But the question of what analysis a sufficiently smart and attentive agent could do, in principle, to take advantage of rare valuable opportunities without being suckered by scam artists is often worth asking anyway.
It depends on the nature of the analysis I’m doing.
I mean, sure, if the scam artist is smart enough to, for example, completely encapsulate my sensorium and provide me with an entirely simulated world that it updates in real time and perfect detail, then all bets are off… it can make me believe anything by manipulating the evidence I observe. (Similarly, if the scam artist is smart enough to directly manipulate my brain/mind.)
But if my reasoning is reliable and I actually have access to evidence about the real world, then the better I am at evaluating that evidence, the harder I am to scam about things relating to that evidence, even by a scam artist far smarter than me.
I disagree. All the scam artist has to know is your method of coming to your conclusions. Once he knows that then he can probably exploit you depending on his cleverness (and then it becomes an arms race). If anything, trying to defend yourself from being manipulated in that way would probably be extremely difficult in of itself. Either way, my initial guess is that your methodology would still be superficial pattern-matching, but it would just be a deeper, more complex level of it.
This seems to be what Eliezer is doing with all the various scenarios. He’s testing his methodology against different attacks and different scenarios. I’m just suggesting is to change your viewpoint to the Bad Guy. Rather than talk about your reliable reasoning, talk about the bad guy and how he can exploit your reasoning.
The concern here is that if I reject all offers that superficially pattern-match to this sort of scam, I run the risk of turning down valuable offers as well. (I’m reminded of a TV show decades ago where they had some guy dress like a bum and wander down the street offering people $20, and everyone ignored him.)
Of course, if I’m not smart enough to actually evaluate the situation, or don’t feel like spending the energy, then superficial pattern-matching and rejection is my safest strategy, as you suggest.
But the question of what analysis a sufficiently smart and attentive agent could do, in principle, to take advantage of rare valuable opportunities without being suckered by scam artists is often worth asking anyway.
But wouldn’t you just be suckered by sufficiently smart and attentive scam artists?
It depends on the nature of the analysis I’m doing.
I mean, sure, if the scam artist is smart enough to, for example, completely encapsulate my sensorium and provide me with an entirely simulated world that it updates in real time and perfect detail, then all bets are off… it can make me believe anything by manipulating the evidence I observe. (Similarly, if the scam artist is smart enough to directly manipulate my brain/mind.)
But if my reasoning is reliable and I actually have access to evidence about the real world, then the better I am at evaluating that evidence, the harder I am to scam about things relating to that evidence, even by a scam artist far smarter than me.
I disagree. All the scam artist has to know is your method of coming to your conclusions. Once he knows that then he can probably exploit you depending on his cleverness (and then it becomes an arms race). If anything, trying to defend yourself from being manipulated in that way would probably be extremely difficult in of itself. Either way, my initial guess is that your methodology would still be superficial pattern-matching, but it would just be a deeper, more complex level of it.
This seems to be what Eliezer is doing with all the various scenarios. He’s testing his methodology against different attacks and different scenarios. I’m just suggesting is to change your viewpoint to the Bad Guy. Rather than talk about your reliable reasoning, talk about the bad guy and how he can exploit your reasoning.
Fair enough. If I accept that guess as true, I agree with your conclusion.
I also agree that adopting the enemy’s perspective is an important—for humans, indispensible—part of strategic thinking.