TurnTrout comments on A shot at the diamond-alignment problem

[TurnTrout]

How is the bolded sentence different from the following:

”Consider the expected consequences of the plan “think a lot longer and harder, considering a lot more possibilities for what you should do, and then make your decision.” I currently predict that such a plan would lead future-me to waste his life doing philosophy or maybe get pascal’s mugged by some longtermist AI bullshit instead of actually helping people with his donations. My helping-people shard doesn’t like this plan, because it predicts abstractly that thinking a lot more will not result in helping people more.”

(Basically I’m saying you should think more, and then write more, about the difference between these two cases because they seem plausibly on a spectrum to me, and this should make us nervous in a couple of ways. Are we actually being really stupid by being EAs and shutting up and calculating? Have we basically adversarial-exampled ourselves away from doing things that we actually thought were altruistic and effective back in the day? If not, what’s different about the kind of extended search process we did, from the logical extension of that which is to do an even more extended search process, a sufficiently extreme search process that outsiders would call the result an adversarial example?)

I think this is a great observation. I thought about it a bit and don’t really find myself worried, based off of some intuitions which I think would take me at least 20 minutes to type up right now, and I really should wrap my commenting up for now. Feel free to ping me if no one else has answered this in a while.

Seems plausible to me but I’m a bit nervous, I think it could totally turn out to not work like that.

Agreed.

[Daniel Kokotajlo]

Consider yourself pinged! No rush to reply though.