I was confused about this post, and… I might have resolve my confusion by the time I got ready to write this comment. Unsure. Here goes:
My first* thought:
Am I not just allowed to precommit to “be the sort of person who always figures out whatever the optimal game theory was, and commit to that?”. I thought that was the point.
i.e. I wouldn’t precommit to treating either the Nash Bargaining Solution or Kalai-Smorodinsky Solution as “the permanent grim trigger bullying point”, I’d precommit to something like “have a meta-policy of not giving into bullying, pick my best-guess-definition-of-bullying as my default trigger, and my best-guess grim-trigger response, but include an ‘oh shit I didn’t think about X’ parameter.” (with some conditional commitments thrown in)
Where X can’t be an arbitrary new belief – the whole point of having a grim trigger clause is to be able to make appropriately weighted threats that AGI-Bob really thinks will happen. But, if I legitimately didn’t think of the Kalai-Smordinwhatever solution as something an agent might legitimately think was a good coordination tool, I want to be able to say. depending on circumstances:
If the deal hasn’t resolved yet “oh, shit I JUUUST thought of the Kalai-whatever thing and this means I shouldn’t execute my grim trigger anti-bullying clause without first offering some kind of further clarification step.”
If the deal already resolved before I thought of it, say “oh shit man I really should realized the Kalai-Smorodinsk thing was a legitimate schelling point and not started defecting hard as punishment. Hey, fellow AGI, would you like me to give you N remorseful utility in return for which I stop grim-triggering you and you stop retaliating at me and we end the punishment spiral?”
My second* thought:
Okay. So. I guess that’s easy for me to say. But, I guess the whole point of all this updateless decision theory stuff was to actually formalize that in a way that you could robustly program an AGI that you were about to give the keys to the universe.
Having a vague handwavy notion of it isn’t reassuring enough if you’re about to build a god.
And while it seems to me like this is (relatively) straightforward… do I really want to bet that?
I guess my implicit assumption was that game theory would turn out to not be that complicated in the grand scheme of thing. Surely once you’re a Jupiter Brain you’ll have it figured out? And, hrmm, maybe that’s true, but but maybe it’s not, or maybe it turns out the fate of the cosmos gets decided with smaller AGIs fighting over Earth which much more limited compute.
Third thought:
Man, just earlier this year, someone offered me a coordination scheme that I didn’t understand, and I fucked it up, and the deal fell through because I didn’t understand the principles underlying it until just-too-late. (this is an anecdote I plan to write up as a blogpost sometime soon)
And… I guess I’d been implicitly assuming that AGIs would just be able to think fast enough that that wouldn’t be a problem.
Like, if you’re talking to a used car salesman, and you say “No more than $10,000”, and then they say “$12,000 is final offer”, and then you turn and walk away, hoping that they’ll say “okay, fine, $10,000”… I suppose metaphorical AGI used car buyers could say “and if you take more than 10 compute-cycles to think about it, the deal is off.” And that might essentially limit you to only be able to make choices you’d precomputed, even if you wanted to give yourself the option to think more.
That seems to explain why my “Just before deal resolves, realize I screwed up my decision theory” idea doesn’t work.
It seems like my “just after deal resolves and I accidentally grim trigger, turn around and say ‘oh shit, I screwed up, here is remorse payment + a costly proof that that I’m not fudging my decision theory’” should still work though?
I guess in the context of Acausal Trade, I can imagine things like “they only bother running a simulation of you for 100 cycles, and it doesn’t matter if on the 101st cycle you realize you made a mistake and am sorry.” They’ll never know it.
But… I dunno man. I figured the first rule of Acausal Trade was “build a galaxy brain and think really goddamn carefully about acausal trade and philosophical competence” before you actually try simulating anything, and I’m skeptical a galaxy brain can’t figure out the right precommitments.
I dunno. Maybe I’m still confused.
But, I wanted to check in on whether I was on the right track in understanding what considerations were at play here.
...
*actually there were like 20 thoughts before I got to the one I’ve labeled ‘first thought’ here. But, “first thought that seemed worth writing down.”
Thanks! Reading this comment makes me very happy, because it seems like you are now in a similar headspace to me back in the day. Writing this post was my response to being in this headspace.
But… I dunno man. I figured the first rule of Acausal Trade was “build a galaxy brain and think really goddamn carefully about acausal trade and philosophical competence” before you actually try simulating anything, and I’m skeptical a galaxy brain can’t figure out the right precommitments.
This sounds like a plausibly good rule to me. But that doesn’t mean that every AI we build will automatically follow it. Moreover, thinking about acausal trade is in some sense engaging in acausal trade. As I put it:
Since real agents can’t be logically omniscient, one needs to decide how much time to spend thinking about things like game theory and what the outputs of various programs are before making commitments. When we add acausal bargaining into the mix, things get even more intense. Scott Garrabrant, Wei Dai, and Abram Demski have described this problem already, so I won’t say more about that here. Basically, in this context, there are many other people observing your thoughts and making decisions on that basis. So bluffing is impossible and there is constant pressure to make commitments quickly before thinking longer. (That’s my take on it anyway)
As for your handwavy proposals, I do agree that they are pretty good. They are somewhat similar to the proposals I favor, in fact. But these are just specific proposals in a big space of possible strategies, and (a) we have reason to think there might be flaws in these proposals that we haven’t discovered yet, and (b) even if these proposals work perfectly there’s still the problem of making sure that our AI follows them:
Objection: “Surely they wouldn’t be so stupid as to make those commitments—even I could see that bad outcome coming. A better commitment would be...”
Reply: The problem is that consequentialist agents are motivated to make commitments as soon as possible, since that way they can influence the behavior of other consequentialist agents who may be learning about them. Of course, they will balance these motivations against the countervailing motive to learn more and think more before doing drastic things. The problem is that the first motivation will push them to make commitments much sooner than would otherwise be optimal. So they might not be as smart as us when they make their commitments, at least not in all the relevant ways. Even if our baby AGIs are wiser than us, they might still make mistakes that we haven’t anticipated yet. The situation is like the centipede game: Collectively, consequentialist agents benefit from learning more about the world and each other before committing to things. But because they are all bullies and cowards, they individually benefit from committing earlier, when they don’t know so much.
If you want to think and talk more about this, I’d be very interested to hear your thoughts. Unfortunately, while my estimate of the commitment races problem’s importance has only increased over the past year, I haven’t done much to actually make intellectual progress on it.
I feel I should disclaim “much of what I’d have to say about this is a watered down version of whatever Andrew Critch would say”. He’s busy a lot, but if you haven’t chatted with him about this yet you probably should, and if you have I’m not sure whether I’ll have much to add.
But I am pretty interested right now in fleshing out my own coordination principles and fleshing out my understanding of how they scale up from “200 human rationalists” to 1000-10,000 sized coalitions to All Humanity and to AGI and beyond. I’m currently working on a sequence that could benefit from chatting with other people who think seriously about this.
I suppose metaphorical AGI used car buyers could say “and if you take more than 10 compute-cycles to think about it, the deal is off.” And that might essentially limit you to only be able to make choices you’d precomputed, even if you wanted to give yourself the option to think more.
“This offer is only valid if you say yes right now—if you go home and come back tomorrow, it will cost more” actually is one of those real-world dirty tricks that salespeople use to rip people off.
I was confused about this post, and… I might have resolve my confusion by the time I got ready to write this comment. Unsure. Here goes:
My first* thought:
Am I not just allowed to precommit to “be the sort of person who always figures out whatever the optimal game theory was, and commit to that?”. I thought that was the point.
i.e. I wouldn’t precommit to treating either the Nash Bargaining Solution or Kalai-Smorodinsky Solution as “the permanent grim trigger bullying point”, I’d precommit to something like “have a meta-policy of not giving into bullying, pick my best-guess-definition-of-bullying as my default trigger, and my best-guess grim-trigger response, but include an ‘oh shit I didn’t think about X’ parameter.” (with some conditional commitments thrown in)
Where X can’t be an arbitrary new belief – the whole point of having a grim trigger clause is to be able to make appropriately weighted threats that AGI-Bob really thinks will happen. But, if I legitimately didn’t think of the Kalai-Smordinwhatever solution as something an agent might legitimately think was a good coordination tool, I want to be able to say. depending on circumstances:
If the deal hasn’t resolved yet “oh, shit I JUUUST thought of the Kalai-whatever thing and this means I shouldn’t execute my grim trigger anti-bullying clause without first offering some kind of further clarification step.”
If the deal already resolved before I thought of it, say “oh shit man I really should realized the Kalai-Smorodinsk thing was a legitimate schelling point and not started defecting hard as punishment. Hey, fellow AGI, would you like me to give you N remorseful utility in return for which I stop grim-triggering you and you stop retaliating at me and we end the punishment spiral?”
My second* thought:
Okay. So. I guess that’s easy for me to say. But, I guess the whole point of all this updateless decision theory stuff was to actually formalize that in a way that you could robustly program an AGI that you were about to give the keys to the universe.
Having a vague handwavy notion of it isn’t reassuring enough if you’re about to build a god.
And while it seems to me like this is (relatively) straightforward… do I really want to bet that?
I guess my implicit assumption was that game theory would turn out to not be that complicated in the grand scheme of thing. Surely once you’re a Jupiter Brain you’ll have it figured out? And, hrmm, maybe that’s true, but but maybe it’s not, or maybe it turns out the fate of the cosmos gets decided with smaller AGIs fighting over Earth which much more limited compute.
Third thought:
Man, just earlier this year, someone offered me a coordination scheme that I didn’t understand, and I fucked it up, and the deal fell through because I didn’t understand the principles underlying it until just-too-late. (this is an anecdote I plan to write up as a blogpost sometime soon)
And… I guess I’d been implicitly assuming that AGIs would just be able to think fast enough that that wouldn’t be a problem.
Like, if you’re talking to a used car salesman, and you say “No more than $10,000”, and then they say “$12,000 is final offer”, and then you turn and walk away, hoping that they’ll say “okay, fine, $10,000”… I suppose metaphorical AGI used car buyers could say “and if you take more than 10 compute-cycles to think about it, the deal is off.” And that might essentially limit you to only be able to make choices you’d precomputed, even if you wanted to give yourself the option to think more.
That seems to explain why my “Just before deal resolves, realize I screwed up my decision theory” idea doesn’t work.
It seems like my “just after deal resolves and I accidentally grim trigger, turn around and say ‘oh shit, I screwed up, here is remorse payment + a costly proof that that I’m not fudging my decision theory’” should still work though?
I guess in the context of Acausal Trade, I can imagine things like “they only bother running a simulation of you for 100 cycles, and it doesn’t matter if on the 101st cycle you realize you made a mistake and am sorry.” They’ll never know it.
But… I dunno man. I figured the first rule of Acausal Trade was “build a galaxy brain and think really goddamn carefully about acausal trade and philosophical competence” before you actually try simulating anything, and I’m skeptical a galaxy brain can’t figure out the right precommitments.
I dunno. Maybe I’m still confused.
But, I wanted to check in on whether I was on the right track in understanding what considerations were at play here.
...
*actually there were like 20 thoughts before I got to the one I’ve labeled ‘first thought’ here. But, “first thought that seemed worth writing down.”
Thanks! Reading this comment makes me very happy, because it seems like you are now in a similar headspace to me back in the day. Writing this post was my response to being in this headspace.
This sounds like a plausibly good rule to me. But that doesn’t mean that every AI we build will automatically follow it. Moreover, thinking about acausal trade is in some sense engaging in acausal trade. As I put it:
As for your handwavy proposals, I do agree that they are pretty good. They are somewhat similar to the proposals I favor, in fact. But these are just specific proposals in a big space of possible strategies, and (a) we have reason to think there might be flaws in these proposals that we haven’t discovered yet, and (b) even if these proposals work perfectly there’s still the problem of making sure that our AI follows them:
If you want to think and talk more about this, I’d be very interested to hear your thoughts. Unfortunately, while my estimate of the commitment races problem’s importance has only increased over the past year, I haven’t done much to actually make intellectual progress on it.
Yeah I’m interested in chatting about this.
I feel I should disclaim “much of what I’d have to say about this is a watered down version of whatever Andrew Critch would say”. He’s busy a lot, but if you haven’t chatted with him about this yet you probably should, and if you have I’m not sure whether I’ll have much to add.
But I am pretty interested right now in fleshing out my own coordination principles and fleshing out my understanding of how they scale up from “200 human rationalists” to 1000-10,000 sized coalitions to All Humanity and to AGI and beyond. I’m currently working on a sequence that could benefit from chatting with other people who think seriously about this.
“This offer is only valid if you say yes right now—if you go home and come back tomorrow, it will cost more” actually is one of those real-world dirty tricks that salespeople use to rip people off.