If that’s the case, then I assume that you defect in the twin prisoner’s dilemma.
I do. I would rather be someone who didn’t. But I don’t see path to becoming that person without lobotomizing myself. And it’s not a huge concern of mine, since I don’t expect to encounter such a dilemma. (Rarely am I the one pointing out that a philosophical thought experiment is unrealistic. It’s not usually the point of thought experiments to be realistic—we usually only talk about them to evaluate the consequences of different positions. But it is worth noting here that I don’t see this as a major issue for me.) I haven’t written this up because I don’t think it’s particularly urgent to explain to people why I think CDT is correct over FDT. Indeed, in one view, it would be cruel of me to do so! And I don’t think it matters much for AI alignment.
Don’t you think that’s at least looking into?
This was partly why I decided to wade into the weeds, because absent a discussion of how plausible it is that we could affect things non-causally, yes, one’s first instinct would be that we should look at least into it. And maybe, like, 0.1% of resources directed toward AI Safety should go toward whether we can change Math, but honestly, even that seems high. Because what we’re talking about is changing logical facts. That might be number 1 on my list of intractable problems.
After all, CDT evaluates causal counterfactuals, which are just as much a fiction as logical counterfactuals.
This is getting subtle :) and it’s hard to make sure our words mean things, but I submit that causal counterfactuals are much less fictitious than logical counterfactuals! I submit that it is less extravagant to claim we can affect this world than it is to claim that we can affect hypothetical worlds with which we are not in causal contact. No matter what action I pick, math stays the same. But it’s not the case that no matter what action I pick, the world stays the same. (In the former case, which action I pick could in theory tell us something about what mathematical object the physical universe implements, but it doesn’t change math.) In both cases, yes, there is only one action that I do take, but assuming we can reason both about causal and logical counterfactuals, we can still talk sensibly about the causal and logical consequences of picking actions I won’t in fact end up picking. I don’t have a complete answer to “how should we define causal/logical counterfactuals” but I don’t think I need to for the sake of this conversation, as long as we both agree that we can use the terms in more or less the same way, which I think we are successfully doing.
I don’t yet see why creating a CDT agent avoids catastrophe better than FDT.
I think running an aligned FDT agent would probably be fine. I’m just arguing that it wouldn’t be any better than running a CDT agent (besides for the interim phase before Son-of-CDT has been created). And indeed, I don’t think any new decision theories will perform any better than Son-of-CDT, so it doesn’t seem to me to be a priority for AGI safety. Finally, the fact that no FDT agent has actually been fully defined certainly weighs in favor of just going with a CDT agent.
I jumped off a small cliff into a lake once, and when I was standing on the rock, I couldn’t bring myself to jump. I stepped back to let another person go, and then I stepped onto the rock and jumped immediately. I might be able to do something similar.
But I wouldn’t be able to endorse such behavior while reflecting on it if I were in that situation, given my conviction that I am unable to change math. Indeed, I don’t think it would be wise of me to cooperate in that situation. What I really mean when I say that I would rather be someone who cooperated in a twin prisoners dilemma is “conditioned the (somewhat odd) hypothetical that I will at some point end up in a high stakes twin prisoner’s dilemma, I would rather it be the case that I am the sort of person who cooperates”, which is really saying that I would rather play a twin prisoner’s dilemma game against a cooperator than against a defector, which is just an obvious preference for a favorable event to befall me rather than an unfavorable one. In similar news, conditioned on my encountering a situation in the future where somebody checks to see if am I good person, and if I am, they destroy the world, then I would like to become a bad person. Conditioned on my encountering a situation in which someone saves the world if I am devout, I would like to become a devout person.
If I could turn off the part of my brain that forms the question “but why should I cooperate, when I can’t change math?” that would be a path to becoming a reliable cooperator, but I don’t see a path to silencing a valid argument in my brain without a lobotomy (short of possibly just cooperating really fast without thinking, and of course without forming the doubt “wait, why am I trying to do this really fast without thinking?”).
I think it’s worth pointing out that I agree that you can’t change math. I don’t think I can change math. Yet, I would still cooperate. The whole thing about whether we can literally change math is missing a crux. Thankfully, logical counterfactuals are not construed in such a silly way.
This is similar to the debate over whether free will exists when physics is deterministic. “You can’t change the future. It is already fixed...” the poor soul said, before walking off a cliff.
I do. I would rather be someone who didn’t. But I don’t see path to becoming that person without lobotomizing myself. And it’s not a huge concern of mine, since I don’t expect to encounter such a dilemma. (Rarely am I the one pointing out that a philosophical thought experiment is unrealistic. It’s not usually the point of thought experiments to be realistic—we usually only talk about them to evaluate the consequences of different positions. But it is worth noting here that I don’t see this as a major issue for me.) I haven’t written this up because I don’t think it’s particularly urgent to explain to people why I think CDT is correct over FDT. Indeed, in one view, it would be cruel of me to do so! And I don’t think it matters much for AI alignment.
This was partly why I decided to wade into the weeds, because absent a discussion of how plausible it is that we could affect things non-causally, yes, one’s first instinct would be that we should look at least into it. And maybe, like, 0.1% of resources directed toward AI Safety should go toward whether we can change Math, but honestly, even that seems high. Because what we’re talking about is changing logical facts. That might be number 1 on my list of intractable problems.
This is getting subtle :) and it’s hard to make sure our words mean things, but I submit that causal counterfactuals are much less fictitious than logical counterfactuals! I submit that it is less extravagant to claim we can affect this world than it is to claim that we can affect hypothetical worlds with which we are not in causal contact. No matter what action I pick, math stays the same. But it’s not the case that no matter what action I pick, the world stays the same. (In the former case, which action I pick could in theory tell us something about what mathematical object the physical universe implements, but it doesn’t change math.) In both cases, yes, there is only one action that I do take, but assuming we can reason both about causal and logical counterfactuals, we can still talk sensibly about the causal and logical consequences of picking actions I won’t in fact end up picking. I don’t have a complete answer to “how should we define causal/logical counterfactuals” but I don’t think I need to for the sake of this conversation, as long as we both agree that we can use the terms in more or less the same way, which I think we are successfully doing.
I think running an aligned FDT agent would probably be fine. I’m just arguing that it wouldn’t be any better than running a CDT agent (besides for the interim phase before Son-of-CDT has been created). And indeed, I don’t think any new decision theories will perform any better than Son-of-CDT, so it doesn’t seem to me to be a priority for AGI safety. Finally, the fact that no FDT agent has actually been fully defined certainly weighs in favor of just going with a CDT agent.
You could just cooperate, without taking such drastic measures, no?
I jumped off a small cliff into a lake once, and when I was standing on the rock, I couldn’t bring myself to jump. I stepped back to let another person go, and then I stepped onto the rock and jumped immediately. I might be able to do something similar.
But I wouldn’t be able to endorse such behavior while reflecting on it if I were in that situation, given my conviction that I am unable to change math. Indeed, I don’t think it would be wise of me to cooperate in that situation. What I really mean when I say that I would rather be someone who cooperated in a twin prisoners dilemma is “conditioned the (somewhat odd) hypothetical that I will at some point end up in a high stakes twin prisoner’s dilemma, I would rather it be the case that I am the sort of person who cooperates”, which is really saying that I would rather play a twin prisoner’s dilemma game against a cooperator than against a defector, which is just an obvious preference for a favorable event to befall me rather than an unfavorable one. In similar news, conditioned on my encountering a situation in the future where somebody checks to see if am I good person, and if I am, they destroy the world, then I would like to become a bad person. Conditioned on my encountering a situation in which someone saves the world if I am devout, I would like to become a devout person.
If I could turn off the part of my brain that forms the question “but why should I cooperate, when I can’t change math?” that would be a path to becoming a reliable cooperator, but I don’t see a path to silencing a valid argument in my brain without a lobotomy (short of possibly just cooperating really fast without thinking, and of course without forming the doubt “wait, why am I trying to do this really fast without thinking?”).
I think it’s worth pointing out that I agree that you can’t change math. I don’t think I can change math. Yet, I would still cooperate. The whole thing about whether we can literally change math is missing a crux. Thankfully, logical counterfactuals are not construed in such a silly way.
This is similar to the debate over whether free will exists when physics is deterministic. “You can’t change the future. It is already fixed...” the poor soul said, before walking off a cliff.