Thanks for responding!
Slow or fast takeoff, in my understanding, refers to how fast an AGI can/will improve itself to (wildly) superintelligent levels. Discontinuity seems to be a key differentiator here.
In the post you link, Christiano is arguing against discontinuity. He may expect quick RSI after AGI is here, though, so I could be mistaken.
Re 1: that’s not what slow takeoff means, and experts don’t agree on FOOM after AGI. Slow takeoff applies to AGI specifically, not to pre-AGI AIs. And I’m pretty sure at least Christiano and Hanson don’t expect FOOM, but like you am open to be corrected.
Alright, thanks for your answer. That doesn’t sound like a promising future to me though—is it that AGI is still a long way off in your view?
Great post! I enjoyed reading it, and agree with most if not all of it. We really should dream bigger!
It also sounds like your estimate of extinction caused by AGI is relatively low. Is that correct?
It seems to me joshc is arguing that it’s relatively easy to make money with AI startups at the moment.
“Thinking “the model says I will choose A, therefore I have to (???) choose A” is total nonsense.”
I choose whatever I want, knowing that it means the predictor predicted that choice.
In Bomb, if I choose Left, the predictor will have predicted that (given subjunctive dependence). Yes, the predictor said it predicted Right in the problem description; but if I choose Left, that simply means the problem ran differently from the start. It means, starting from the beginning, the predictor predicts I will choose Left, doesn’t put a bomb in Left, doesn’t leave the “I predicted you will pick Right”-note (but maybe leaves a “I predicted you will pick Left”-note) , and then I indeed choose Left, letting me live for free.
If the model is in fact (near) perfect, then choosing B means the model chose B too. That may seem like changing the past, but it really isn’t, that’s just the confusing way these problems are set up.
Claiming you can choose something a (near) perfect model of you didn’t predict is like claiming two identical calculators can give a different answer to 2 + 2.
Yes, FDT insists that actually, you must choose in advance (by “choosing your algorithm” or what have you), and must stick to the choice no matter what. But that is a feature of FDT, it is not a feature of the scenario!
FDT doesn’t insist on this at all. FDT recognizes that IF your decision procedure is modelled prior to your current decision, than you did in fact choose in advance. If an FDT’er playing Bomb doesn’t believe her decision procedure was being modelled this way, she wouldn’t take Left!
If and only if it is a feature of the scenario, then FDT recognizes it. FDT isn’t insisting the world to be a certain way. I wouldn’t be a proponent of it if it did.
But this assumption has to be added, since otherwise FDT says to take Right.
Why? FDT isn’t influenced in its decision by the note, so there is no loss of subjunctive dependence when this assumption isn’t added. (Or so it seems to me: I am operating at the limits of my FDT-knowledge here.)
Thanks for responding!
I personally agree that cooperating in the Twin PD is clearly the correct choice, but I don’t think it is meaningful to argue for this on the grounds of decision-theoretic performance (as you seem to do). From The lack of performance metrics for CDT versus EDT, etc. by Caspar Oesterheld:
I disagree. There’s a clear measure of performance given in the Twin PD: the utilities.
I disagree with Oesterheld’s point about CDT vs EDT and metrics; I think we know enough math to say EDT is simply a wrong decision theory. We could, in principle, even demonstrate this in real life, by having e.g. 1000 people play a version of XOR Blackmail (500 people with and 500 people without a “termite infestation”) and see which theory performs best. It’ll be trivial to see EDT makes the wrong decision.
Thanks for your reaction!
This doesn’t seem right: you already exist!
Sure, I already exist; together with the fact that I make the exact same decision my father made, that implies I procreate and therefore I’m not a CDT’er.
The point with these problems is, I believe, that your decision procedure is implemented at least 1 time, but possibly 2 times throughout time—depending on what your decision procedure outputs.
In Procreation*, if “my” decision procedure outputs “procreate”, it first does so “in” my father, who then procreates, causing me to exist. I then also procreate.
But if “my” decision procedure outputs “don’t procreate”, it also first does so “in” my father, who then doesn’t procreate, and then I don’t exist.
The question “Should I procreate?” is a bit misleading, then, as I possibly don’t exist.
Or, we indeed assume I do exist; but then it’s not much of a decision problem anymore. If I exist, then my father procreated, and I necessarily procreate too.
I am much more optimistic about our future than Yudkowsky is. But that is not the topic of this post.
I would very much like to know why you are much more optimistic! Have you written about that?
Thanks! I should have been more clear; “Nature is healing” has some “EY was wrong in his post” energy I was wondering about.
What do you mean with “Finally, nature is healing”?
Sorry, I should have been more clear. I know about FOOM; I was curious as to why you believe EY was wrong on FOOM and why you suggest the update on x-risk.
Could you explain point A?
Suppose the reader has a well defined utility function where death of torture are set to minus infinity. Then the writer can’t persuade them to trade off death or torture against any finite amount of utility. So, in what sense is the reader wrong about their own preferences?
I think the original Bomb scenario should have come with a, say, $1,000,000 value for “not being blown up”. That would have allowed for easy and agreed-upon expected utility calculations.
What makes the bomb dilemma seem unfair to me is the fact that it’s conditioning on an extremely unlikely event. The only way we blow up is if the predictor predicted incorrectly. But by assumption, the predictor is near-perfect. So it seems implausible that this outcome would ever happen.
Although I strongly disagree with Achmiz on the Bomb scenario in general, here we agree: Bomb is perfectly fair. You just have to take the probabilities into account, after which—if we value life at, say, $1,000,000 - Left-boxing is the only correct strategy.
CDT indeed Right-boxes, thereby losing utility.
I’d question the “quick” of “quick RSI”, but yes, he expects AI to make better AI before AGI.