For what it’s worth, I was just learning about the basics of MIRI’s research when this came out, and reading it made me less convinced of the value of MIRI’s research agenda. That’s not necessarily a major problem, since the expected change in belief after encountering a given post should be 0, and I already had a lot of trust in MIRI. However, I found this post by Jessica Taylor vastly clearer and more persuasive (it was written before “Rocket Alignment”, but I read “Rocket Alignment” first). In particular, I would expect AI researchers to be much more competent than the portrayal of spaceplane engineers in the post, and it wasn’t clear to me why the analogy should be strong Bayesian evidence for MIRI being correct.
Liam Donovan
Karma: 177
Maybe people who rationalized their failure to lose weight by “well, even Eliezer is overweight, it’s just metabolic disprivilege”
How many people raised their hands when Eliezer asked about the probability estimate? When I was watching the video I gave a probability estimate of 65%, and I’m genuinely shocked that “not many” people thought he had over a 55% chance. This is Eliezer we’re talking about.............
I wonder if it negatively impacts the cohesiveness/teamwork ability of the resulting AI safety community by disproportionately attracting a certain type of person? It seems unlikely that everyone would enjoy this style
.
FWIW you can bet on some of these on PredictIt—for example, Predictit assigns only a 47% chance Trump will win in 2020. That’s not a huge difference, but still worth betting 5% of your bankroll (after fees) on if you bet half-Kelly. (if you want to bet with me for whatever reason, I’d also be willing to bet up to $700 that Trump doesn’t win at PredictIt odds if I don’t have to tie up capital)
We can test if the most popular books & music of 2019 sold less copies than the most popular books & music of 2009 (I might or might not look into this later)
GDP is 2x higher than in 2000
Why not use per capita real GDP (+25% since 2000)?
I’m thinking that if there were liquid prediction markets for amplifying ESCs, people could code bots to do exactly what John suggests and potentially make money. This suggests to me that there’s no principled difference between the two ideas, though I could be missing something (maybe you think the bot is unlikely to beat the market?)
Based on the quote from Jessica Taylor, it seems like the FDT agents are trying to maximize their long-term share of the population, rather than their absolute payoffs in a single generation? If I understand the model correctly, that means the FDT agents should try to maximize the ratio of FDT payoff : 9-bot payoff (to maximize the ratio of FDT:9-bot in the next generation). The algebra then shows that they should refuse to submit to 9-bots once the population of FDT agents gets high enough (Wolfram|Alpha link), without needing to drop the random encounters assumption.
It still seems like CDT agents would behave the same way given the same goals, though?
What’s the difference between John’s suggestion and amplifying ESCs with prediction markets? (not rhetorical)
I was somewhat confused by the discussion of LTFF grants being rejected by CEA; is there a public writeup of which grants were rejected?
In order to do this, the agent needs to be able to reason approximately about the results of their own computations, which is where logical uncertainty comes in
Why does being updateless require thinking through all possibilities in advance? Can you not make a general commitment to follow UDT, but wait until you actually face the decision problem to figure out which specific action UDT recommends taking?
Well, it’s been 8 years; how close are ML researchers to a “proto-AGI” with the capabilities listed? (embarassingly, I have no idea what the answer is)
Apparently an LW user did a series of interviews with AI researchers in 2011, some of which included a similar question. I know most LW users have probably seen this, but I only found it today and thought it was worth flagging here.
What are the competing explanations for high time preference?
A better way to phrase my confusion: How do we know the current time preference is higher than what we would see in a society that was genuinely at peace?
The competing explanations I was thinking of were along the lines of “we instinctively prefer having stuff now to having stuff later”
Yeah, I was implicitly assuming that initiating a successor agent would force Omega to update its predictions about the new agent (and put the $1m in the box). As you say, that’s actually not very relevant, because it’s a property of a specific decision problem rather than CDT or son-of-CDT.
(I apologize in advance if this is too far afield of the intended purpose of this post)
How does the claim that “group agents require membranes” interact with the widespread support for dramatically reducing or eliminating restrictions to immigration (“open borders” for short) within the EA/LW community? I can think of several possibilities, but I’m not sure which is true:
There actually isn’t much support for open borders
Open borders supporters believe that “group agents require membranes” is a reasonable generaliation, but borders are not a relevant kind of “membrane”, or nations are not “group agents” in the relevant sense
The people who support open borders generally aren’t the same people who are thinking about group agency at all
Open borders supporters have thought about group agency and concluded that “group agents require membranes” is not a reasonable generalization
Open borders supporters believe that there is no need for nations to have group agency
Something else I haven’t thought of
Context: I have an intuition that reduced/eliminated immigration restrictions reduce global coordination, and this post helped me crystallize it (if nations have less group agency, it’s harder to coordinate)
Why are you calling this a nitpick? IMO it’s a major problem with the post—I was very unhappy that no mention was made of this obvious problem with the reasoning presented.