if AGI goes well, economics won’t matter much. helping slow down AI progress is probably the best way to purchase shares of the LDT utility function handshake: in winning timelines, whoever did end up solving alignment will have done that thanks to having the time to pay the alignment tax on their research.
My best guess as to what you mean by “economics won’t matter much” is that (absent catastrophe) AGI will usher in an age of abundance. But abundance can’t be unlimited, and even if you’re satisfied with limited abundance, that era won’t last forever.
It’s critical to enter the post-AGI era with either wealth or wealthy connections, because labor will no longer be available as an opportunity to bootstrap your personal net worth.
what i mean is that despite the foundamental scarcity of negentropy-until-heat-death, aligned superintelligent AI will be able to better allocate resources than any human-designed system. i expect that people will still be able to “play at money” if they want, but pre-singularity allocations of wealth/connections are unlikely to be relevant what maximizes nice-things utility.
it’s entirely useless to enter the post-AGI era with either wealth or wealthy connections. in fact, it’s a waste to not have spent it on increasing-the-probability-that-AGI-goes-well while money was still meaningful.
aligned superintelligent AI will be able to better allocate resources than any human-designed system.
Sure, but allocate to what end? Somebody gets to decide the goal, and you get more say if you have money than if you don’t. Same as in all of history, really.
As a concrete example, if you want to do something with the GPT-4 API, it costs money. When someday there’s an AGI API, it’ll cost money too.
the GPT-4 API has not taken over the world. there is a singular-point-in-time at which some AI will take over everything with a particular utility function and, if AI goes well, create utopia.
Sure, but allocate to what end?
whatever utility function it’s been launched with. which is particularly representative of who currently has money. it’s not somebody who decides resource-allocation-in-the-post-singularity-future, it’s some-utility-function, and the utility function is picked by whoever built the thing, and they’re unlikely to type a utility function saying “people should have control over the future proportional to their current allocation of wealth”. they’re a lot more likely to type something like “make a world that people would describe as good under CEV”.
It’s true that if the transition to the AGI era involves some sort of 1917-Russian-revolution-esque teardown of existing forms of social organization to impose a utopian ideology, pre-existing property isn’t going to help much.
Unless you’re all-in on such a scenario, though, it’s still worth preparing for other scenarios too. And I don’t think it makes sense to be all-in on a scenario that many people (including me) would consider to be a bad outcome.
LDT does not say that if you invoke the LDT god while doing something nice for someone, they ought to compensate you for it later. Tamsin Leake does not have the kind of info on who/what will control the lightcone that would allow them to cooperate in PDs.
the point i was trying to make is that if you expect someone to reliably implement LDT, then you can expect to be rewarded for help them (actually helping them) solve alignment because they’d be the kind of agent who, if they solve alignment is solved, will retroactively allocate some of their utility function handshake to you.
LDT-ers reliably one-box, and LDT-ers reliably retroactively-reward people who help them, including in ways that they can’t percieve before alignment is solved.
it’s not about “doing something nice”, it’s about LDT agents who end do well, retroactively repaying the agents who helped them get there, because being the kind of agent who reliably does that causes them to more often do well.
The point i was trying to make is that if you expect someone to reliably implement LDT, then you can expect to be rewarded for help them because they’d be the kind of agent who, if they solve alignment is solved, will retroactively allocate some of their utility function handshake to you.
Yes, and the point I am making is that this is not what LDT is or how it works. LDT agents perform prudentbot, not fairbot. An AGI will only reward you with cooperation if you conditionally cooperate, on something you’re unable to “condition” on because it would mean looking at the AGI’s code and analyzing it beyond what anyone is capable of at present.
i have read that post before and i do not think that it applies here? can you please expand on your disagreement?
Tamsin Leake does not have the kind of info on who/what will control the lightcone that would allow them to cooperate in PDs.
you don’t need to know this to probabilistically-help whoever will control the lightcone, right? if you take actions that help them-whoever-they-are, then you’re getting some of that share from them-whoever-they-are. (i think?)
you don’t need to know this to probabilistically-help whoever will control the lightcone, right? if you take actions that help them-whoever-they-are, then you’re getting some of that share from them-whoever-they-are. (i think?)
My point is not that you can’t affect the outcome of the future. That may also be impossible, but regardless, any intervention you make will be independent of whether or not the person you’re rewarding gives you a share of the lightcone. You can’t actually tell in advance whether or not that AI/person is going to give you that share, in the sense that would incentivize someone to give it to you after they’ve already seized control.
you don’t think there are humans whom i can expect to reliably reward-me-as-per-LDT after-the-fact? it doesn’t have to be a certainty, i can merely have some confidence that some person will give me that share, and weigh the action based on that confidence.
That might happen, but they wouldn’t be doing it because they’re maximizing their utility via acausal trade, they’d be doing it because they value reciprocity.
why wouldn’t it be because they’re maximizing their utility via acausal trade?
do you also think people who don’t-intrinsically-value-reciprocity are doomed to never get picked up by rational agents in parfit’s hitchhiker? or doomed to two-box in newcomb?
to take an example: i would expect that even if he didn’t value reciprocity at all, yudkowsky would reliably cooperate as the hitchhiker in parfit’s hitchhiker, or one-box in newcomb, or retroactively-give-utility-function-shares-to-people-who-helped-if-he-grabbed-the-lightcone. he seems like the-kind-of-person-who-tries-to-reliably-implement-LDT.
if AGI goes well, economics won’t matter much. helping slow down AI progress is probably the best way to purchase shares of the LDT utility function handshake: in winning timelines, whoever did end up solving alignment will have done that thanks to having the time to pay the alignment tax on their research.
My best guess as to what you mean by “economics won’t matter much” is that (absent catastrophe) AGI will usher in an age of abundance. But abundance can’t be unlimited, and even if you’re satisfied with limited abundance, that era won’t last forever.
It’s critical to enter the post-AGI era with either wealth or wealthy connections, because labor will no longer be available as an opportunity to bootstrap your personal net worth.
what i mean is that despite the foundamental scarcity of negentropy-until-heat-death, aligned superintelligent AI will be able to better allocate resources than any human-designed system. i expect that people will still be able to “play at money” if they want, but pre-singularity allocations of wealth/connections are unlikely to be relevant what maximizes nice-things utility.
it’s entirely useless to enter the post-AGI era with either wealth or wealthy connections. in fact, it’s a waste to not have spent it on increasing-the-probability-that-AGI-goes-well while money was still meaningful.
Sure, but allocate to what end? Somebody gets to decide the goal, and you get more say if you have money than if you don’t. Same as in all of history, really.
As a concrete example, if you want to do something with the GPT-4 API, it costs money. When someday there’s an AGI API, it’ll cost money too.
the GPT-4 API has not taken over the world. there is a singular-point-in-time at which some AI will take over everything with a particular utility function and, if AI goes well, create utopia.
whatever utility function it’s been launched with. which is particularly representative of who currently has money. it’s not somebody who decides resource-allocation-in-the-post-singularity-future, it’s some-utility-function, and the utility function is picked by whoever built the thing, and they’re unlikely to type a utility function saying “people should have control over the future proportional to their current allocation of wealth”. they’re a lot more likely to type something like “make a world that people would describe as good under CEV”.
It’s true that if the transition to the AGI era involves some sort of 1917-Russian-revolution-esque teardown of existing forms of social organization to impose a utopian ideology, pre-existing property isn’t going to help much.
Unless you’re all-in on such a scenario, though, it’s still worth preparing for other scenarios too. And I don’t think it makes sense to be all-in on a scenario that many people (including me) would consider to be a bad outcome.
That’s not how LDT works
Isn’t it? It doesn’t seem clearly ruled out by my understanding of LDT(but not certain to happen either)
LDT does not say that if you invoke the LDT god while doing something nice for someone, they ought to compensate you for it later. Tamsin Leake does not have the kind of info on who/what will control the lightcone that would allow them to cooperate in PDs.
the point i was trying to make is that if you expect someone to reliably implement LDT, then you can expect to be rewarded for help them (actually helping them) solve alignment because they’d be the kind of agent who, if they solve alignment is solved, will retroactively allocate some of their utility function handshake to you.
LDT-ers reliably one-box, and LDT-ers reliably retroactively-reward people who help them, including in ways that they can’t percieve before alignment is solved.
it’s not about “doing something nice”, it’s about LDT agents who end do well, retroactively repaying the agents who helped them get there, because being the kind of agent who reliably does that causes them to more often do well.
Yes, and the point I am making is that this is not what LDT is or how it works. LDT agents perform prudentbot, not fairbot. An AGI will only reward you with cooperation if you conditionally cooperate, on something you’re unable to “condition” on because it would mean looking at the AGI’s code and analyzing it beyond what anyone is capable of at present.
i have read that post before and i do not think that it applies here? can you please expand on your disagreement?
you don’t need to know this to probabilistically-help whoever will control the lightcone, right? if you take actions that help them-whoever-they-are, then you’re getting some of that share from them-whoever-they-are. (i think?)
My point is not that you can’t affect the outcome of the future. That may also be impossible, but regardless, any intervention you make will be independent of whether or not the person you’re rewarding gives you a share of the lightcone. You can’t actually tell in advance whether or not that AI/person is going to give you that share, in the sense that would incentivize someone to give it to you after they’ve already seized control.
you don’t think there are humans whom i can expect to reliably reward-me-as-per-LDT after-the-fact? it doesn’t have to be a certainty, i can merely have some confidence that some person will give me that share, and weigh the action based on that confidence.
That might happen, but they wouldn’t be doing it because they’re maximizing their utility via acausal trade, they’d be doing it because they value reciprocity.
why wouldn’t it be because they’re maximizing their utility via acausal trade?
do you also think people who don’t-intrinsically-value-reciprocity are doomed to never get picked up by rational agents in parfit’s hitchhiker? or doomed to two-box in newcomb?
to take an example: i would expect that even if he didn’t value reciprocity at all, yudkowsky would reliably cooperate as the hitchhiker in parfit’s hitchhiker, or one-box in newcomb, or retroactively-give-utility-function-shares-to-people-who-helped-if-he-grabbed-the-lightcone. he seems like the-kind-of-person-who-tries-to-reliably-implement-LDT.