Hi everyone, I’m fairly new to the community, though I’ve been lurking on and off for a few years, and I would like to hear the opinions on a key question I am unsure about.
What is the ultimate goal of the rationalist enterprise?
I understand there are clear goals to establish ‘Friendly AI’, to realize intelligence ‘upgrading’, if achievable, life extension, and so on. But what is unclear to me is what comes next in the ideal case where all these goals have been achieved, and to what ultimate end.
For example,
I’ve encountered discussions about eudaimonia scenarios (private galaxies, etc.), though I’m not sure how seriously to take those, as surely the possibilities of the co-moving light cone that is within our capacity to inhabit are exhaustible in finite time, especially if all these designs reach their ultimate fruition?
Well that is an article that, although interesting, seems to miss a key factor in presenting the eudaemonia scenario (‘maximizing fun’). Because it does not define ‘fun’. e.g. a paperclip maximizer would consider more paperclips brought into existence as more ‘fun’.
And we know from game theory that when there is more than 1 player in any game… the inter-player dynamics ultimately decide their actions as rational agents.
So I cannot see how an individual’s aspirations (‘fun’) are relevant to determining a future state without considering the total sum of all aspirations (sum of all ‘funs’) as well. Unless there is only 1 conscious entity remaining, which to be fair is not out of the realm of possibility in some very distant future.
Also, this section of the article:
Fun Theory is also the fully general reply to religious theodicy (attempts to justify why God permits evil). Our present world has flaws even from the standpoint of such eudaimonic considerations as freedom, personal responsibility, and self-reliance. Fun Theory tries to describe the dimensions along which a benevolently designed world can and should be optimized, and our present world is clearly not the result of such optimization.
Is not convincing, because it does not actually refute Leibniz’s old argument, that only an omniscient and omnipresent being could ‘clearly’ see whether the world is benevolently designed or not, whether it has been optimized, along all dimensions, to the greatest extent possible or not, even whether it even has any flaws on a total net basis or not.
And I’ve not yet seen a convincing disproof of those arguments.
Now of course I personally am leery of believing those claims to be true, but then I also cannot prove with 100% certainty that they are false. And the ‘Fun Theory’ article is certainly presented as if there was such proof.
What’s the rational basis for preferring all mass-energy consuming grey goo created by humans over all mass-energy consuming grey goo created by a paperclip optimizer? The only possible ultimate end in both scenarios is heat death anyways.
If no one’s goals can be definitely proven to be better than anyone else’s goals, then it doesnt seem like we can automatically conclude the majority of present or future humans, or our descendants, will prioritize maximizing fun, happiness, etc.
If some want to pursue that then fine, if others want to pursue different goals, even ones that are deleterious to overall fun, happiness, etc., then there doesn’t seem to be a credible argument to dissuade them?
Let’s think about it another way. Consider the thought experiment where a single normal cell is removed from the body of any randomly selected human. Clearly they would still be human.
If you keep on removing normal cells though eventually they would die. And if you keep on plucking away cells eventually the entire body would be gone and only cancerous cells would be left, i.e. only a ‘paperclip optimizer’ would remain from the original human, albeit inefficient and parasitic ‘paperclips’ that need a organic host.
(Due to the fact that everyone has some small number of cancerous cells at any given time that are taken care of by regular processes)
At what point does the human stop being ‘human’ and starts being a lump of flesh? And at what point does the lump of flesh become a latent ‘paperclip optimizer’?
Without a sharp cutoff, which I don’t think there is, there will inevitably be inbetween cases where your proposed methods cannot be applied consistently.
The trouble is if we, or the decision makers of the future, accept even one idea that is not internally consistent then it hardly seems like anyone will be able to refrain from accepting other ideas that are internally contradictory too. Nor will everyone err in the same way. There is no rational basis to accept one or another as a contradiction can imply anything at all, as we know from basic logic.
Then the end result will appear quite like monkey tribes fighting each other, agitating against each and all based on which inconsistencies they accept or not. Regardless of what they call each other, humans, aliens, AI, machines, organism, etc…
It does seem like alignment for all intents and purposes is impossible. Creating an AI truly beyond us then is really creating future, hopefully doting, parents to live under.
But what is unclear to me is what comes next in the ideal case where all these goals have been achieved
You live happily ever after.
I’ve encountered discussions about eudaimonia scenarios (private galaxies, etc.), though I’m not sure how seriously to take those, as surely the possibilities of the co-moving light cone that is within our capacity to inhabit are exhaustible in finite time, especially if all these designs reach their ultimate fruition?
Hi everyone, I’m fairly new to the community, though I’ve been lurking on and off for a few years, and I would like to hear the opinions on a key question I am unsure about.
What is the ultimate goal of the rationalist enterprise?
I understand there are clear goals to establish ‘Friendly AI’, to realize intelligence ‘upgrading’, if achievable, life extension, and so on. But what is unclear to me is what comes next in the ideal case where all these goals have been achieved, and to what ultimate end.
For example,
I’ve encountered discussions about eudaimonia scenarios (private galaxies, etc.), though I’m not sure how seriously to take those, as surely the possibilities of the co-moving light cone that is within our capacity to inhabit are exhaustible in finite time, especially if all these designs reach their ultimate fruition?
There are no shared ultimate goals of the rationalist enterprise. Different rationalist have different goals.
I think the idea is to have as much fun as possible, and keep doing science (which might expand our opportunities to have fun).
In the very long term, if the universe runs out of energy and nothing in the new science allows us to overcome this issue, then, sadly, we die.
Well that is an article that, although interesting, seems to miss a key factor in presenting the eudaemonia scenario (‘maximizing fun’). Because it does not define ‘fun’. e.g. a paperclip maximizer would consider more paperclips brought into existence as more ‘fun’.
And we know from game theory that when there is more than 1 player in any game… the inter-player dynamics ultimately decide their actions as rational agents.
So I cannot see how an individual’s aspirations (‘fun’) are relevant to determining a future state without considering the total sum of all aspirations (sum of all ‘funs’) as well. Unless there is only 1 conscious entity remaining, which to be fair is not out of the realm of possibility in some very distant future.
Also, this section of the article:
Is not convincing, because it does not actually refute Leibniz’s old argument, that only an omniscient and omnipresent being could ‘clearly’ see whether the world is benevolently designed or not, whether it has been optimized, along all dimensions, to the greatest extent possible or not, even whether it even has any flaws on a total net basis or not.
And I’ve not yet seen a convincing disproof of those arguments.
Now of course I personally am leery of believing those claims to be true, but then I also cannot prove with 100% certainty that they are false. And the ‘Fun Theory’ article is certainly presented as if there was such proof.
So why must we prevent paperclip optimizers from bringing about their own ‘fun’?
What’s the rational basis for preferring all mass-energy consuming grey goo created by humans over all mass-energy consuming grey goo created by a paperclip optimizer? The only possible ultimate end in both scenarios is heat death anyways.
If no one’s goals can be definitely proven to be better than anyone else’s goals, then it doesnt seem like we can automatically conclude the majority of present or future humans, or our descendants, will prioritize maximizing fun, happiness, etc.
If some want to pursue that then fine, if others want to pursue different goals, even ones that are deleterious to overall fun, happiness, etc., then there doesn’t seem to be a credible argument to dissuade them?
Those appear to be examples of arguments from consequences, a logical fallacy. How could similar reasoning be derived from axioms, if at all?
Let’s think about it another way. Consider the thought experiment where a single normal cell is removed from the body of any randomly selected human. Clearly they would still be human.
If you keep on removing normal cells though eventually they would die. And if you keep on plucking away cells eventually the entire body would be gone and only cancerous cells would be left, i.e. only a ‘paperclip optimizer’ would remain from the original human, albeit inefficient and parasitic ‘paperclips’ that need a organic host.
(Due to the fact that everyone has some small number of cancerous cells at any given time that are taken care of by regular processes)
At what point does the human stop being ‘human’ and starts being a lump of flesh? And at what point does the lump of flesh become a latent ‘paperclip optimizer’?
Without a sharp cutoff, which I don’t think there is, there will inevitably be inbetween cases where your proposed methods cannot be applied consistently.
The trouble is if we, or the decision makers of the future, accept even one idea that is not internally consistent then it hardly seems like anyone will be able to refrain from accepting other ideas that are internally contradictory too. Nor will everyone err in the same way. There is no rational basis to accept one or another as a contradiction can imply anything at all, as we know from basic logic.
Then the end result will appear quite like monkey tribes fighting each other, agitating against each and all based on which inconsistencies they accept or not. Regardless of what they call each other, humans, aliens, AI, machines, organism, etc…
It does seem like alignment for all intents and purposes is impossible. Creating an AI truly beyond us then is really creating future, hopefully doting, parents to live under.
You live happily ever after.
Where is the contradiction here?
What occurs after the finite period of expansion and enjoyment ends?
Death, presumably.
(the finiteness is actually far from certain, but that’s neither here nor there)