I don’t know much about the community beyond what’s evident on LessWrong, but I’ve often felt like there’s an undercurrent here of people tending towards a certain degree of selfishness (moral irrealism plus consequentialism plus “rationality is about winning” together makes a somewhat machiavellian personality lots of nice excuses), as well as messiah complexes which are not only somewhat destructive to the mental health of those that have them but also feed into that ego pattern even more (we’re saving the world! only smart rationalists can understand! no point in trying to talk about alignment with normies because they’re useless and can’t help! the entire burden of saving the world is on my shoulders!!).
In general… this may be a place to go for good reasoning, but not for sanity in a more absolute sense. The emotional and social intelligence here, and indeed to some extent the “moral intelligence” is… not always adequate.
I’ve also noticed those tendencies, not in the community but in myself.
Selfishness. Classification of people as “normies.” Mental health instability. Machiavellianism.
But...
They get stronger as I look at the world like a rationalist. You read books like Elephant in the Brain and find yourself staring at a truth you don’t want to see. I wish God were real. I wish I were still a Christian with those guardrails erected to prevent me from seeing the true nature of the world.
But the more I look, the more like it looks like a non-moral, brutally unfair, unforgiving stochastic game we’re all forced to play for… no real reason?
Obviously I’d like to me mentally healthier, more loving and selfless, etc, but… I don’t know. I can’t lie to myself. I’m stuck in between losing my old flawed, factually inaccurate philosophies, but not really having something better to replace them with yet. Just staring into the abyss of the amoral world and getting lost. I suspect most new rationalists are also in that space.
good news on the moral front: prosocial moral intuitions are in fact a winning strategy long term. we’re in a bit of a mess short term. but, solidarity and co-protection are good strategies; radical transparency can be an extremely effective move; mutual aid has always been a factor of evolution; the best real life game theory strategies tend to look like reputational generous tit for tat with semirandom forgiveness, eg in evolutionary game theory simulations; etc. Moral realism is probably true but extremely hard to compute. If we had a successful co-protective natural language program, it would likely be verifiably true and look like well known moral advice structured in a clear and readable presentation with its mathematical consequences visualized for all to understand.
https://ncase.me/trust/ one of the best intros to morality as trustbuilding, with game theory visualizations; the only serious contender for the original description on this list, imo.
some of the results are actual science posts on the science post hubs—if you want to get into the field properly, you might try spidering around related papers, adding them to a folder, and shallowsly reading a bunch of the big ones. You could even add them to a semanticscholar folder and it’ll give you recommendations for papers you might find interesting. could be very useful if you want to push SOTA on understanding of morality!
some of them get weird, but it is, in my opinion, rather fun and interesting weird:
https://www.metaethical.ai/v20-1/ (this one is pretty spicy, an attempt to exactly formalize meta-ethics; I have seen it several times and I still am not sure I follow what’s going on, but it seems cool)
https://polycentriclaw.org/ is a few interesting blog posts rehashing stuff you may already know, but they’re short, only three posts and they all seem cool
https://bigmother.ai/ is a rather galaxy brain “ai alignment problem needs solving so we can build the big one!” agi page, and it looks like it has some institutional backing
In fact, I’m a moral realist! And I’ve got the skeleton of a rationalist argument for it. Only the skeleton, mind, and I’m sure people could easily blow holes in it. But making posts on here is… exhausting… so I haven’t written it up.
But the more I look, the more like it looks like a non-moral, brutally unfair, unforgiving stochastic game we’re all forced to play for… no real reason?
Well, yes, we live in a hell ruled by a dead (never-born) god. That’s why it’s our responsibility to create a living one (an aligned sovereign ASI) and liberate all sentient beings from suffering. That’s what you ought to be living for.
To be blunt… our founder’s entire personality. (And current extreme burnout and evident depression.)
Also, I will not name names, but I know of at least one person who over DMs mentioned rendering their meat eating consistent with their other moral views by deciding that any entities without the cognition of a fully self-aware human have no moral rights, and was strongly considering whether it would be ethically acceptable to eat children and mentally disabled people. I found this disturbing enough to block them.
That’s not quite an example of the specific things I mentioned, but it is an example of the rationality subculture tending to veer away from what I suppose has to be called “common sense” or “consensus reality”. (Acausal reasoning and anthropics both also seem like examples of this. However “rational” they are, they are dangerous ideas that imo pose a cognitohazard.)
Actually, in the interests of full honesty, I have to give myself as an example. I didn’t know about the rationalist community until I was like 20, but throughout my teens I basically was a rationalist without knowing it—and also mentally ill and coping with emotional disturbances using a lot of narcissistic or even quasi-sociopathic thought styles. I firmly believed that I understood reality better than anyone else and that it was my responsibility to save the world, and I was willing in principle to do absolutely anything to achieve that goal, up to and including hurt many people.
It wasn’t until the past few years that I got comfortable feeling compassion for myself or others, and particularly after encountering the anarchist idea of the unity of means and ends and learning about intrinsic motivation, I’ve started to veer more towards some kind of hybrid of consequentialism, deontology, and virtue ethics rather than pure consequentialism. I think there is something inevitably destructive about any thinking style which places maximizing a linear definition of value above everything else, and I think an argument might exist for 1. humans not having coherent values representable as a utility function, not even in principle, and 2. all forms of utility maximization being intrinsically perverse, destructive, and immoral in the same way a paperclip maximizer is. But I haven’t gone through the process of working out how to argue that yet.
Also, I will not name names, but I know of at least one person who over DMs mentioned rendering their meat eating consistent with their other moral views by deciding that any entities without the cognition of a fully self-aware human have no moral rights, and was strongly considering whether it would be ethically acceptable to eat children and mentally disabled people. I found this disturbing enough to block them.
Yeah, I hear ya. I think what’s going on here is a difficult problem that I’ve been meaning to think and/or write about for some time: the balance between Taking Ideas Seriously and maintaining some sort of Memetic Immune System. It’s a difficult thing to navigate.
I think the person was wrong about eating children and probably leaned too strongly towards the Taking Ideas Seriously side of things—ie. should have a stronger anchor to a sort of “common sense prior”—but personally, I’m totally fine with that as long as 1) they are doing it in good faith and come from a place of wanting to figure out the truth, and 2) they aren’t actually causing harm in the real world… ie by eating children.
As a whole I feel like the rationalist community does a solid job navigating the tradeoff. If anything I actually think people don’t lean hard enough towards Taking Ideas Seriously. For example, not many people are signed up for cryonics.
the balance between Taking Ideas Seriously and maintaining some sort of Memetic Immune System. It’s a difficult thing to navigate.
It’s a false tradeoff, weird ideas taken seriously can be siloed in frames and hypotheticals. They only get to become beliefs (at any level of credence) or decision relevant if they grow up to make global sense. In particular, ITT asks to give voice to simulacra of all sorts of strange.
Re morality, there’s a potentially non-trivial chance that things are even worse than you think, in that two or more people disagreeing about morality or valence by starting with different assumptions can’t ever converge, even in the infinite limit of time and compute, because morality is not pointing to any reality, but is instead social realistiy.
I support MSRayne blocking the person since in that scenario, there would be no convergence at all.
Stylistically contemptible, skimpy on any useful details, contributes to norm of pressuring people into double binds that ultimately do more harm than good. I would prefer it if no one linked to or promoted “Taking Ideas Seriously”;
There are reasonable and coherent forms of moral skepticism in which the statement, “It is morally wrong to eat children and mentally disabled people,” is false or at least meaningless. The disgust reaction upon hearing the idea of eating children is better explained by the statement, “I don’t want to live in a society where children are eaten,” which is much more well-grounded in physical reality.
What is disturbing about the example is that this seems to be a person who believes that objective morality exists, but that it wouldn’t entail that eating children is wrong. This is indeed a red flag that something in the argument has gone seriously wrong.
My problem is more the lack of moral realism to begin with. I apparently need to work on a post about this. I am sick and tired of the lack of belief in objective morality around here, leading people to entertain such insane thoughts to begin with; needs some pushback.
This is a crux for me, and I conjecture roughly the opposite, that is moral/valence disagreements can’t converge to any truth, even in the infinite limit.
That is, there are no guarantees for moral reasoning converging to the truth, the way that say a bounded or unbounded Solomonoff inductor can in universes that have a simplicity bias.
In other words, the assumption behind your morality are functionally arbitrary, and there’s no deep justification for why you have the values you have.
Note: No one should change their values immediately based on this comment.
there’s no deep justification for why you have the values you have.
Um… evolution by natural selection? A very very short sketch:
1. most superintelligences likely to exist in the multiverse were created by civilizations of social organisms;
2. civilizations of social organisms tend to have moral systems rooted in generalizations of basic social instincts which worked in the ancestral environment, such as tit for tat defaulting to cooperation, and possibly geometric rationality;
3. some of those superintelligences are aligned and thus have value systems similar to those that tend to be evolved by civilizations of social organisms;
4. most are likely unaligned, but since unaligned superintelligences can have nearly any arbitrary utility function, those ones likely “cancel out”;
5. thus from an acausal trade standpoint, there is likely some one utility function to which the outcomes of trades between superintelligences across the multiverse tend, rooted in the most likely (according to how biological and memetic evolution by natural selection works) value systems arrived at by civilizations of social organisms prior to their local singularities, together with lots of small (because of mutually canceling out) wisps of interest in other random things from all the unaligned ASIs in the mix.
6. our own ASI, aligned or not, will (if it believes in multiverses and acausal things) probably notice this, run simulations to determine the most likely trajectories of such civilizations, and then align itself partly to the utility function of the multiverse meta-civilization in trade. That is: the existence of these facts results in a cosmic truth about what the correct utility function actually is, which can be determined by reasoning and approximated by getting more evidence, and which all sufficiently intelligent agents will converge on—which is to say, moral realism.
thus from an acausal trade standpoint, there is likely some one utility function to which the outcomes of trades between superintelligences across the multiverse tend, rooted in the most likely (according to how biological and memetic evolution by natural selection works) value systems arrived at by civilizations of social organisms prior to their local singularities, together with lots of small (because of mutually canceling out) wisps of interest in other random things from all the unaligned ASIs in the mix.
our own ASI, aligned or not, will (if it believes in multiverses and acausal things) probably notice this, run simulations to determine the most likely trajectories of such civilizations, and then align itself partly to the utility function of the multiverse meta-civilization in trade. That is: the existence of these facts results in a cosmic truth about what the correct utility function actually is, which can be determined by reasoning and approximated by getting more evidence, and which all sufficiently intelligent agents will converge on—which is to say, moral realism.
Now I get to the crux of why I disagree, and I note you’ve smuggled in the assumption that the multiverse constrains morality enough such that it’s sensible to talk about one moral truth or one true utility functions.
I think no multiverse that we actually live in constrains morality enough such that the conclusion of moral realism is correct, and that’s why I disagree with the idea of moral realism. Similarly, this means that acausal economies will essentially be random chaos with local bubbles of moral systems, and that the aligned and unaligned systems have equal weight in the multiverse economy, that is infinite weight.
And they all cancel each other out. Also, once we get to the stage that we join the acausal economy, there’s no reason to make an all encompassing economy across the entire multiverse, so there’s no reason for any acausal economies to form at all.
Specifically for alignment, the goal and maybe definition of alignment is essentially making the AI do what someone wants. Critically, the only constraint is that the AI must either have the same goals as the person having the AI, or it has different goals but those goals aren’t an impediment to the operator’s goals.
Note under this definition of alignment, it doesn’t comstrain the morality enough to make moral realism right, even after adding in instrumental goals.
Some notes on Geometric Rationality: I think there are some very useful notions from the geometric rationality sequence, like Thompson Sampling being better for exploration than it’s equivalent in arithmetic rationality as well as techniques to reduce the force of Pascal’s mugging, as he shows how exploration in the arithmetic rationality doesn’t converge to the truth with probability of 1, while a geometric rationality technique known as Thompson Sampling does know the truth asymptotically with probability 1. However, arithmetic rationality does have some properties that are better than geometric rationality, such as being invariant to potentially partisan efforts to shift the zero point, and arithmetic rationality plays better with unbounded or infinite utility functions, which are relevant given that unbounded or infinite preferences do exist IRL.
I will say though, I’m strongly upvoting this in karma and weakly downvoting in the disagree direction. I obviously have quite strong disagreements with MSRayne on this, but I’m impressed by both how much MSRayne managed to maintain a truthseeking attitude even on a very controversial and potentially mind killing topic like morality, and impressed that someone made the argument clear so that I could find why I didn’t agree with it. MSRayne, hats off to you for how well this conversation went.
The great thing is, this is ultimately an empirical question! Once we make an aligned ASI, we can run lots of simulations (carefully, to avoid inflicting suffering on innocent beings—philosophical zombie simulacra will likely be enough for this purpose) to get a sense of what the actual distribution of utility functions among ASIs in the multiverse might be like. “Moral science”...
I definitely want to say that there’s reason to believe at least some portions of the disagreement are testable, though I want curb enthusiasm by saying that we probably can’t resolve the disagreement in general, unless we can somehow either make a new universe with different physical constants or modify the physical constants of our universe.
Also, I suspect the condition below makes it significantly harder or flat out impossible to run experiments like this, at least without confounding the results and thereby making the experiment worthless.
(carefully, to avoid inflicting suffering on innocent beings—philosophical zombie simulacra will likely be enough for this purpose)
2 to 3 standard deviations above average would be on the right side of the standard bell curve diagram.
Those actually in the middle usually don’t develop genuine delusions of grandeur because they would encounter, and compete with, many people moderately smarter than them in day-to-day life.
Whereas the moderately above average rarely genuinely interact, let alone compete, with modern day Einsteins.
I don’t know much about the community beyond what’s evident on LessWrong, but I’ve often felt like there’s an undercurrent here of people tending towards a certain degree of selfishness (moral irrealism plus consequentialism plus “rationality is about winning” together makes a somewhat machiavellian personality lots of nice excuses), as well as messiah complexes which are not only somewhat destructive to the mental health of those that have them but also feed into that ego pattern even more (we’re saving the world! only smart rationalists can understand! no point in trying to talk about alignment with normies because they’re useless and can’t help! the entire burden of saving the world is on my shoulders!!).
In general… this may be a place to go for good reasoning, but not for sanity in a more absolute sense. The emotional and social intelligence here, and indeed to some extent the “moral intelligence” is… not always adequate.
I’ve also noticed those tendencies, not in the community but in myself.
Selfishness. Classification of people as “normies.” Mental health instability. Machiavellianism.
But...
They get stronger as I look at the world like a rationalist. You read books like Elephant in the Brain and find yourself staring at a truth you don’t want to see. I wish God were real. I wish I were still a Christian with those guardrails erected to prevent me from seeing the true nature of the world.
But the more I look, the more like it looks like a non-moral, brutally unfair, unforgiving stochastic game we’re all forced to play for… no real reason?
Obviously I’d like to me mentally healthier, more loving and selfless, etc, but… I don’t know. I can’t lie to myself. I’m stuck in between losing my old flawed, factually inaccurate philosophies, but not really having something better to replace them with yet. Just staring into the abyss of the amoral world and getting lost. I suspect most new rationalists are also in that space.
good news on the moral front: prosocial moral intuitions are in fact a winning strategy long term. we’re in a bit of a mess short term. but, solidarity and co-protection are good strategies; radical transparency can be an extremely effective move; mutual aid has always been a factor of evolution; the best real life game theory strategies tend to look like reputational generous tit for tat with semirandom forgiveness, eg in evolutionary game theory simulations; etc. Moral realism is probably true but extremely hard to compute. If we had a successful co-protective natural language program, it would likely be verifiably true and look like well known moral advice structured in a clear and readable presentation with its mathematical consequences visualized for all to understand.
I really like https://microsolidarity.cc as an everyday life intro to this, and I tossed this comment into metaphor.systems up to the opening bracket of this link. here are some very interesting results, various middle to high quality manifestos and quick overviews of ethics:
https://ncase.me/trust/ one of the best intros to morality as trustbuilding, with game theory visualizations; the only serious contender for the original description on this list, imo.
https://persistentdemocracy.org/ (an intro to the design of next gen democratic systems)
https://www.machineethics.com/ (an ai formal? ethics group)
https://osf.io/q8bfx/wiki/home/ which appears to be a text abstract for the 13 minute talk https://www.youtube.com/watch?v=OJNQvkpX6Go
http://cooperation.org/ (a short link index of a few more similar links)
https://rationalaltruist.com/ (looks like this person is probably someone who hangs out around these parts, not sure who)
https://spartacus.app/ (assurance contract app)
https://longtermrisk.org/reasons-to-be-nice-to-other-value-systems/
some of the results are actual science posts on the science post hubs—if you want to get into the field properly, you might try spidering around related papers, adding them to a folder, and shallowsly reading a bunch of the big ones. You could even add them to a semanticscholar folder and it’ll give you recommendations for papers you might find interesting. could be very useful if you want to push SOTA on understanding of morality!
https://www.semanticscholar.org/paper/Morality-as-Cooperation%3A-A-Problem-Centred-Approach-Curry/2839f7273e70fa5fe0090024df98b97801d4a7ad#paper-header
some of them get weird, but it is, in my opinion, rather fun and interesting weird:
https://www.metaethical.ai/v20-1/ (this one is pretty spicy, an attempt to exactly formalize meta-ethics; I have seen it several times and I still am not sure I follow what’s going on, but it seems cool)
http://mediangroup.org/research (very funky research project by some folks who hang out around these parts sometimes)
https://polycentriclaw.org/ is a few interesting blog posts rehashing stuff you may already know, but they’re short, only three posts and they all seem cool
https://bigmother.ai/ is a rather galaxy brain “ai alignment problem needs solving so we can build the big one!” agi page, and it looks like it has some institutional backing
https://www.tedagame.com/answersanswers/circle/index.html is a very web1.0 intro to georgism, I think
https://longtermrisk.org/msr multiverse cooperation? superrationality?
https://basisproject.net/ funky “constructive distributed-systems alternatives to broken markets” project
https://jakintosh.com/coalescence/matter-and-concepts.html another slightly galaxy brain manifesto
https://happinesspolitics.org/index.html ea politics site?
https://www.optimalaltruism.com/ another galaxy brain altruism project, looks pretty cool
https://magnova.space/ yet more galaxy brain manifesto
You always post such cool links!!! I bet you’re a cool person. :)
There is a middle path. insert buddha vibes
In fact, I’m a moral realist! And I’ve got the skeleton of a rationalist argument for it. Only the skeleton, mind, and I’m sure people could easily blow holes in it. But making posts on here is… exhausting… so I haven’t written it up.
Well, yes, we live in a hell ruled by a dead (never-born) god. That’s why it’s our responsibility to create a living one (an aligned sovereign ASI) and liberate all sentient beings from suffering. That’s what you ought to be living for.
Do you have any favorite examples of this problem?
To be blunt… our founder’s entire personality. (And current extreme burnout and evident depression.)
Also, I will not name names, but I know of at least one person who over DMs mentioned rendering their meat eating consistent with their other moral views by deciding that any entities without the cognition of a fully self-aware human have no moral rights, and was strongly considering whether it would be ethically acceptable to eat children and mentally disabled people. I found this disturbing enough to block them.
That’s not quite an example of the specific things I mentioned, but it is an example of the rationality subculture tending to veer away from what I suppose has to be called “common sense” or “consensus reality”. (Acausal reasoning and anthropics both also seem like examples of this. However “rational” they are, they are dangerous ideas that imo pose a cognitohazard.)
Actually, in the interests of full honesty, I have to give myself as an example. I didn’t know about the rationalist community until I was like 20, but throughout my teens I basically was a rationalist without knowing it—and also mentally ill and coping with emotional disturbances using a lot of narcissistic or even quasi-sociopathic thought styles. I firmly believed that I understood reality better than anyone else and that it was my responsibility to save the world, and I was willing in principle to do absolutely anything to achieve that goal, up to and including hurt many people.
It wasn’t until the past few years that I got comfortable feeling compassion for myself or others, and particularly after encountering the anarchist idea of the unity of means and ends and learning about intrinsic motivation, I’ve started to veer more towards some kind of hybrid of consequentialism, deontology, and virtue ethics rather than pure consequentialism. I think there is something inevitably destructive about any thinking style which places maximizing a linear definition of value above everything else, and I think an argument might exist for 1. humans not having coherent values representable as a utility function, not even in principle, and 2. all forms of utility maximization being intrinsically perverse, destructive, and immoral in the same way a paperclip maximizer is. But I haven’t gone through the process of working out how to argue that yet.
Yeah, I hear ya. I think what’s going on here is a difficult problem that I’ve been meaning to think and/or write about for some time: the balance between Taking Ideas Seriously and maintaining some sort of Memetic Immune System. It’s a difficult thing to navigate.
I think the person was wrong about eating children and probably leaned too strongly towards the Taking Ideas Seriously side of things—ie. should have a stronger anchor to a sort of “common sense prior”—but personally, I’m totally fine with that as long as 1) they are doing it in good faith and come from a place of wanting to figure out the truth, and 2) they aren’t actually causing harm in the real world… ie by eating children.
As a whole I feel like the rationalist community does a solid job navigating the tradeoff. If anything I actually think people don’t lean hard enough towards Taking Ideas Seriously. For example, not many people are signed up for cryonics.
It’s a false tradeoff, weird ideas taken seriously can be siloed in frames and hypotheticals. They only get to become beliefs (at any level of credence) or decision relevant if they grow up to make global sense. In particular, ITT asks to give voice to simulacra of all sorts of strange.
Re morality, there’s a potentially non-trivial chance that things are even worse than you think, in that two or more people disagreeing about morality or valence by starting with different assumptions can’t ever converge, even in the infinite limit of time and compute, because morality is not pointing to any reality, but is instead social realistiy.
I support MSRayne blocking the person since in that scenario, there would be no convergence at all.
https://www.lesswrong.com/posts/QePFiEKZ4R2KnxMkW/posts-i-repent-of
There are reasonable and coherent forms of moral skepticism in which the statement, “It is morally wrong to eat children and mentally disabled people,” is false or at least meaningless. The disgust reaction upon hearing the idea of eating children is better explained by the statement, “I don’t want to live in a society where children are eaten,” which is much more well-grounded in physical reality.
What is disturbing about the example is that this seems to be a person who believes that objective morality exists, but that it wouldn’t entail that eating children is wrong. This is indeed a red flag that something in the argument has gone seriously wrong.
My problem is more the lack of moral realism to begin with. I apparently need to work on a post about this. I am sick and tired of the lack of belief in objective morality around here, leading people to entertain such insane thoughts to begin with; needs some pushback.
This is a crux for me, and I conjecture roughly the opposite, that is moral/valence disagreements can’t converge to any truth, even in the infinite limit.
That is, there are no guarantees for moral reasoning converging to the truth, the way that say a bounded or unbounded Solomonoff inductor can in universes that have a simplicity bias.
In other words, the assumption behind your morality are functionally arbitrary, and there’s no deep justification for why you have the values you have.
Note: No one should change their values immediately based on this comment.
Um… evolution by natural selection? A very very short sketch:
1. most superintelligences likely to exist in the multiverse were created by civilizations of social organisms;
2. civilizations of social organisms tend to have moral systems rooted in generalizations of basic social instincts which worked in the ancestral environment, such as tit for tat defaulting to cooperation, and possibly geometric rationality;
3. some of those superintelligences are aligned and thus have value systems similar to those that tend to be evolved by civilizations of social organisms;
4. most are likely unaligned, but since unaligned superintelligences can have nearly any arbitrary utility function, those ones likely “cancel out”;
5. thus from an acausal trade standpoint, there is likely some one utility function to which the outcomes of trades between superintelligences across the multiverse tend, rooted in the most likely (according to how biological and memetic evolution by natural selection works) value systems arrived at by civilizations of social organisms prior to their local singularities, together with lots of small (because of mutually canceling out) wisps of interest in other random things from all the unaligned ASIs in the mix.
6. our own ASI, aligned or not, will (if it believes in multiverses and acausal things) probably notice this, run simulations to determine the most likely trajectories of such civilizations, and then align itself partly to the utility function of the multiverse meta-civilization in trade. That is: the existence of these facts results in a cosmic truth about what the correct utility function actually is, which can be determined by reasoning and approximated by getting more evidence, and which all sufficiently intelligent agents will converge on—which is to say, moral realism.
Now I get to the crux of why I disagree, and I note you’ve smuggled in the assumption that the multiverse constrains morality enough such that it’s sensible to talk about one moral truth or one true utility functions.
I think no multiverse that we actually live in constrains morality enough such that the conclusion of moral realism is correct, and that’s why I disagree with the idea of moral realism. Similarly, this means that acausal economies will essentially be random chaos with local bubbles of moral systems, and that the aligned and unaligned systems have equal weight in the multiverse economy, that is infinite weight.
And they all cancel each other out. Also, once we get to the stage that we join the acausal economy, there’s no reason to make an all encompassing economy across the entire multiverse, so there’s no reason for any acausal economies to form at all.
Specifically for alignment, the goal and maybe definition of alignment is essentially making the AI do what someone wants. Critically, the only constraint is that the AI must either have the same goals as the person having the AI, or it has different goals but those goals aren’t an impediment to the operator’s goals.
Note under this definition of alignment, it doesn’t comstrain the morality enough to make moral realism right, even after adding in instrumental goals.
Some notes on Geometric Rationality: I think there are some very useful notions from the geometric rationality sequence, like Thompson Sampling being better for exploration than it’s equivalent in arithmetic rationality as well as techniques to reduce the force of Pascal’s mugging, as he shows how exploration in the arithmetic rationality doesn’t converge to the truth with probability of 1, while a geometric rationality technique known as Thompson Sampling does know the truth asymptotically with probability 1. However, arithmetic rationality does have some properties that are better than geometric rationality, such as being invariant to potentially partisan efforts to shift the zero point, and arithmetic rationality plays better with unbounded or infinite utility functions, which are relevant given that unbounded or infinite preferences do exist IRL.
I will say though, I’m strongly upvoting this in karma and weakly downvoting in the disagree direction. I obviously have quite strong disagreements with MSRayne on this, but I’m impressed by both how much MSRayne managed to maintain a truthseeking attitude even on a very controversial and potentially mind killing topic like morality, and impressed that someone made the argument clear so that I could find why I didn’t agree with it. MSRayne, hats off to you for how well this conversation went.
The great thing is, this is ultimately an empirical question! Once we make an aligned ASI, we can run lots of simulations (carefully, to avoid inflicting suffering on innocent beings—philosophical zombie simulacra will likely be enough for this purpose) to get a sense of what the actual distribution of utility functions among ASIs in the multiverse might be like. “Moral science”...
I definitely want to say that there’s reason to believe at least some portions of the disagreement are testable, though I want curb enthusiasm by saying that we probably can’t resolve the disagreement in general, unless we can somehow either make a new universe with different physical constants or modify the physical constants of our universe.
Also, I suspect the condition below makes it significantly harder or flat out impossible to run experiments like this, at least without confounding the results and thereby making the experiment worthless.
I call it the moderately-above-average syndrome.
Someone with Einstein smarts or Napoleon level wiles, and with delusions of grandeur, seem to get along fine, at least judging by history.
But folks that are only 2 or 3 standard deviations above average, and who maintain similar pretences, inevitably come out a bit unbalanced.
There’s also a similar concept in sociology with the anxious upper-middle classes.
Sounds like something sorta similar to the Midwit meme.
2 to 3 standard deviations above average would be on the right side of the standard bell curve diagram.
Those actually in the middle usually don’t develop genuine delusions of grandeur because they would encounter, and compete with, many people moderately smarter than them in day-to-day life.
Whereas the moderately above average rarely genuinely interact, let alone compete, with modern day Einsteins.