To be blunt… our founder’s entire personality. (And current extreme burnout and evident depression.)
Also, I will not name names, but I know of at least one person who over DMs mentioned rendering their meat eating consistent with their other moral views by deciding that any entities without the cognition of a fully self-aware human have no moral rights, and was strongly considering whether it would be ethically acceptable to eat children and mentally disabled people. I found this disturbing enough to block them.
That’s not quite an example of the specific things I mentioned, but it is an example of the rationality subculture tending to veer away from what I suppose has to be called “common sense” or “consensus reality”. (Acausal reasoning and anthropics both also seem like examples of this. However “rational” they are, they are dangerous ideas that imo pose a cognitohazard.)
Actually, in the interests of full honesty, I have to give myself as an example. I didn’t know about the rationalist community until I was like 20, but throughout my teens I basically was a rationalist without knowing it—and also mentally ill and coping with emotional disturbances using a lot of narcissistic or even quasi-sociopathic thought styles. I firmly believed that I understood reality better than anyone else and that it was my responsibility to save the world, and I was willing in principle to do absolutely anything to achieve that goal, up to and including hurt many people.
It wasn’t until the past few years that I got comfortable feeling compassion for myself or others, and particularly after encountering the anarchist idea of the unity of means and ends and learning about intrinsic motivation, I’ve started to veer more towards some kind of hybrid of consequentialism, deontology, and virtue ethics rather than pure consequentialism. I think there is something inevitably destructive about any thinking style which places maximizing a linear definition of value above everything else, and I think an argument might exist for 1. humans not having coherent values representable as a utility function, not even in principle, and 2. all forms of utility maximization being intrinsically perverse, destructive, and immoral in the same way a paperclip maximizer is. But I haven’t gone through the process of working out how to argue that yet.
Also, I will not name names, but I know of at least one person who over DMs mentioned rendering their meat eating consistent with their other moral views by deciding that any entities without the cognition of a fully self-aware human have no moral rights, and was strongly considering whether it would be ethically acceptable to eat children and mentally disabled people. I found this disturbing enough to block them.
Yeah, I hear ya. I think what’s going on here is a difficult problem that I’ve been meaning to think and/or write about for some time: the balance between Taking Ideas Seriously and maintaining some sort of Memetic Immune System. It’s a difficult thing to navigate.
I think the person was wrong about eating children and probably leaned too strongly towards the Taking Ideas Seriously side of things—ie. should have a stronger anchor to a sort of “common sense prior”—but personally, I’m totally fine with that as long as 1) they are doing it in good faith and come from a place of wanting to figure out the truth, and 2) they aren’t actually causing harm in the real world… ie by eating children.
As a whole I feel like the rationalist community does a solid job navigating the tradeoff. If anything I actually think people don’t lean hard enough towards Taking Ideas Seriously. For example, not many people are signed up for cryonics.
the balance between Taking Ideas Seriously and maintaining some sort of Memetic Immune System. It’s a difficult thing to navigate.
It’s a false tradeoff, weird ideas taken seriously can be siloed in frames and hypotheticals. They only get to become beliefs (at any level of credence) or decision relevant if they grow up to make global sense. In particular, ITT asks to give voice to simulacra of all sorts of strange.
Re morality, there’s a potentially non-trivial chance that things are even worse than you think, in that two or more people disagreeing about morality or valence by starting with different assumptions can’t ever converge, even in the infinite limit of time and compute, because morality is not pointing to any reality, but is instead social realistiy.
I support MSRayne blocking the person since in that scenario, there would be no convergence at all.
Stylistically contemptible, skimpy on any useful details, contributes to norm of pressuring people into double binds that ultimately do more harm than good. I would prefer it if no one linked to or promoted “Taking Ideas Seriously”;
There are reasonable and coherent forms of moral skepticism in which the statement, “It is morally wrong to eat children and mentally disabled people,” is false or at least meaningless. The disgust reaction upon hearing the idea of eating children is better explained by the statement, “I don’t want to live in a society where children are eaten,” which is much more well-grounded in physical reality.
What is disturbing about the example is that this seems to be a person who believes that objective morality exists, but that it wouldn’t entail that eating children is wrong. This is indeed a red flag that something in the argument has gone seriously wrong.
My problem is more the lack of moral realism to begin with. I apparently need to work on a post about this. I am sick and tired of the lack of belief in objective morality around here, leading people to entertain such insane thoughts to begin with; needs some pushback.
This is a crux for me, and I conjecture roughly the opposite, that is moral/valence disagreements can’t converge to any truth, even in the infinite limit.
That is, there are no guarantees for moral reasoning converging to the truth, the way that say a bounded or unbounded Solomonoff inductor can in universes that have a simplicity bias.
In other words, the assumption behind your morality are functionally arbitrary, and there’s no deep justification for why you have the values you have.
Note: No one should change their values immediately based on this comment.
there’s no deep justification for why you have the values you have.
Um… evolution by natural selection? A very very short sketch:
1. most superintelligences likely to exist in the multiverse were created by civilizations of social organisms;
2. civilizations of social organisms tend to have moral systems rooted in generalizations of basic social instincts which worked in the ancestral environment, such as tit for tat defaulting to cooperation, and possibly geometric rationality;
3. some of those superintelligences are aligned and thus have value systems similar to those that tend to be evolved by civilizations of social organisms;
4. most are likely unaligned, but since unaligned superintelligences can have nearly any arbitrary utility function, those ones likely “cancel out”;
5. thus from an acausal trade standpoint, there is likely some one utility function to which the outcomes of trades between superintelligences across the multiverse tend, rooted in the most likely (according to how biological and memetic evolution by natural selection works) value systems arrived at by civilizations of social organisms prior to their local singularities, together with lots of small (because of mutually canceling out) wisps of interest in other random things from all the unaligned ASIs in the mix.
6. our own ASI, aligned or not, will (if it believes in multiverses and acausal things) probably notice this, run simulations to determine the most likely trajectories of such civilizations, and then align itself partly to the utility function of the multiverse meta-civilization in trade. That is: the existence of these facts results in a cosmic truth about what the correct utility function actually is, which can be determined by reasoning and approximated by getting more evidence, and which all sufficiently intelligent agents will converge on—which is to say, moral realism.
thus from an acausal trade standpoint, there is likely some one utility function to which the outcomes of trades between superintelligences across the multiverse tend, rooted in the most likely (according to how biological and memetic evolution by natural selection works) value systems arrived at by civilizations of social organisms prior to their local singularities, together with lots of small (because of mutually canceling out) wisps of interest in other random things from all the unaligned ASIs in the mix.
our own ASI, aligned or not, will (if it believes in multiverses and acausal things) probably notice this, run simulations to determine the most likely trajectories of such civilizations, and then align itself partly to the utility function of the multiverse meta-civilization in trade. That is: the existence of these facts results in a cosmic truth about what the correct utility function actually is, which can be determined by reasoning and approximated by getting more evidence, and which all sufficiently intelligent agents will converge on—which is to say, moral realism.
Now I get to the crux of why I disagree, and I note you’ve smuggled in the assumption that the multiverse constrains morality enough such that it’s sensible to talk about one moral truth or one true utility functions.
I think no multiverse that we actually live in constrains morality enough such that the conclusion of moral realism is correct, and that’s why I disagree with the idea of moral realism. Similarly, this means that acausal economies will essentially be random chaos with local bubbles of moral systems, and that the aligned and unaligned systems have equal weight in the multiverse economy, that is infinite weight.
And they all cancel each other out. Also, once we get to the stage that we join the acausal economy, there’s no reason to make an all encompassing economy across the entire multiverse, so there’s no reason for any acausal economies to form at all.
Specifically for alignment, the goal and maybe definition of alignment is essentially making the AI do what someone wants. Critically, the only constraint is that the AI must either have the same goals as the person having the AI, or it has different goals but those goals aren’t an impediment to the operator’s goals.
Note under this definition of alignment, it doesn’t comstrain the morality enough to make moral realism right, even after adding in instrumental goals.
Some notes on Geometric Rationality: I think there are some very useful notions from the geometric rationality sequence, like Thompson Sampling being better for exploration than it’s equivalent in arithmetic rationality as well as techniques to reduce the force of Pascal’s mugging, as he shows how exploration in the arithmetic rationality doesn’t converge to the truth with probability of 1, while a geometric rationality technique known as Thompson Sampling does know the truth asymptotically with probability 1. However, arithmetic rationality does have some properties that are better than geometric rationality, such as being invariant to potentially partisan efforts to shift the zero point, and arithmetic rationality plays better with unbounded or infinite utility functions, which are relevant given that unbounded or infinite preferences do exist IRL.
I will say though, I’m strongly upvoting this in karma and weakly downvoting in the disagree direction. I obviously have quite strong disagreements with MSRayne on this, but I’m impressed by both how much MSRayne managed to maintain a truthseeking attitude even on a very controversial and potentially mind killing topic like morality, and impressed that someone made the argument clear so that I could find why I didn’t agree with it. MSRayne, hats off to you for how well this conversation went.
The great thing is, this is ultimately an empirical question! Once we make an aligned ASI, we can run lots of simulations (carefully, to avoid inflicting suffering on innocent beings—philosophical zombie simulacra will likely be enough for this purpose) to get a sense of what the actual distribution of utility functions among ASIs in the multiverse might be like. “Moral science”...
I definitely want to say that there’s reason to believe at least some portions of the disagreement are testable, though I want curb enthusiasm by saying that we probably can’t resolve the disagreement in general, unless we can somehow either make a new universe with different physical constants or modify the physical constants of our universe.
Also, I suspect the condition below makes it significantly harder or flat out impossible to run experiments like this, at least without confounding the results and thereby making the experiment worthless.
(carefully, to avoid inflicting suffering on innocent beings—philosophical zombie simulacra will likely be enough for this purpose)
Do you have any favorite examples of this problem?
To be blunt… our founder’s entire personality. (And current extreme burnout and evident depression.)
Also, I will not name names, but I know of at least one person who over DMs mentioned rendering their meat eating consistent with their other moral views by deciding that any entities without the cognition of a fully self-aware human have no moral rights, and was strongly considering whether it would be ethically acceptable to eat children and mentally disabled people. I found this disturbing enough to block them.
That’s not quite an example of the specific things I mentioned, but it is an example of the rationality subculture tending to veer away from what I suppose has to be called “common sense” or “consensus reality”. (Acausal reasoning and anthropics both also seem like examples of this. However “rational” they are, they are dangerous ideas that imo pose a cognitohazard.)
Actually, in the interests of full honesty, I have to give myself as an example. I didn’t know about the rationalist community until I was like 20, but throughout my teens I basically was a rationalist without knowing it—and also mentally ill and coping with emotional disturbances using a lot of narcissistic or even quasi-sociopathic thought styles. I firmly believed that I understood reality better than anyone else and that it was my responsibility to save the world, and I was willing in principle to do absolutely anything to achieve that goal, up to and including hurt many people.
It wasn’t until the past few years that I got comfortable feeling compassion for myself or others, and particularly after encountering the anarchist idea of the unity of means and ends and learning about intrinsic motivation, I’ve started to veer more towards some kind of hybrid of consequentialism, deontology, and virtue ethics rather than pure consequentialism. I think there is something inevitably destructive about any thinking style which places maximizing a linear definition of value above everything else, and I think an argument might exist for 1. humans not having coherent values representable as a utility function, not even in principle, and 2. all forms of utility maximization being intrinsically perverse, destructive, and immoral in the same way a paperclip maximizer is. But I haven’t gone through the process of working out how to argue that yet.
Yeah, I hear ya. I think what’s going on here is a difficult problem that I’ve been meaning to think and/or write about for some time: the balance between Taking Ideas Seriously and maintaining some sort of Memetic Immune System. It’s a difficult thing to navigate.
I think the person was wrong about eating children and probably leaned too strongly towards the Taking Ideas Seriously side of things—ie. should have a stronger anchor to a sort of “common sense prior”—but personally, I’m totally fine with that as long as 1) they are doing it in good faith and come from a place of wanting to figure out the truth, and 2) they aren’t actually causing harm in the real world… ie by eating children.
As a whole I feel like the rationalist community does a solid job navigating the tradeoff. If anything I actually think people don’t lean hard enough towards Taking Ideas Seriously. For example, not many people are signed up for cryonics.
It’s a false tradeoff, weird ideas taken seriously can be siloed in frames and hypotheticals. They only get to become beliefs (at any level of credence) or decision relevant if they grow up to make global sense. In particular, ITT asks to give voice to simulacra of all sorts of strange.
Re morality, there’s a potentially non-trivial chance that things are even worse than you think, in that two or more people disagreeing about morality or valence by starting with different assumptions can’t ever converge, even in the infinite limit of time and compute, because morality is not pointing to any reality, but is instead social realistiy.
I support MSRayne blocking the person since in that scenario, there would be no convergence at all.
https://www.lesswrong.com/posts/QePFiEKZ4R2KnxMkW/posts-i-repent-of
There are reasonable and coherent forms of moral skepticism in which the statement, “It is morally wrong to eat children and mentally disabled people,” is false or at least meaningless. The disgust reaction upon hearing the idea of eating children is better explained by the statement, “I don’t want to live in a society where children are eaten,” which is much more well-grounded in physical reality.
What is disturbing about the example is that this seems to be a person who believes that objective morality exists, but that it wouldn’t entail that eating children is wrong. This is indeed a red flag that something in the argument has gone seriously wrong.
My problem is more the lack of moral realism to begin with. I apparently need to work on a post about this. I am sick and tired of the lack of belief in objective morality around here, leading people to entertain such insane thoughts to begin with; needs some pushback.
This is a crux for me, and I conjecture roughly the opposite, that is moral/valence disagreements can’t converge to any truth, even in the infinite limit.
That is, there are no guarantees for moral reasoning converging to the truth, the way that say a bounded or unbounded Solomonoff inductor can in universes that have a simplicity bias.
In other words, the assumption behind your morality are functionally arbitrary, and there’s no deep justification for why you have the values you have.
Note: No one should change their values immediately based on this comment.
Um… evolution by natural selection? A very very short sketch:
1. most superintelligences likely to exist in the multiverse were created by civilizations of social organisms;
2. civilizations of social organisms tend to have moral systems rooted in generalizations of basic social instincts which worked in the ancestral environment, such as tit for tat defaulting to cooperation, and possibly geometric rationality;
3. some of those superintelligences are aligned and thus have value systems similar to those that tend to be evolved by civilizations of social organisms;
4. most are likely unaligned, but since unaligned superintelligences can have nearly any arbitrary utility function, those ones likely “cancel out”;
5. thus from an acausal trade standpoint, there is likely some one utility function to which the outcomes of trades between superintelligences across the multiverse tend, rooted in the most likely (according to how biological and memetic evolution by natural selection works) value systems arrived at by civilizations of social organisms prior to their local singularities, together with lots of small (because of mutually canceling out) wisps of interest in other random things from all the unaligned ASIs in the mix.
6. our own ASI, aligned or not, will (if it believes in multiverses and acausal things) probably notice this, run simulations to determine the most likely trajectories of such civilizations, and then align itself partly to the utility function of the multiverse meta-civilization in trade. That is: the existence of these facts results in a cosmic truth about what the correct utility function actually is, which can be determined by reasoning and approximated by getting more evidence, and which all sufficiently intelligent agents will converge on—which is to say, moral realism.
Now I get to the crux of why I disagree, and I note you’ve smuggled in the assumption that the multiverse constrains morality enough such that it’s sensible to talk about one moral truth or one true utility functions.
I think no multiverse that we actually live in constrains morality enough such that the conclusion of moral realism is correct, and that’s why I disagree with the idea of moral realism. Similarly, this means that acausal economies will essentially be random chaos with local bubbles of moral systems, and that the aligned and unaligned systems have equal weight in the multiverse economy, that is infinite weight.
And they all cancel each other out. Also, once we get to the stage that we join the acausal economy, there’s no reason to make an all encompassing economy across the entire multiverse, so there’s no reason for any acausal economies to form at all.
Specifically for alignment, the goal and maybe definition of alignment is essentially making the AI do what someone wants. Critically, the only constraint is that the AI must either have the same goals as the person having the AI, or it has different goals but those goals aren’t an impediment to the operator’s goals.
Note under this definition of alignment, it doesn’t comstrain the morality enough to make moral realism right, even after adding in instrumental goals.
Some notes on Geometric Rationality: I think there are some very useful notions from the geometric rationality sequence, like Thompson Sampling being better for exploration than it’s equivalent in arithmetic rationality as well as techniques to reduce the force of Pascal’s mugging, as he shows how exploration in the arithmetic rationality doesn’t converge to the truth with probability of 1, while a geometric rationality technique known as Thompson Sampling does know the truth asymptotically with probability 1. However, arithmetic rationality does have some properties that are better than geometric rationality, such as being invariant to potentially partisan efforts to shift the zero point, and arithmetic rationality plays better with unbounded or infinite utility functions, which are relevant given that unbounded or infinite preferences do exist IRL.
I will say though, I’m strongly upvoting this in karma and weakly downvoting in the disagree direction. I obviously have quite strong disagreements with MSRayne on this, but I’m impressed by both how much MSRayne managed to maintain a truthseeking attitude even on a very controversial and potentially mind killing topic like morality, and impressed that someone made the argument clear so that I could find why I didn’t agree with it. MSRayne, hats off to you for how well this conversation went.
The great thing is, this is ultimately an empirical question! Once we make an aligned ASI, we can run lots of simulations (carefully, to avoid inflicting suffering on innocent beings—philosophical zombie simulacra will likely be enough for this purpose) to get a sense of what the actual distribution of utility functions among ASIs in the multiverse might be like. “Moral science”...
I definitely want to say that there’s reason to believe at least some portions of the disagreement are testable, though I want curb enthusiasm by saying that we probably can’t resolve the disagreement in general, unless we can somehow either make a new universe with different physical constants or modify the physical constants of our universe.
Also, I suspect the condition below makes it significantly harder or flat out impossible to run experiments like this, at least without confounding the results and thereby making the experiment worthless.