I think that an important underlying difference of perspective here is that the Less Wrong memes tend to automatically think of all AGIs as essentially computer programs whereas Goertzel-like memes tend to automatically think of at least some AGIs as non-negligibly essentially person-like. I think this is at least partially because the Less Wrong memes want to write an FAI that is essentially some machine learning algorithms plus a universal prior on top of sound decision theory whereas the Goertzel-like memes want to write an FAI that is essentially roughly half progam-like and half person-like. Less Wrong memes think that person AIs won’t be sufficiently person-like but they sort of tend to assume that conclusion rather than argue for it, which causes memes that aren’t familiar with Less Wrong memes to wonder why Less Wrong memes are so incredibly confident that all AIs will necessarily act like autistic OCD people without any possibility at all of acting like normal reasonable people. From that perspective the Goertzel-like memes look justified in being rather skeptical of Less Wrong memes. After all, it is easy to imagine a gradation between AIXI and whole brain emulations. Goertzel-like memes wish to create an AI somewhere between those two points, Less Wrong memes wish to create an AI that’s even more AIXI-like than AIXI is (in the sense of being more formally and theoretically well-founded than AIXI is). It’s important that each look at the specific kinds of AI that the other has in mind and start the exchange from there.
We don’t know if AIXI-approximating AIs would even be intelligent; how then can we be so confident that AIXI is a normative model and a definition of intelligence? This and other intuitions are likely underlying Goertzel’s cautious epistemic state, and LessWrong/SingInst truly hasn’t addressed issues like this. We don’t know what it takes to build AGI, we don’t know if intelligence runs on Bayes structure. Modern decision theory indicates that Eliezer was wrong, that Bayes structure isn’t fundamental to agentic optimization, that it only applies in certain cases, that Bayesian information theoretic models of cognition might not capture the special sauce of intelligence. What is fundamental? We don’t know! In the meantime we should be careful about drawing conclusions based on the assumed fundamental-ness of mathematical models which may or may not ultimately be accurate models, may or may not actually let you build literalistic self-improving AIs of the sort that LessWrong likes to speculate about.
I have no idea what your second paragraph is about—“modern decision theory” is not a very specific citation. If there is research concluding that probability theory only applies to certain special cases of optimization, it would be awesome if you could make a top-level post explaining it to us!
There have already been many top-level posts, but you’re right that I should have linked to them. Here is the LessWrong Wiki hub, here is a post by Wei Dai that cuts straight to the point.
Less Wrong memes think that person AIs won’t be sufficiently person-like but they sort of tend to assume that conclusion rather than argue for it, which causes memes that aren’t familiar with Less Wrong memes to wonder why Less Wrong memes are so incredibly confident that all AIs will necessarily act like autistic OCD people without any possibility at all of acting like normal reasonable people.
A whole lot of the sequences are dedicated to outlining just how reasonably normal people don’t act. I would want any Strong AI in charge of our fates to be person-like in that it is aware of what humans want in a way that we would accept, because the alternative to that is probably disaster, but I wouldn’t want one to be person-like in that its inductive biases are more like a human’s than an ideal Bayesian reasoner’s, or that it reasons about moral issues the way humans do intuitively, because our biases are often massively inappropriate, and our moral intuitions incoherent.
inductive biases are more like a human’s than an ideal Bayesian reasoner’s
Check out this post by Vladimir Nesov: “The problem of choosing Bayesian priors is in general the problem of formalizing preference, it can’t be solved completely without considering utility, without formalizing values, and values are very complicated. No simple morality, no simple probability.” Of course, having a human prior doesn’t necessitate being human-like… Or does it? Duh duh duh.
Today I’d rather say that we don’t know if “priors” is a fundamentally meaningful decision-theoretic idea, and so discussing what does or doesn’t determine it would be premature.
(Anybody who thinks I’m missing something, ask yourself: what do you think you know that you think I don’t think you know? How could I have come to not think you know something that you think you know? Are you confident of that model? This is where chess-playing subskills are very useful.)
Wow, I only associate that level of arrogance with Eliezer.
I don’t see how it’s arrogance, except maybe by insinuation/connotation; I’ll think about how to remove the insinuation/connotation. I was trying to describe an important skill of rationality, not assert my supremacy at that skill. But describing a skill sort of presupposes that the audience lacks the skill. So it’s awkward.
Your comments are probably better without such meta appendices. I lambast LW for being wrong about many worlds and for having a crypto-dualist philosophy of mind, and I find directness is better than intricate attempts to preempt the reader’s default epistemology. Going meta is not always for the best; save it up and then use it in the second round if you have to.
Your comments are probably better without such meta appendices.
This applies doubly for those whose ‘meta’ position is so closely associated with either fundamental quantum monads or outright support of theism based on the Catholic god.
(Inconsequential stylistic complaint: Atheists like to do it all the time, but it strikes me as juvenile not to capitalize “Catholic” or “God”. If you don’t capitalize “catholic” then it just means “universal”, and not capitalizing “God” is like making a point of writing Eliezer’s name as “eliezer” just because you think he’s the Antichrist. It’s contemptibly petty. Is there some justification I’m missing? (I’m not judging you by the way, just imagining a third party judge.))
If you don’t capitalize “catholic” then it just means “universal”
That’s true. Not writing “Catholic” was an error. It’s not like the Catholic religion is any more universal than, say, the ‘Liberal’ party here is particularly liberal. Names get capitals so we don’t confuse them with real words.
and not capitalizing “God” is like making a point of writing Eliezer’s name as “eliezer” just because you think he’s the Antichrist.
But here you are wrong.
When referring to supernatural entities that fall into the class ‘divine’ the label that applies is ‘god’. For example, Zeus is a god, Allah is a god and God is a god. If you happened to base your theology around Belar I would have written “the Alorn god”. Writing “the Alorn God” would be a corruption of grammar. If I was making a direct reference to God I would capitalize His name. I wasn’t. I was referring to a religion which, being monotheistic can be dereferenced to specify a particular fictional entity.
Other phrases I may utter:
The Arendish god is Chaldan
The Protestant god is God.
Children believe in believing in the Easter Bunny.
Is there some justification I’m missing?
The historic conceit that makes using capitalization appropriate when referring to God does not extend to all usages of the word ‘god’, even when the ultimate referent is Him. For all the airs we may give Him, God is just a god—with all that entails.
By the way I’ve come to think that your intuitions re quantum mind/monadology are at least plausibly correct/in-the-right-direction, but this epistemic shift hasn’t changed my thoughts about FAI at all; thus I fear compartmentalization on my part, and I’d like to talk with you about it when I’m able to reliably respond to email. It seems to me that there’s insufficient disturbed-ness about disagreement amongst the serious-minded Friendliness community.
Also, what’s your impression re psi? Or maybe it’s best not to get into that here.
Sounds like a good thing to have in a “before hitting ‘reply,’ consider these” checklist; but not to add to your own comment (for, as Will might say, “game-theoretic and signaling reasons.”)
Less Wrong memes think that person AIs won’t be sufficiently person-like but they sort of tend to assume that conclusion rather than argue for it, which causes memes that aren’t familiar with Less Wrong memes to wonder why Less Wrong memes are so incredibly confident that all AIs will necessarily act like autistic OCD people without any possibility at all of acting like normal reasonable people.
This exposes a circularity in lesswrongian reasoning: if you think of an AI as fundamental non-person like, then there is a need to bolt on human values. If you think of it as human—like , then huma-like values are more likely to be inhrerent or acquired naturally through interaction.
I don’t see the circularity. “human” is a subset of “person”; there’s no reason an AI that is a “person” will have “human” values. Also, just thinking of the AI as being human-like doesn’t actually make it human-like.
I feel morally obligated to restate a potentially relevant observation:
I think that an important underlying difference of perspective here is that the Less Wrong memes tend to automatically think of all AGIs as essentially computer programs whereas Goertzel-like memes tend to automatically think of at least some AGIs as non-negligibly essentially person-like. I think this is at least partially because the Less Wrong memes want to write an FAI that is essentially some machine learning algorithms plus a universal prior on top of sound decision theory whereas the Goertzel-like memes want to write an FAI that is essentially roughly half progam-like and half person-like. Less Wrong memes think that person AIs won’t be sufficiently person-like but they sort of tend to assume that conclusion rather than argue for it, which causes memes that aren’t familiar with Less Wrong memes to wonder why Less Wrong memes are so incredibly confident that all AIs will necessarily act like autistic OCD people without any possibility at all of acting like normal reasonable people. From that perspective the Goertzel-like memes look justified in being rather skeptical of Less Wrong memes. After all, it is easy to imagine a gradation between AIXI and whole brain emulations. Goertzel-like memes wish to create an AI somewhere between those two points, Less Wrong memes wish to create an AI that’s even more AIXI-like than AIXI is (in the sense of being more formally and theoretically well-founded than AIXI is). It’s important that each look at the specific kinds of AI that the other has in mind and start the exchange from there.
We don’t know if AIXI-approximating AIs would even be intelligent; how then can we be so confident that AIXI is a normative model and a definition of intelligence? This and other intuitions are likely underlying Goertzel’s cautious epistemic state, and LessWrong/SingInst truly hasn’t addressed issues like this. We don’t know what it takes to build AGI, we don’t know if intelligence runs on Bayes structure. Modern decision theory indicates that Eliezer was wrong, that Bayes structure isn’t fundamental to agentic optimization, that it only applies in certain cases, that Bayesian information theoretic models of cognition might not capture the special sauce of intelligence. What is fundamental? We don’t know! In the meantime we should be careful about drawing conclusions based on the assumed fundamental-ness of mathematical models which may or may not ultimately be accurate models, may or may not actually let you build literalistic self-improving AIs of the sort that LessWrong likes to speculate about.
I think your first paragraph was very useful.
I have no idea what your second paragraph is about—“modern decision theory” is not a very specific citation. If there is research concluding that probability theory only applies to certain special cases of optimization, it would be awesome if you could make a top-level post explaining it to us!
There have already been many top-level posts, but you’re right that I should have linked to them. Here is the LessWrong Wiki hub, here is a post by Wei Dai that cuts straight to the point.
A whole lot of the sequences are dedicated to outlining just how reasonably normal people don’t act. I would want any Strong AI in charge of our fates to be person-like in that it is aware of what humans want in a way that we would accept, because the alternative to that is probably disaster, but I wouldn’t want one to be person-like in that its inductive biases are more like a human’s than an ideal Bayesian reasoner’s, or that it reasons about moral issues the way humans do intuitively, because our biases are often massively inappropriate, and our moral intuitions incoherent.
Check out this post by Vladimir Nesov: “The problem of choosing Bayesian priors is in general the problem of formalizing preference, it can’t be solved completely without considering utility, without formalizing values, and values are very complicated. No simple morality, no simple probability.” Of course, having a human prior doesn’t necessitate being human-like… Or does it? Duh duh duh.
Today I’d rather say that we don’t know if “priors” is a fundamentally meaningful decision-theoretic idea, and so discussing what does or doesn’t determine it would be premature.
Wow, I only associate that level of arrogance with Eliezer.
I don’t see how it’s arrogance, except maybe by insinuation/connotation; I’ll think about how to remove the insinuation/connotation. I was trying to describe an important skill of rationality, not assert my supremacy at that skill. But describing a skill sort of presupposes that the audience lacks the skill. So it’s awkward.
It’s arrogance because you’re implying that you’ve already thought of and rejected any objection the reader could come up with.
Didn’t mean to imply that; deleted the offending paragraph at any rate.
Your comments are probably better without such meta appendices. I lambast LW for being wrong about many worlds and for having a crypto-dualist philosophy of mind, and I find directness is better than intricate attempts to preempt the reader’s default epistemology. Going meta is not always for the best; save it up and then use it in the second round if you have to.
This applies doubly for those whose ‘meta’ position is so closely associated with either fundamental quantum monads or outright support of theism based on the Catholic god.
(Inconsequential stylistic complaint: Atheists like to do it all the time, but it strikes me as juvenile not to capitalize “Catholic” or “God”. If you don’t capitalize “catholic” then it just means “universal”, and not capitalizing “God” is like making a point of writing Eliezer’s name as “eliezer” just because you think he’s the Antichrist. It’s contemptibly petty. Is there some justification I’m missing? (I’m not judging you by the way, just imagining a third party judge.))
That’s true. Not writing “Catholic” was an error. It’s not like the Catholic religion is any more universal than, say, the ‘Liberal’ party here is particularly liberal. Names get capitals so we don’t confuse them with real words.
But here you are wrong.
When referring to supernatural entities that fall into the class ‘divine’ the label that applies is ‘god’. For example, Zeus is a god, Allah is a god and God is a god. If you happened to base your theology around Belar I would have written “the Alorn god”. Writing “the Alorn God” would be a corruption of grammar. If I was making a direct reference to God I would capitalize His name. I wasn’t. I was referring to a religion which, being monotheistic can be dereferenced to specify a particular fictional entity.
Other phrases I may utter:
The Arendish god is Chaldan
The Protestant god is God.
Children believe in believing in the Easter Bunny.
The historic conceit that makes using capitalization appropriate when referring to God does not extend to all usages of the word ‘god’, even when the ultimate referent is Him. For all the airs we may give Him, God is just a god—with all that entails.
Sorry, you’re right, what confused me was “catholic god” in conjunction; “Catholic god” wouldn’t have tripped me up.
I think you’re right, I’ll just remove it.
By the way I’ve come to think that your intuitions re quantum mind/monadology are at least plausibly correct/in-the-right-direction, but this epistemic shift hasn’t changed my thoughts about FAI at all; thus I fear compartmentalization on my part, and I’d like to talk with you about it when I’m able to reliably respond to email. It seems to me that there’s insufficient disturbed-ness about disagreement amongst the serious-minded Friendliness community.
Also, what’s your impression re psi? Or maybe it’s best not to get into that here.
Sounds like a good thing to have in a “before hitting ‘reply,’ consider these” checklist; but not to add to your own comment (for, as Will might say, “game-theoretic and signaling reasons.”)
This exposes a circularity in lesswrongian reasoning: if you think of an AI as fundamental non-person like, then there is a need to bolt on human values. If you think of it as human—like , then huma-like values are more likely to be inhrerent or acquired naturally through interaction.
I don’t see the circularity. “human” is a subset of “person”; there’s no reason an AI that is a “person” will have “human” values. Also, just thinking of the AI as being human-like doesn’t actually make it human-like.
I dont’ see the relevance. Goetzel isn’t talking about building non-human persons.
If you design an AI on x-like principles, it will probably be X-like, unless something goes wrong.
Ah, I may not have gotten all the context.
If “something goes wrong” with high probability, it will probably not be X-like.
More the reverse. I don’t support your representation of either what LW memes or Eliezer’s. I’d call this a straw man.