I think everyone sane agrees that we’re doomed and soon.
Even as a doomer among doomers, you, with respect, come off as a rambling madman.
The problem is that the claim you’re making, such that alignment is so doomed that Eliezer Yudkowsky, one of the most if not the most of pessimistic voices among alignment people, is still somehow over optimistic about humanity’s prospects, is unsubstantiated.
It’s a claim, I think, that deserves some substantiation. Maybe you believe you’ve already provided as much. I disagree.
I’m guessing you’re operating on strong intuition here; and you know what, great, share your model of the world! But you apparently made this post with the intention to persuade, and I’m telling you you’ve done a poor job.
EDIT: To be clear, even if I were somehow granted vivid knowledge of the future through precognition, you’d still seem crazy to me at this point.
To be clear, even if I were somehow granted vivid knowledge of the future through precognition, you’d still seem crazy to me at this point.
(I assume you mean vivid knowledge of the future in which we are destroyed, obviously in the case where everything goes well I’ve got some problem with my reasoning)
That’s a good distinction to make, a man can be right for the wrong reasons.
Even as a doomer among doomers, you, with respect, come off as a rambling madman.
Certainly mad enough to take “madman” as a compliment, thank you!
I’d be interested if you know a general method I could use to tell if I’m mad. The only time I actually know it happened (thyroid overdose caused a manic episode) I noticed pretty quickly and sought help. What test should I try today?
Obviously “everyone disagrees with me and I can’t convince most people” is a bad sign. But after long and patient effort I have convinced a number of unfortunates in my circle of friends. Some of whom have always seemed pretty sharp to me.
And you must admit, the field as a whole seems to be coming round to my point of view!
Rambling I do not take as a compliment. But nevertheless I thank you for the feedback.
I thought I’d written the original post pretty clearly and succinctly. If not, advice on how to write more clearly is always welcome. If you get my argument, can you steelman it?
I’m guessing you’re operating on strong intuition here
Your guess is correct, I literally haven’t shifted my position on all this since 2010. Except to notice that everything’s happening much faster than I expected it to. Thirteen years ago I expected this to kill our children. Now I worry that it’s going to kill my parents. AlphaZero was the fire alarm for me. General Game Playing was one of the more important sub-problems.
I agree that if you haven’t changed your mind for thirteen years in a field that’s moving fast, you’re probably stuck.
I think my basic intuitions are:
“It’s a terrible idea to create a really strong mind that doesn’t like you.”
“Really strong minds are physically possible, humans are nowhere near.”
“Human-level AI is easy because evolution did it to us, quickly, and evolution is stupid.”
“Recursive self-improvement is possible.”
Which of these four things do you disagree with? Or do you think the four together are insufficient?
I get that your argument is essentially as follows:
1.) Solving the problem of what values to put into an ai, even given the other technical issues being solved, is impossibly difficult in real life.
2.) To prove the problem’s impossible difficulty, here’s a much kinder version of reality where the problem still remains impossible.
I don’t think you did 2, and it requires me to already accept 1 is true, which I think it probably isn’t, and I think that most would agree with me on this point, at least in principle.
Which of these four things do you disagree with?
I don’t disagree with any of them. I doubt there’s a convincing argument that could get me to disagree with any of those as presented.
What I am not convinced of, is that given all those assumptions being true, certain doom necessarily follows, or that there is no possible humanly tractable scheme which avoids doom in whatever time we have left.
I’m not clever enough to figure out what the solution is mind you, nor am I especially confident that someone else is necessarily going to. Please don’t confuse me for someone who doesn’t often worry about these things.
What I am not convinced of, is that given all those assumptions being true, certain doom necessarily follows, or that there is no possible humanly tractable scheme which avoids doom in whatever time we have left.
OK, cool, I mean “just not building the AI” is a good way to avoid doom, and that still seems at least possible, so we’re maybe on the same page there.
And I think you got what I was trying to say, solving 1 and/or 2 can’t be done iteratively or by patching together a huge list of desiderata. We have to solve philosophy somehow, without superintelligent help. As I say, that looks like the harder part to me.
Please don’t confuse me for someone who doesn’t often worry about these things.
None. But if a problem’s not solvable in an easy case, it’s not solvable in a harder case.
Same argument as for thinking about Solomonoff Induction or Halting Oracles. If you can’t even do it with magic powers, that tells you something about what you can really do.
In what version of reality do you think anyone has hope for an ai alignment Groundhog Day?
Even as a doomer among doomers, you, with respect, come off as a rambling madman.
The problem is that the claim you’re making, such that alignment is so doomed that Eliezer Yudkowsky, one of the most if not the most of pessimistic voices among alignment people, is still somehow over optimistic about humanity’s prospects, is unsubstantiated.
It’s a claim, I think, that deserves some substantiation. Maybe you believe you’ve already provided as much. I disagree.
I’m guessing you’re operating on strong intuition here; and you know what, great, share your model of the world! But you apparently made this post with the intention to persuade, and I’m telling you you’ve done a poor job.
EDIT: To be clear, even if I were somehow granted vivid knowledge of the future through precognition, you’d still seem crazy to me at this point.
(I assume you mean vivid knowledge of the future in which we are destroyed, obviously in the case where everything goes well I’ve got some problem with my reasoning)
That’s a good distinction to make, a man can be right for the wrong reasons.
Certainly mad enough to take “madman” as a compliment, thank you!
I’d be interested if you know a general method I could use to tell if I’m mad. The only time I actually know it happened (thyroid overdose caused a manic episode) I noticed pretty quickly and sought help. What test should I try today?
Obviously “everyone disagrees with me and I can’t convince most people” is a bad sign. But after long and patient effort I have convinced a number of unfortunates in my circle of friends. Some of whom have always seemed pretty sharp to me.
And you must admit, the field as a whole seems to be coming round to my point of view!
Rambling I do not take as a compliment. But nevertheless I thank you for the feedback.
I thought I’d written the original post pretty clearly and succinctly. If not, advice on how to write more clearly is always welcome. If you get my argument, can you steelman it?
Your guess is correct, I literally haven’t shifted my position on all this since 2010. Except to notice that everything’s happening much faster than I expected it to. Thirteen years ago I expected this to kill our children. Now I worry that it’s going to kill my parents. AlphaZero was the fire alarm for me. General Game Playing was one of the more important sub-problems.
I agree that if you haven’t changed your mind for thirteen years in a field that’s moving fast, you’re probably stuck.
I think my basic intuitions are:
“It’s a terrible idea to create a really strong mind that doesn’t like you.”
“Really strong minds are physically possible, humans are nowhere near.”
“Human-level AI is easy because evolution did it to us, quickly, and evolution is stupid.”
“Recursive self-improvement is possible.”
Which of these four things do you disagree with? Or do you think the four together are insufficient?
I get that your argument is essentially as follows:
1.) Solving the problem of what values to put into an ai, even given the other technical issues being solved, is impossibly difficult in real life.
2.) To prove the problem’s impossible difficulty, here’s a much kinder version of reality where the problem still remains impossible.
I don’t think you did 2, and it requires me to already accept 1 is true, which I think it probably isn’t, and I think that most would agree with me on this point, at least in principle.
I don’t disagree with any of them. I doubt there’s a convincing argument that could get me to disagree with any of those as presented.
What I am not convinced of, is that given all those assumptions being true, certain doom necessarily follows, or that there is no possible humanly tractable scheme which avoids doom in whatever time we have left.
I’m not clever enough to figure out what the solution is mind you, nor am I especially confident that someone else is necessarily going to. Please don’t confuse me for someone who doesn’t often worry about these things.
OK, cool, I mean “just not building the AI” is a good way to avoid doom, and that still seems at least possible, so we’re maybe on the same page there.
And I think you got what I was trying to say, solving 1 and/or 2 can’t be done iteratively or by patching together a huge list of desiderata. We have to solve philosophy somehow, without superintelligent help. As I say, that looks like the harder part to me.
I promise I’ll try not to!
None. But if a problem’s not solvable in an easy case, it’s not solvable in a harder case.
Same argument as for thinking about Solomonoff Induction or Halting Oracles. If you can’t even do it with magic powers, that tells you something about what you can really do.