but that stable eventual goal may be very difficult to predict in advance
No, the point of that section is that there are many AI designs in which we can’t explicitly make goals.
This task is perhaps the primary difficulty in designing friendly AI.
Some at SIAI disagree. I’ve already qualified with ‘perhaps’.
not just these two, bad wording
Fixed.
should be a link/reference, a FAQ can be entered at any question
Alas, I think no such documents exist. But luckily, the sentence is unneeded.
a textbook error in machine learning methodology is a bad match for a fundamental problem, unless argued as being such in this particular case
I disagree. A textbook error in machine learning that has not yet been solved is good match for a fundamental problem.
just two? Bad wording.
Fixed.
the same can be said of humans (correctly, but as a result it doesn’t work as a simple distinguishing argument)
Again, I’m not claiming that these aren’t also problems elsewhere.
“utilitarian”: a potentially new term without any introduction, even with a link, is better to be avoided
Maybe. If you can come up with a concise way to get around it, I’m all ears.
could/might would be better
Agreed.
this statement is repeated about 5 times in close forms, should change the wording somehow
Why? I’ve already varied the wording, and the point of a FAQ with link anchors is that not everybody will read the whole FAQ from start to finish. I repeat the phrase ‘machine superintelligence’ in variations a lot, too.
an even more opaque term without explanation
Hence, the link, for people who don’t know.
not actually clear (from my point of view, not simulated naive point of view)
Changed to ‘might’.
again, these things are there to be decided upon, not “predicted”
Fixed.
Thanks for your comments. As you can see I am revising, so please do continue!
No, the point of that section is that there are many AI designs in which we can’t explicitly make goals.
Some at SIAI disagree. I’ve already qualified with ‘perhaps’.
Fixed.
Alas, I think no such documents exist. But luckily, the sentence is unneeded.
I disagree. A textbook error in machine learning that has not yet been solved is good match for a fundamental problem.
Fixed.
Again, I’m not claiming that these aren’t also problems elsewhere.
Maybe. If you can come up with a concise way to get around it, I’m all ears.
Agreed.
Why? I’ve already varied the wording, and the point of a FAQ with link anchors is that not everybody will read the whole FAQ from start to finish. I repeat the phrase ‘machine superintelligence’ in variations a lot, too.
Hence, the link, for people who don’t know.
Changed to ‘might’.
Fixed.
Thanks for your comments. As you can see I am revising, so please do continue!
I know, but you use the word “predict”, which is what I was pointing out.
What do you mean, “has not yet been solved”? This kind of error is routinely being solved in practice, which is why it’s a textbook example.
Yes, but that makes it a bad illustration.
Because it’s bad prose, it sounds unnatural (YMMV).
This doesn’t address my argument. I know there is a link and I know that people could click on it, so that’s not what I meant.
(More later, maybe.)