@Eli: “An AI obeys the laws of physics; it is a lawful system just like you; everything in the AI happens for a reason, just like in your own brain. So if this judgment doesn’t come from a ghost that you summoned into the machine… where does it come from? How does the AI know what is “correct”? What code executes while it’s making that judgment
sure—some piece of code will indeed be executing if/when a smarter than human AI tells us what our ethical mistakes are and why. But if we knew what that piece of code was right now, then the AI wouldn’t be smarter than human.
“and where did the code come from?”
Such code may have been written by other pieces of meta-code acting under the influence of data from the real world, that meta-code may in turn have been written by other pieces of code. This process will, of course, track back to a programmer hitting keys on a keyboard, but necessarily not in a way that you or I could understand. If it tracked back to a programmer hitting keys on a keyboard in an easy-to-understand way, you would not be dealing with a super-intelligence.
“Oh, and you should probably Taboo that there word “correct” while you’re at it; if you can’t, don’t worry, it’s coming up.”
@Eli: “An AI obeys the laws of physics; it is a lawful system just like you; everything in the AI happens for a reason, just like in your own brain. So if this judgment doesn’t come from a ghost that you summoned into the machine… where does it come from? How does the AI know what is “correct”? What code executes while it’s making that judgment
sure—some piece of code will indeed be executing if/when a smarter than human AI tells us what our ethical mistakes are and why. But if we knew what that piece of code was right now, then the AI wouldn’t be smarter than human.
“and where did the code come from?”
Such code may have been written by other pieces of meta-code acting under the influence of data from the real world, that meta-code may in turn have been written by other pieces of code. This process will, of course, track back to a programmer hitting keys on a keyboard, but necessarily not in a way that you or I could understand. If it tracked back to a programmer hitting keys on a keyboard in an easy-to-understand way, you would not be dealing with a super-intelligence.
“Oh, and you should probably Taboo that there word “correct” while you’re at it; if you can’t, don’t worry, it’s coming up.”
I look forward to that!