Eliezer,
I’d like to take a stab at the internal criterion question.
One differerence between me and the program you describe is that I have a hoped for future. Say “I’d like to play golf on Wednesday.” Now, I could calculate the odds of Wednesday not actually arriving (nuclear war,asteroid impact...), or me not being alive to see it (sudden heartattack...), and I would get an answer greater than zero. Why don’t I operate on those non-zero probabilities? (The other difference between me and the program you describe)
I think it has to do with faith. That is I have faith that my hoped for future will occur, or at least some semblance of it. I seem to have this faith despite previous losses.
Take the field of AI. There is a hoped for future, a computer will demonstrate intelligence, some hope the machine will become conscious. There is a faith that “we can solve these problems” I’m not sure the machine you describe would have either characteristic.
I don’t know how to formalize this, but it seems an important aspect of the situation.
Eliezer, I’d like to take a stab at the internal criterion question. One differerence between me and the program you describe is that I have a hoped for future. Say “I’d like to play golf on Wednesday.” Now, I could calculate the odds of Wednesday not actually arriving (nuclear war,asteroid impact...), or me not being alive to see it (sudden heartattack...), and I would get an answer greater than zero. Why don’t I operate on those non-zero probabilities? (The other difference between me and the program you describe) I think it has to do with faith. That is I have faith that my hoped for future will occur, or at least some semblance of it. I seem to have this faith despite previous losses. Take the field of AI. There is a hoped for future, a computer will demonstrate intelligence, some hope the machine will become conscious. There is a faith that “we can solve these problems” I’m not sure the machine you describe would have either characteristic. I don’t know how to formalize this, but it seems an important aspect of the situation.