In AI research, intelligent agents typically have a clear-cut and well-defined final goal, e.g., win the chess game or drive the car to the destination legally. The same holds for most tasks that we assign to humans, because the time horizon and context is known and limited. (...) a truly well-defined goal would specify how all particles in our Universe should be arranged at the end of time.
We typically only care about the arrangement of particles at the end of the task, because that is the nature of the simple tasks we usually use machines for today. Actually, even that is not true: when “driving the car to destination legally” we care not only about the arrangement of the particles of the car at the end of the trip, but also about what happened on the way—that’s what “legally” means here. (Unless we also count “police sending us tickets” as particles. But I guess the car is supposed to follow the laws even when the police does not look.)
We can define “journey” goals e.g. by calculating score at each time interval, and trying to maximize the sum or the average (or some other function) of all the intervals. This can make sense even if we don’t know how long the task will last.
treat experience as inherently positive and not try to distinguish between positive and negative experiences.
This sounds wrong. But I am not even sure what exactly would we measure here, if both positive and negative experience count the same. Is it the intensity of the experience (in either direction) which counts? (That is, would you rather be tortured than bored? Would you rather be tortured really painfully than enjoying a mild pleasure?) Or is it duration of the experience? (That is, we want to maximize the subjective time of sentient beings, regardless of what happens during the time? Would you rather live 1001 years in hell than 1000 years in heaven?)
During the end of a drive there would either be/not be a configuration of particles in the shape of a paper ticket memorializing your transgression of the law. And if not that, there is a configuration of particles in the heads of the law enforcement officials recalling your transgression and planning on writing you a ticket or whatever. Any universe-wide configuration of particles contains the history of all of the events preceding it, even if they are opaque to us, because the possibility space of particle configurations is (probably?) larger than utility relevant histories.
I’m talking about particles at the quantum level here. Subatomic particles are ridiculously small. The amount of empty space in the universe is incomprehensibly vaster than the amount of particles that inhabit it, so it wouldn’t be surprising to me if it were impossible to arrive at the same universe-wide state from different starting points, but I don’t know that that’s true as a matter of fact. But if there were a perfectly symmetrical perturbation, by definition it would be unobservable to us, since we would end up in the exact same state along either pathway.
We typically only care about the arrangement of particles at the end of the task, because that is the nature of the simple tasks we usually use machines for today. Actually, even that is not true: when “driving the car to destination legally” we care not only about the arrangement of the particles of the car at the end of the trip, but also about what happened on the way—that’s what “legally” means here. (Unless we also count “police sending us tickets” as particles. But I guess the car is supposed to follow the laws even when the police does not look.)
We can define “journey” goals e.g. by calculating score at each time interval, and trying to maximize the sum or the average (or some other function) of all the intervals. This can make sense even if we don’t know how long the task will last.
This sounds wrong. But I am not even sure what exactly would we measure here, if both positive and negative experience count the same. Is it the intensity of the experience (in either direction) which counts? (That is, would you rather be tortured than bored? Would you rather be tortured really painfully than enjoying a mild pleasure?) Or is it duration of the experience? (That is, we want to maximize the subjective time of sentient beings, regardless of what happens during the time? Would you rather live 1001 years in hell than 1000 years in heaven?)
Of course. That’s why I proposed refining it.
I thought it was obvious. It is the integral of total experience (suitably defined) through time that counts.
During the end of a drive there would either be/not be a configuration of particles in the shape of a paper ticket memorializing your transgression of the law. And if not that, there is a configuration of particles in the heads of the law enforcement officials recalling your transgression and planning on writing you a ticket or whatever. Any universe-wide configuration of particles contains the history of all of the events preceding it, even if they are opaque to us, because the possibility space of particle configurations is (probably?) larger than utility relevant histories.
So there is no way that we can arrive at the same state from different starting points? That seems ridiculous to me.
I’m talking about particles at the quantum level here. Subatomic particles are ridiculously small. The amount of empty space in the universe is incomprehensibly vaster than the amount of particles that inhabit it, so it wouldn’t be surprising to me if it were impossible to arrive at the same universe-wide state from different starting points, but I don’t know that that’s true as a matter of fact. But if there were a perfectly symmetrical perturbation, by definition it would be unobservable to us, since we would end up in the exact same state along either pathway.