1) seems like mostly a sideshow — while we could get agency from this, unless people are trying hard I don’t think it would tend to find especially competent agents to emulate, and may not have a good handle on what’s going on in the world.
I’m very puzzled by this opinion. If we can reduce the ‘drunkenness’ issue, this type of agency scales to at least the competence level of most competent humans (or indeed, fictional characters) in existence, and probably at least some distance beyond by extrapolation (and run cheaply in faster than realtime). These agents are not safe: humans are not fully aligned to human values, power corrupts, and Joseph Stalin was not well aligned with the needs to the citizenry of Russia. This seems like plenty to be concerned about, rather than a sideshow. Now, the ways in which they’re not aligned are at least ones we have a good intuitive and practical understanding of, and some partial solutions for controlling (things like love, guilt, salaries, and law enforcement).
I’m very puzzled by this opinion. If we can reduce the ‘drunkenness’ issue, this type of agency scales to at least the competence level of most competent humans (or indeed, fictional characters) in existence, and probably at least some distance beyond by extrapolation (and run cheaply in faster than realtime). These agents are not safe: humans are not fully aligned to human values, power corrupts, and Joseph Stalin was not well aligned with the needs to the citizenry of Russia. This seems like plenty to be concerned about, rather than a sideshow. Now, the ways in which they’re not aligned are at least ones we have a good intuitive and practical understanding of, and some partial solutions for controlling (things like love, guilt, salaries, and law enforcement).