Just read your latest post on your research program and attempt to circumvent social reward, then came here to get a sense at your hunt for a paradigm.
Here are some notes on Human in the Loop.
You say,
“We feed our preferences in to an aggregator, the AI reads out the aggregator.”
One thing to notice is that this framing makes some assumptions that might be too specific. It’s really hard, I know, to be general enough while still having content. But my ears pricked up at this one. Does it have to be an ‘aggregator’ maybe the best way of revealing preferences is not through an aggregator? Notice that I use the more generic ‘reveal’ as opposed to ‘feed’ because feed at least to me implies some methods of data discovery and not others. Also, I worry about what useful routes aggregation might fail to imply.
I hope this doesn’t sound too stupid and semantic.
You also say, “This schema relies on a form of corrigibility.” My first thought was actually that it implies human corrigibility, which I don’t think is a settled question. Our difficulty having political preferences that are not self-contradictory, preferences that don’t poll one way then vote another, makes me wonder about the problems of thinking about preferences over all worlds and preference aggregation as part of the difficulty of our own corrigibility. Combine that with the incorrigibility of the AI makes for a difficult solution space.
On emergent properties, I see no way to escape the “First we shape our spaces, then our spaces shape us” conundrum. Any capacity that is significantly useful will change its users from their previous set of preferences. Just as certain AI research might be distorted by social reward, so too can AI capabilities be a distorting reward. That’s not necessarily bad, but it is an unpredictable dynamic, since value drift when dealing with previously unknown capabilities seems hard to stop (especially since intuitions will be weak to nonexistent).
Just read your latest post on your research program and attempt to circumvent social reward, then came here to get a sense at your hunt for a paradigm.
Here are some notes on Human in the Loop.
You say, “We feed our preferences in to an aggregator, the AI reads out the aggregator.” One thing to notice is that this framing makes some assumptions that might be too specific. It’s really hard, I know, to be general enough while still having content. But my ears pricked up at this one. Does it have to be an ‘aggregator’ maybe the best way of revealing preferences is not through an aggregator? Notice that I use the more generic ‘reveal’ as opposed to ‘feed’ because feed at least to me implies some methods of data discovery and not others. Also, I worry about what useful routes aggregation might fail to imply.
I hope this doesn’t sound too stupid and semantic.
You also say, “This schema relies on a form of corrigibility.” My first thought was actually that it implies human corrigibility, which I don’t think is a settled question. Our difficulty having political preferences that are not self-contradictory, preferences that don’t poll one way then vote another, makes me wonder about the problems of thinking about preferences over all worlds and preference aggregation as part of the difficulty of our own corrigibility. Combine that with the incorrigibility of the AI makes for a difficult solution space.
On emergent properties, I see no way to escape the “First we shape our spaces, then our spaces shape us” conundrum. Any capacity that is significantly useful will change its users from their previous set of preferences. Just as certain AI research might be distorted by social reward, so too can AI capabilities be a distorting reward. That’s not necessarily bad, but it is an unpredictable dynamic, since value drift when dealing with previously unknown capabilities seems hard to stop (especially since intuitions will be weak to nonexistent).