John Maxwell-
I thought the security/adventrure example was good, but that the way I portrayed it might make it seem that ever-alternating IS the answer. Heregoes: A man lives as a bohemian out on the street, nomadically day to day solving his problems of how to get food and shelter. It seems to him that he would be better off looking for a secure life, and thus gets a job to make money. Working for money for a secure life is difficult and tiring and it seems to him that he will be better off once he has the money and is secure. Now he’s worked a long time and has the money and is secure, which he now finds is boring both in comparison to working and living a bohemian life with uncertainty in it. People do value uncertainty and ‘authenticity’ to a very high degree. Thus Being Secure is > Working to be secure > Not being secure > being secure.
Now, Eliezer would appropriately point out that the man only got trapped in this loop, because he didn’t actually know what would make him happiest, but assumed without having the experience. But, that being said, do we think this fellow would have been satisfied being told to start with ‘Don’t bother working son, this is better for you, trust me!’ There’s no obvious reason to me why the fAI will allow people the autonomy they so desire to pursue their own mistakes unless the final calculation of human utility determines that it wins out, and this is dubious… I’m saying that I don’t care if what in truth maximizes utility is for everyone to believe they’re 19th century god-fearing farmers, or to be on a circular magic quest the memory of the earliest day of which disappears each day, such that it replays for eternity, or whatever simulation the fAI decides on for post-singularity humanity, I think I’d rather be free of it to fuck up my own life. Me and many others.
I guess this goes to another more important problem than human nonlinear preference- Why should we trust an AI that maximizes human utility, even if it understands what that means? Why should we, from where we sit now, like what human volition (a collection of non-linear preferences) extrapolates to, and what value do we place on our own autonomy?
John Maxwell- I thought the security/adventrure example was good, but that the way I portrayed it might make it seem that ever-alternating IS the answer. Heregoes: A man lives as a bohemian out on the street, nomadically day to day solving his problems of how to get food and shelter. It seems to him that he would be better off looking for a secure life, and thus gets a job to make money. Working for money for a secure life is difficult and tiring and it seems to him that he will be better off once he has the money and is secure. Now he’s worked a long time and has the money and is secure, which he now finds is boring both in comparison to working and living a bohemian life with uncertainty in it. People do value uncertainty and ‘authenticity’ to a very high degree. Thus Being Secure is > Working to be secure > Not being secure > being secure.
Now, Eliezer would appropriately point out that the man only got trapped in this loop, because he didn’t actually know what would make him happiest, but assumed without having the experience. But, that being said, do we think this fellow would have been satisfied being told to start with ‘Don’t bother working son, this is better for you, trust me!’ There’s no obvious reason to me why the fAI will allow people the autonomy they so desire to pursue their own mistakes unless the final calculation of human utility determines that it wins out, and this is dubious… I’m saying that I don’t care if what in truth maximizes utility is for everyone to believe they’re 19th century god-fearing farmers, or to be on a circular magic quest the memory of the earliest day of which disappears each day, such that it replays for eternity, or whatever simulation the fAI decides on for post-singularity humanity, I think I’d rather be free of it to fuck up my own life. Me and many others.
I guess this goes to another more important problem than human nonlinear preference- Why should we trust an AI that maximizes human utility, even if it understands what that means? Why should we, from where we sit now, like what human volition (a collection of non-linear preferences) extrapolates to, and what value do we place on our own autonomy?