If the AGi is a human mind upload, it is in no way a FAI, and I don’t think it is what MIRI is aiming.
In case a neuromorphic AI is created, diferent arrays of neurons can give weidly diferent minds, We should not reason about a hipotetical AI using a human minds has a model and make predictions about it, even if that mind is based on biological minds.
What if the first neuron based AI has a mind more similar than a ant than a human, in that case anger, jealousy, freedom, etc are not longer part of the mind, or the mind could have totally new emotions, or things that are not emotions that we known not.
A mind that we don’t understand enought, should not be said to be friendly and set free to the world, and I don’t think that is being said here.
If you leave their mind unaltered, you just have a human. They’re not smart enough to really be useful. Once you start altering it, insanity becomes a likely result.
Best case scenario, you get one person’s CEV. Most likely scenario, you get someone too insane to be useful. Worst case, you have an insane supergenius.
Do you have a precise definition of “ethical” in mind? Where by “precise” I mean something roughly equivalent to a math paper.
Without such a definition, how will you know the person in question is ethical? With such a definition, how will you guarantee that the person in question meets it, will continue to meet it, etc.? How certain are you such a person exists?
Do you have a precise definition of “ethical” in mind?
No. Don’t need one either.
Where by “precise” I mean something roughly equivalent to a math paper.Without such a definition, how will you know the person in question is ethical?
By ordinary standards.. Eg, Einstein was more ethical than von Neuman.
With such a definition, how will you guarantee that the person in question meets it, will continue to meet it, etc.?
Since when did functional duplicates start diverging unaccountably?
How certain are you such a person exists?
I’m not talking about mathematically proveable ethics OR about superintelligence. I’m talking about entrusting (superior) human level ems less than absolute power....ie what we do already with real humans,
I’m fairly willing to believe that intuitive understandings of “more ethical” will do well for imprecise things like “we’ll probably get better results by instantiating a more ethical person as an em than a less ethical one”. I’m less convinced the results will be good compared to obvious alternatives like not instantiating anyone as an em.
We see value drift as a result of education, introspection, religious conversion or deconversion, rationality exposure, environment, and societal power. Why would you expect not to see value drift in the face of a radical change in environment, available power, and thinking speed? I’m not concerned about whether or not the value drift is “accountable”, I’m concerned that it might be large and not precisely predicted in advance.
Once you entrust the em with large but less than absolute power, how do you plan to keep its power less than absolute? Why do you expect this to be an easier problem than it would be for a non-em AI?
I’m less convinced the results will be good compared to obvious alternatives like not instantiating anyone as an em.
Not building an AI at all is not seen by MIRI as an obvious alternative. That seems an uneven playing field.
Why would you expect not to see value drift in the face of a radical change in environment, available power, and thinking speed?
I don’t require the only acceptable level of value drift to be zero, since I am not proposing giving an em absolute power. I am talking about giving human level (or incrementally more) ems human style (ditto) jobs. That being the case, human style levels of drift will not make things worse,
Once you entrust the em with large but less than absolute power, how do you plan to keep its power less than absolute?
We have ways of reducing humans from office. Why would that be a novel, qualitatively different problem in the case of an em that is 10% or 5% or 1% smarter than a smart human?
If the AGi is a human mind upload, it is in no way a FAI, and I don’t think it is what MIRI is aiming.
In case a neuromorphic AI is created, diferent arrays of neurons can give weidly diferent minds, We should not reason about a hipotetical AI using a human minds has a model and make predictions about it, even if that mind is based on biological minds.
What if the first neuron based AI has a mind more similar than a ant than a human, in that case anger, jealousy, freedom, etc are not longer part of the mind, or the mind could have totally new emotions, or things that are not emotions that we known not.
A mind that we don’t understand enought, should not be said to be friendly and set free to the world, and I don’t think that is being said here.
How could a functional duplicate of a person known to ethical fail to be friendly?
Power corrupts, and absolute power corrupts absolutely.
Any AI is a super AII?
If you leave their mind unaltered, you just have a human. They’re not smart enough to really be useful. Once you start altering it, insanity becomes a likely result.
Best case scenario, you get one person’s CEV. Most likely scenario, you get someone too insane to be useful. Worst case, you have an insane supergenius.
If humans weren’t useful, humans wouldn’t employ humans. A Hawking brain that needed no sleep would be a good start,
Do you have a precise definition of “ethical” in mind? Where by “precise” I mean something roughly equivalent to a math paper.
Without such a definition, how will you know the person in question is ethical? With such a definition, how will you guarantee that the person in question meets it, will continue to meet it, etc.? How certain are you such a person exists?
No. Don’t need one either.
By ordinary standards.. Eg, Einstein was more ethical than von Neuman.
Since when did functional duplicates start diverging unaccountably?
I’m not talking about mathematically proveable ethics OR about superintelligence. I’m talking about entrusting (superior) human level ems less than absolute power....ie what we do already with real humans,
I’m fairly willing to believe that intuitive understandings of “more ethical” will do well for imprecise things like “we’ll probably get better results by instantiating a more ethical person as an em than a less ethical one”. I’m less convinced the results will be good compared to obvious alternatives like not instantiating anyone as an em.
We see value drift as a result of education, introspection, religious conversion or deconversion, rationality exposure, environment, and societal power. Why would you expect not to see value drift in the face of a radical change in environment, available power, and thinking speed? I’m not concerned about whether or not the value drift is “accountable”, I’m concerned that it might be large and not precisely predicted in advance.
Once you entrust the em with large but less than absolute power, how do you plan to keep its power less than absolute? Why do you expect this to be an easier problem than it would be for a non-em AI?
Not building an AI at all is not seen by MIRI as an obvious alternative. That seems an uneven playing field.
I don’t require the only acceptable level of value drift to be zero, since I am not proposing giving an em absolute power. I am talking about giving human level (or incrementally more) ems human style (ditto) jobs. That being the case, human style levels of drift will not make things worse,
We have ways of reducing humans from office. Why would that be a novel, qualitatively different problem in the case of an em that is 10% or 5% or 1% smarter than a smart human?