The qualification with “perhaps” makes it tautological and therefore silly. (You may notice that my comment was also tautological).
The slight strawman with “insisting on theoretically perfect” is, well, I’ll call it silly. As Eliezer replied, the goal is more like theoretically not doomed.
And last, the typo in “SI is perhaps being too cautious by insisting on theoretically perfect SI” makes it funny.
The article did mention that even with a “perfect” theory , there may be mistakes in the proof or the implementation may go wrong. I don’t remember him saying so as clearly in earlier writings as he did in this comment, so it’s good we raised the issue.
When a heuristic AI is creating a successor that shares its goals, does it insist on formally-verified self improvements? Does it try understanding its mushy, hazy goal system so as to avoid reifying something it would regret given its current goals? It seems to me like some mind eventually will have to confront the FAI issue, why not humans then?
If you check with Creating Friendly AI you will see that the term is defined by its primary proponent as follows:
The term “Friendly AI” refers to the production of human-benefiting, non-human harming actions in Artificial Intelligence systems that have advanced to the point of making real-world plans in pursuit of goals.
It’s an anthropocentric term. Only humans would care about creating this sort of agent. You would have to redefine the term if you want to use it to refer to something more general.
Half specifically referred to “creating a successor that shares it’s goals”; this is the problem we face when building an FAI. Nobody is saying an agent with arbitrary goals must at some point face the challenge of building an FAI.
(Incidentally, while Friendly is anthropocentric by default, in common usage analogous concepts relating to other species are referred to as “Friendly to X” of “X-Friendly”, just a good is by default used to mean by human standards, but is sometimes used in “good for X”.
does it insist on formally-verified self improvements. Does it try understanding its mushy, hazy goal system so as to avoid reifying something it would regret given its current goals.
Apparently not. If it did do these things perfectly, it would not be what we are here calling the “heuristic AI.”
Good reference. SI is perhaps being too cautious by insisting on theoretically perfect AI only.
This is perhaps a silly statement.
Why do you think it is “silly”?
The qualification with “perhaps” makes it tautological and therefore silly. (You may notice that my comment was also tautological).
The slight strawman with “insisting on theoretically perfect” is, well, I’ll call it silly. As Eliezer replied, the goal is more like theoretically not doomed.
And last, the typo in “SI is perhaps being too cautious by insisting on theoretically perfect SI” makes it funny.
Thanks, at least I corrected the typo.
The article did mention that even with a “perfect” theory , there may be mistakes in the proof or the implementation may go wrong. I don’t remember him saying so as clearly in earlier writings as he did in this comment, so it’s good we raised the issue.
When a heuristic AI is creating a successor that shares its goals, does it insist on formally-verified self improvements? Does it try understanding its mushy, hazy goal system so as to avoid reifying something it would regret given its current goals? It seems to me like some mind eventually will have to confront the FAI issue, why not humans then?
If you check with Creating Friendly AI you will see that the term is defined by its primary proponent as follows:
It’s an anthropocentric term. Only humans would care about creating this sort of agent. You would have to redefine the term if you want to use it to refer to something more general.
Half specifically referred to “creating a successor that shares it’s goals”; this is the problem we face when building an FAI. Nobody is saying an agent with arbitrary goals must at some point face the challenge of building an FAI.
(Incidentally, while Friendly is anthropocentric by default, in common usage analogous concepts relating to other species are referred to as “Friendly to X” of “X-Friendly”, just a good is by default used to mean by human standards, but is sometimes used in “good for X”.
Apparently not. If it did do these things perfectly, it would not be what we are here calling the “heuristic AI.”