One important point for AI safety, at least in the early stages, is a inability to change it’s source code. A whole lot of problems seem related to recursive self improvement within it’s source code, so cutting off that area of improvement seems wise in the early stages. What do you think.
I don’t think there’s much difference in existential risk between AGIs that can modify their own code running on their own hardware, and those that can only create better successors sharing their goals but running on some other hardware.
That might be a crux here, because my view is that hardware improvements are much harder to do effectively, especially in secret around the human level, due to Landauer’s Principle essentially bounding efficiency of small scale energy usage close to that of the brain (20 Watts.) Combine this with 2-3 orders of magnitude worse efficiency than the brain and basically any evolutionary object compared to human objects, and the fact it’s easier to get better software than hardware due to the virtual/real life distinction, and this is a crux for me.
I’m not sure how this is a crux. Hardware improvements are irrelevant to what either of us were saying.
I’m saying that there is little risk difference between an AGI reprogramming itself to have better software, and programming some other computer with better software.
One important point for AI safety, at least in the early stages, is a inability to change it’s source code. A whole lot of problems seem related to recursive self improvement within it’s source code, so cutting off that area of improvement seems wise in the early stages. What do you think.
I don’t think there’s much difference in existential risk between AGIs that can modify their own code running on their own hardware, and those that can only create better successors sharing their goals but running on some other hardware.
That might be a crux here, because my view is that hardware improvements are much harder to do effectively, especially in secret around the human level, due to Landauer’s Principle essentially bounding efficiency of small scale energy usage close to that of the brain (20 Watts.) Combine this with 2-3 orders of magnitude worse efficiency than the brain and basically any evolutionary object compared to human objects, and the fact it’s easier to get better software than hardware due to the virtual/real life distinction, and this is a crux for me.
I’m not sure how this is a crux. Hardware improvements are irrelevant to what either of us were saying.
I’m saying that there is little risk difference between an AGI reprogramming itself to have better software, and programming some other computer with better software.