Having a system that stops when you reach a level of tested intelligence sounds appealing, but I’d be afraid of the measure of intelligence at hand being too fuzzy.
So the system would not detect it had achieved that level of intelligence, and it would bootstrap and take off in time to destroy the control system which was supposed to halt it. This would happen if we failed to solve any of many distinct problems that we don’t know how to solve yet, like symbol grounding, analogical and metaphorical processing and 3 more complex ones that I can think of, but don’t want to spend resources in this thread mentioning. Same for simulation argument.
Having a system that stops when you reach a level of tested intelligence sounds appealing, but I’d be afraid of the measure of intelligence at hand being too fuzzy.
So the system would not detect it had achieved that level of intelligence, and it would bootstrap and take off in time to destroy the control system which was supposed to halt it. This would happen if we failed to solve any of many distinct problems that we don’t know how to solve yet, like symbol grounding, analogical and metaphorical processing and 3 more complex ones that I can think of, but don’t want to spend resources in this thread mentioning. Same for simulation argument.