That sounds way less scary when you consider actual software that is approaching recursive self improvement and get more specific than vague “increase in power”. It’s just generic ignorant anti-technology talk that relies on vague concepts like “power” and dissipates once you get in any way specific.
The software also tends not to do what you want it to do for sake of this argument. There’s an enormous gap between ‘not doing exactly what we want’ and doing exactly what you want for this argument to work. The automated engineering software simulates microscopic material interaction; vague self improvement and increases in power only make it better at not doing unrelated stuff.
That sounds way less scary when you consider actual software that is approaching recursive self improvement and get more specific than vague “increase in power”. It’s just generic ignorant anti-technology talk that relies on vague concepts like “power” and dissipates once you get in any way specific.
The software also tends not to do what you want it to do for sake of this argument. There’s an enormous gap between ‘not doing exactly what we want’ and doing exactly what you want for this argument to work. The automated engineering software simulates microscopic material interaction; vague self improvement and increases in power only make it better at not doing unrelated stuff.