The paper starts with the assumption that humans will create many AI-agents and assign some of them selfish goals and that combined with competitive pressure and other factors may presumably create a Molochy -situation where most selfish and immoral AI’s will propagate and evolve—leading to loss of control and downfall of the human race. The paper in fact does not advocate the idea of a single AI foom. While the paper itself makes some valid points it does not answer my initial question and critique of OP.
Plenty of humans will give their AIs explicit goals. Evidence: plenty of humans do so now. Sure, purely self-supervised models are safer than people here were anticipating, and those of us who saw that coming and were previously laughed out of town are now vindicated. But that does not mean we’re safe, it just means that wasn’t enough to build a desperation bomb, a superreplicator that can actually eat, in the literal sense of the word, the entire world. that is what we’re worried about—AI causing a sudden jump in the competitive fitness of hypersimple life. It’s not quite as easy as some have anticipated, sure, but it’s very permitted by physics.
The question as stated was: But WHY would the AGI “want” anything at all unless humans gave it a goal(/s)?
ok how’s this then https://arxiv.org/abs/2303.16200
The paper starts with the assumption that humans will create many AI-agents and assign some of them selfish goals and that combined with competitive pressure and other factors may presumably create a Molochy -situation where most selfish and immoral AI’s will propagate and evolve—leading to loss of control and downfall of the human race. The paper in fact does not advocate the idea of a single AI foom. While the paper itself makes some valid points it does not answer my initial question and critique of OP.
Fair enough.