I don’t know if LLM Ems can really be a significant factorizable part of the AI tech tree. If they have anything like today’s LLM limitations, they’re not as powerful as humans and ems. If they’re much more powerful than today’s LLMs, they’re likely to have powerful submodules that are qualitatively different from what we think of as LLMs.
I’m thinking of LLMs that are not necessarily more powerful than GPT-4, but have auxiliary routines for studyingspecific skills or topics that don’t automatically fall out of SSL and instead require deliberate practice (because there are currently no datasets that train them out of the box). This would make them AGI in a singularity-relevant sense, and shore up coherent agency, if it’s practiced as skills.
That doesn’t move them significantly above human level, and I suspect improving quality of their thinking (as opposed to depth of technical knowledge) might prove difficult without risking misalignment, because capabilities of LLM characters are borrowed from humans, not spun up from first principles. At this point, these are essentially people, human imitations, slightly alien but still mostly aligned ems, ready to destroy the world by making further AGI capability progress.
I guess that’s plausible, but then my main doom scenario would involve them getting leapfrogged by a different AI that has hit a rapid positive feedback loop of how to keep amplifying its consequentialist planning abilities.
my main doom scenario would involve them getting leapfrogged by a different AI
Mine as well, hence the reference to AGI capabilities at the end of my comment, though given the premise I expect them to build it, not us. But in the meantime, there’ll be great em cities.
I don’t know if LLM Ems can really be a significant factorizable part of the AI tech tree. If they have anything like today’s LLM limitations, they’re not as powerful as humans and ems. If they’re much more powerful than today’s LLMs, they’re likely to have powerful submodules that are qualitatively different from what we think of as LLMs.
I’m thinking of LLMs that are not necessarily more powerful than GPT-4, but have auxiliary routines for studying specific skills or topics that don’t automatically fall out of SSL and instead require deliberate practice (because there are currently no datasets that train them out of the box). This would make them AGI in a singularity-relevant sense, and shore up coherent agency, if it’s practiced as skills.
That doesn’t move them significantly above human level, and I suspect improving quality of their thinking (as opposed to depth of technical knowledge) might prove difficult without risking misalignment, because capabilities of LLM characters are borrowed from humans, not spun up from first principles. At this point, these are essentially people, human imitations, slightly alien but still mostly aligned ems, ready to destroy the world by making further AGI capability progress.
I guess that’s plausible, but then my main doom scenario would involve them getting leapfrogged by a different AI that has hit a rapid positive feedback loop of how to keep amplifying its consequentialist planning abilities.
Mine as well, hence the reference to AGI capabilities at the end of my comment, though given the premise I expect them to build it, not us. But in the meantime, there’ll be great em cities.