But seriously, it definitely has a lot of unwarranted confidence in its accomplishments.
I guess the connection to the real world is what will throw off such systems until they are trained on more real-world-like data.
I wouldn’t phrase it that it needs to be trained on more data. More like it needs to be retrained within an actual R&D loop. Have it actually write and execute its own code, test its hypotheses, evaluate the results, and iterate. Use RLHF to evaluate its assessments and a debugger to evaluate its code. It doesn’t matter whether this involves interacting with the “real world,” only that it learns to make its beliefs pay rent.
Anyway, that would help with its capabilities in this area, but it might be just a teensy bit dangerous to teach an LLM to do R&D like this without putting it in an air-gapped virtual sandbox, unless you can figure out how to solve alignment first.
You heard the LLM, alignment is solved!
But seriously, it definitely has a lot of unwarranted confidence in its accomplishments.
I wouldn’t phrase it that it needs to be trained on more data. More like it needs to be retrained within an actual R&D loop. Have it actually write and execute its own code, test its hypotheses, evaluate the results, and iterate. Use RLHF to evaluate its assessments and a debugger to evaluate its code. It doesn’t matter whether this involves interacting with the “real world,” only that it learns to make its beliefs pay rent.
Anyway, that would help with its capabilities in this area, but it might be just a teensy bit dangerous to teach an LLM to do R&D like this without putting it in an air-gapped virtual sandbox, unless you can figure out how to solve alignment first.
I agree. A while back, I asked Does non-access to outputs prevent recursive self-improvement? I think that letting such systems learn from experiments with the real world is very dangerous.