I agree this is something I could do that would be valuable, I just haven’t prioritized it alas. I did say I wanted to write the sequel to that story at some point...
I encourage other people to think about it too. I suggest reasoning as follows: (1) What skills/properties would combine to create APS-AI? Agency, reasoning, situational awareness… maybe persuasion, deception, hacking… etc. (2) Does the training process of OmegaStar or whatever incentivize the growth of those skills? That is, does the net developing those skills help it to perform better in that training process, get reinforced, etc.? (3) Is the model big enough that it could in principle come to develop those skills? (4) Is the model trained long enough that it likely will in practice come to have those skills?
I agree this is something I could do that would be valuable, I just haven’t prioritized it alas. I did say I wanted to write the sequel to that story at some point...
I encourage other people to think about it too. I suggest reasoning as follows: (1) What skills/properties would combine to create APS-AI? Agency, reasoning, situational awareness… maybe persuasion, deception, hacking… etc. (2) Does the training process of OmegaStar or whatever incentivize the growth of those skills? That is, does the net developing those skills help it to perform better in that training process, get reinforced, etc.? (3) Is the model big enough that it could in principle come to develop those skills? (4) Is the model trained long enough that it likely will in practice come to have those skills?