But how good it can be, realistically? I will be so so much surprised if all this details wont be leaked in next week. May be they will try to make several false leaks to muddle things a bit.
Muyyd
Karma: 0
Lack of clarity when i think about this limits makes hard for me to see how end result will change if we could somehow “stop discounting” them.
It seems to my that we will have to be much more elaborete in describing parameters of this thought experiment. In particular we will have to agree on deeds and real world achivments that hypothetical AI has, so we will both agree to call it AGI (like writing interesting story and making illustrations so this particular research team now have a new revenue strem from selling it online—will this make AI an AGI?). And security conditions (air-gapped server-room?). This will get us closer to understanding “the rationale”.
But then your question is not about AGI but “superintelligent AI” so we will have to do elaborate describing again with new parameters. And that is what i expect Eliezer (alone and with other people) had done a lot. And look what it did to him (this is a joke but at the same time—not). So i will not be an active participant further.
It is not even about a single SAI in some box: compeeting teams, people running copies (legal and not) and changing code, corporate espionage, dirty code...
It is the very same rationale that stands behind assumptions like “why Stockfish won’t execute losing set of moves”—it is just that good at chess. Or better—it is just that smart when it come down to chess.
In this thought experiment the way to go is not to “i see that AGI could likely fail at this step, therefore it will fail” but to keep thinking and inventing better moves for AGI to execute, which won’t be countered as easily. It is an important part of “security mindset” and probably major reason why Eliezer speaks about lack of pessimism in the field.
Both times my talks went that way (why they did not raise him good—why we could not program AI to be good; cant we keep on eye on them, and so on), but it would take to long to summarise something like 10 minutes dialog, so i am not going to do this. Sorry.
Evolution: taste buds and ice cream, sex and condoms… This analogy always was difficult to use in my experience. A year ago i came up with less technical. KPIs (key performance indicators) as inevitable way to communicate goals (to AI) to ultra-high-IQ psycopath-genius who’s into malicious compliance (kinda cant help himself being clone of Nicola Tesla, Einstain and bunch of different people, some of them probably CEO, becouse she can).
I have used it only 2 times and it was way easier than talks about different optimisation processes. And it took me only something like 8 years to come up with!
Does ‘ethical safe AI investments’ means ‘to help make AI safer and make some money at the same time’?