The outer optimizer is the more obvious thing: it’s straightforward to say there’s a big difference in dealing with a superhuman Oracle AI with only the goal of answering each question accurately, versus one whose goals are only slightly different from that in some way.
GPT generates text by repeatedly picking whatever word seems highest probability given all the words that came before. So if its notion of “highest probability” is almost, but not quite, answering every question accurately, I would expect a system which usually answers questions accurately but sometimes answers them inaccurately. That doesn’t sound very scary?
GPT generates text by repeatedly picking whatever word seems highest probability given all the words that came before. So if its notion of “highest probability” is almost, but not quite, answering every question accurately, I would expect a system which usually answers questions accurately but sometimes answers them inaccurately. That doesn’t sound very scary?