then you’re not guaranteed that your AI gets anywhere at all; its knightian uncertainty might remain so immense that the AI keeps picking the null action all the time because some of its knightian hypotheses still say that anything else is a bad idea.
@Tamsin, The knightian in IB is related to limits of what hypotheses you can possibly find/write down, not—if i understand so far—about an adversary. The adversary stuff is afaict mostly to make proofs work.
@all, anyway the big issue I (still) have with this is still that, if the user is trying to give these statements, how bad is it if they screw up in some nonobvious fundamental way? Does this prior instantly collapse if the user is a kinda bad predictor on some important subset of logic or makes only statements that aren’t particularly connected to some part of the statements needed to describe reality?
I’d be particularly interested to see Garrabrant, Kosoy, Diffractor, Gurkenglas comment on where they think this works or doesn’t
The knightian in IB is related to limits of what hypotheses you can possibly find/write down, not—if i understand so far—about an adversary. The adversary stuff is afaict mostly to make proofs work.
I don’t think this makes a difference here? If you say “what’s the best not-blacklisted-by-any-knightian-hypothesis action”, then it doesn’t really matter if you’re thinking of your knightian hypotheses as adversaries trying to screw you over by blacklisting actions that are fine, or if you’re thinking of your knightian hypotheses as a more abstract worst-case-scenario. In both cases, for any reasonable action, there’s probly a knightian hypothesis which blacklists it.
Regardless of whether you think of it as “because adversaries” or just “because we’re cautious”, knightian uncertainty works the same way. The issue is fundamental to doing maximin over knightian hypotheses.
@Tamsin, The knightian in IB is related to limits of what hypotheses you can possibly find/write down, not—if i understand so far—about an adversary. The adversary stuff is afaict mostly to make proofs work.
@all, anyway the big issue I (still) have with this is still that, if the user is trying to give these statements, how bad is it if they screw up in some nonobvious fundamental way? Does this prior instantly collapse if the user is a kinda bad predictor on some important subset of logic or makes only statements that aren’t particularly connected to some part of the statements needed to describe reality?
I’d be particularly interested to see Garrabrant, Kosoy, Diffractor, Gurkenglas comment on where they think this works or doesn’t
I don’t think this makes a difference here? If you say “what’s the best not-blacklisted-by-any-knightian-hypothesis action”, then it doesn’t really matter if you’re thinking of your knightian hypotheses as adversaries trying to screw you over by blacklisting actions that are fine, or if you’re thinking of your knightian hypotheses as a more abstract worst-case-scenario. In both cases, for any reasonable action, there’s probly a knightian hypothesis which blacklists it.
Regardless of whether you think of it as “because adversaries” or just “because we’re cautious”, knightian uncertainty works the same way. The issue is fundamental to doing maximin over knightian hypotheses.