Lots of search time alone does NOT indicate extremal results—it indicates lots of things about your domain, and perhaps the inefficiency of your search, but not overoptimization.
Thoughts on early stopping? (Maybe it works because if you keep optimizing long enough, you’re liable to find yourself on a tall and narrow peak which generalizes poorly? However it seems like maybe the phenomenon doesn’t just apply to gradient descent?)
BTW, I suspect there is only so much you can do with abstractions like these. At the end of the day, any particular concrete technique may not exhibit the flaws you predicted based on the abstract category you placed it in, and it may exhibit flaws which you wouldn’t have predicted based on its abstract category. Maybe abstract categories are best seen as brainstorming tools for finding flaws in techniques.
I agree that these abstractions are very limited and should mainly be used to raise concerns. Due to existential risk, there’s an asymmetry between concerns vs positive arguments against concerns: if we want to avoid large negative outcomes, we have to take vague concerns seriously in the absence of good arguments against them; but, asymmetrically, seek strong arguments that systems avoid risk. Recently I worry that this can give people an incorrect picture of which ideas I think are firm enough to take seriously. I’ll happily discuss fairly vague ideas such as instrumental convergence when talking about risk. But (at least for me personally) the very same ideas will seem overly vague and suspicious, if used in an argument that things will go well. I think this is basically the right attitude to take, but could be confusing to other people.
This seems related to a comment Rohin made recently. It sounds like you are working from Rohin’s “normative claim”, not his “empirical claim”? (From an empirical perspective, holding arguments for ¬A to a higher standard than arguments for A is obviously a great way to end up with false beliefs :P)
Anyway, just like Rohin, I’m uncertain re: the normative claim. But even if one believes the normative claim, I think in some cases a concern can be too vague to be useful.
Here’s an extreme example to make the point. Biotech research also presents existential risks. Suppose I object to your biotech strategy, on the grounds that you don’t have a good argument that your strategy is robust against adversarial examples.
What does it even mean for a biotech strategy to be robust against adversarial examples?
Without further elaboration, my concern re: your biotech strategy is too vague. Trying to come up with a good argument against my concern would be a waste of your time.
Maybe there is a real problem here. But our budget of research hours is limited. If we want to investigate this further, the thing to do make the concern less vague, and get more precise about the sense in which your biotech strategy is vulnerable to adversarial examples.
I agree vague concerns should be taken seriously. But I think in some cases, we will ultimately dismiss the concern not because we thought of a strong argument against it, but because multiple people thought creatively about how it might apply and just weren’t able to find anything.
You can’t prove things about something which hasn’t been formalized. And good luck formalizing something without any concrete examples of it! Trying to offer strong arguments against a concern that is still vague seems like putting the cart before the horse.
I don’t think FAI work should be overly guided by vague analogies, not because I’m unconcerned about UFAI, but because vague analogies just don’t provide much evidence about the world. Especially if there’s a paucity of data to inform our analogizing.
It’s possible that I’m talking past you a bit in this comment, so to clarify: I don’t think instrumental convergence is too vague to be useful. But for some other concerns, such as daemons, I would argue that the most valuable contribution at this point is trying to make the concern more concrete.
Thoughts on early stopping? (Maybe it works because if you keep optimizing long enough, you’re liable to find yourself on a tall and narrow peak which generalizes poorly? However it seems like maybe the phenomenon doesn’t just apply to gradient descent?)
BTW, I suspect there is only so much you can do with abstractions like these. At the end of the day, any particular concrete technique may not exhibit the flaws you predicted based on the abstract category you placed it in, and it may exhibit flaws which you wouldn’t have predicted based on its abstract category. Maybe abstract categories are best seen as brainstorming tools for finding flaws in techniques.
Good point about early stopping.
I agree that these abstractions are very limited and should mainly be used to raise concerns. Due to existential risk, there’s an asymmetry between concerns vs positive arguments against concerns: if we want to avoid large negative outcomes, we have to take vague concerns seriously in the absence of good arguments against them; but, asymmetrically, seek strong arguments that systems avoid risk. Recently I worry that this can give people an incorrect picture of which ideas I think are firm enough to take seriously. I’ll happily discuss fairly vague ideas such as instrumental convergence when talking about risk. But (at least for me personally) the very same ideas will seem overly vague and suspicious, if used in an argument that things will go well. I think this is basically the right attitude to take, but could be confusing to other people.
This seems related to a comment Rohin made recently. It sounds like you are working from Rohin’s “normative claim”, not his “empirical claim”? (From an empirical perspective, holding arguments for ¬A to a higher standard than arguments for A is obviously a great way to end up with false beliefs :P)
Anyway, just like Rohin, I’m uncertain re: the normative claim. But even if one believes the normative claim, I think in some cases a concern can be too vague to be useful.
Here’s an extreme example to make the point. Biotech research also presents existential risks. Suppose I object to your biotech strategy, on the grounds that you don’t have a good argument that your strategy is robust against adversarial examples.
What does it even mean for a biotech strategy to be robust against adversarial examples?
Without further elaboration, my concern re: your biotech strategy is too vague. Trying to come up with a good argument against my concern would be a waste of your time.
Maybe there is a real problem here. But our budget of research hours is limited. If we want to investigate this further, the thing to do make the concern less vague, and get more precise about the sense in which your biotech strategy is vulnerable to adversarial examples.
I agree vague concerns should be taken seriously. But I think in some cases, we will ultimately dismiss the concern not because we thought of a strong argument against it, but because multiple people thought creatively about how it might apply and just weren’t able to find anything.
You can’t prove things about something which hasn’t been formalized. And good luck formalizing something without any concrete examples of it! Trying to offer strong arguments against a concern that is still vague seems like putting the cart before the horse.
I don’t think FAI work should be overly guided by vague analogies, not because I’m unconcerned about UFAI, but because vague analogies just don’t provide much evidence about the world. Especially if there’s a paucity of data to inform our analogizing.
It’s possible that I’m talking past you a bit in this comment, so to clarify: I don’t think instrumental convergence is too vague to be useful. But for some other concerns, such as daemons, I would argue that the most valuable contribution at this point is trying to make the concern more concrete.