If there are two research projects that are roughly equivalent, but one seems deep while the other seems boring, the deep one will garner more attention and interest. The spread and discovery of research ideas thus has a bias towards profound ideas, as profundity is more memetically fit than its absence.
This is baffling; why is this a “bias”? Why wouldn’t we expect the sense of deepness to strongly correlate with some relevant features of the projects? It seems good to strongly interrogate these intuitions, but it’s not like they’re meaningless intuitions that come from nowhere. If something seems deep, it touches on stuff that’s important and general, which we would expect to be important for alignment.
I think researchers looking to start projects in theoretical alignment should keep these issues in mind, and not necessarily expect this status quo to change in the near future. It may be more promising to consider other directions.
So I would agree with this, but I would amend it to: it may be more promising to consider other directions, and to try to recover from the sense of deepness some pointers at what seemed deep in the research projects. Like, if there’s a research project that seems deep, don’t just ignore that sense of deepness, but also don’t take it on faith that it’s a good research project as-is; instead, interrogate the research project especially critically, looking for the core of what’s actually deep and discarding the parts of the research project that were mistaken / irrelevant.
If something seems deep, it touches on stuff that’s important and general, which we would expect to be important for alignment.
The specific scenario I talk about in the paragraph you’re responding too is one where everything except for the sense of deepness is the same for both ideas, such that someone who doesn’t have a sense of what ideas are deep or profound would find the ideas basically equivalent. In such a scenario my argument is that we should expect the deep idea to receive a more attention, despite their not existing legible or well grounded reasons for this. Some amount of preference for the deep idea might be justifiable on the grounds of trusting intuitive insight, but I don’t think the record of intuitive insight as to what ideas are good is actually very impressive—there are a huge amount of ideas that didn’t work out that sounded deep (see some philosophy, psychoanalysis, ect.) and very few that did work out[1].
try to recover from the sense of deepness some pointers at what seemed deep in the research projects
I think on the margin new theoretical alignment researchers should do less of this, as I think most deep sounding ideas just genuinely aren’t very productive to research and aren’t amenable to being proven to be unproductive to work on—often times the only evidence that a deep idea isn’t productive to work on is that nothing concrete has come of it yet.
I don’t have empirical analysis showing this—I would probably gesture to various prior alignment research projects to support this if I had to, though I worry that would devolve into arguing about what ‘success’ meant.
The specific scenario I talk about in the paragraph you’re responding too is one where everything except for the sense of deepness is the same for both ideas, such that someone who doesn’t have a sense of what ideas are deep or profound would find the ideas basically equivalent.
But if that’s not what the distribution looks like, but rather the distribution looks like a strong correlation, then it’s not a bias, it’s just following what the distribution says. Maybe to shore up / expand on your argument, you’re talking about the optimizer’s curse: https://www.lesswrong.com/posts/5gQLrJr2yhPzMCcni/the-optimizer-s-curse-and-how-to-beat-it
So like, the most deep-seeming idea will tend to regress to the mean more than a random idea would regress. But this doesn’t argue to not pay attention to things that seem deep. (It argues for a portfolio approach, but there’s lots of arguments for a portfolio approach.)
Maybe another intuition you’re drawing on is information cascades. If there’s a lot of information cascades, then a lot of people are paying attention to a few very deep-seeming ideas. Which we can agree is dumb.
I think on the margin new theoretical alignment researchers should do less of this, as I think most deep sounding ideas just genuinely aren’t very productive to research and aren’t amenable to being proven to be unproductive to work on—often times the only evidence that a deep idea isn’t productive to work on is that nothing concrete has come of it yet.
I think this is pretty wrong, though it seems hard to resolve. I would guess that a lot of things that are later concretely productive started with someone hearing something that struck them as deep, and then chewing on it and transforming it.
This is baffling; why is this a “bias”? Why wouldn’t we expect the sense of deepness to strongly correlate with some relevant features of the projects? It seems good to strongly interrogate these intuitions, but it’s not like they’re meaningless intuitions that come from nowhere. If something seems deep, it touches on stuff that’s important and general, which we would expect to be important for alignment.
So I would agree with this, but I would amend it to: it may be more promising to consider other directions, and to try to recover from the sense of deepness some pointers at what seemed deep in the research projects. Like, if there’s a research project that seems deep, don’t just ignore that sense of deepness, but also don’t take it on faith that it’s a good research project as-is; instead, interrogate the research project especially critically, looking for the core of what’s actually deep and discarding the parts of the research project that were mistaken / irrelevant.
The specific scenario I talk about in the paragraph you’re responding too is one where everything except for the sense of deepness is the same for both ideas, such that someone who doesn’t have a sense of what ideas are deep or profound would find the ideas basically equivalent. In such a scenario my argument is that we should expect the deep idea to receive a more attention, despite their not existing legible or well grounded reasons for this. Some amount of preference for the deep idea might be justifiable on the grounds of trusting intuitive insight, but I don’t think the record of intuitive insight as to what ideas are good is actually very impressive—there are a huge amount of ideas that didn’t work out that sounded deep (see some philosophy, psychoanalysis, ect.) and very few that did work out[1].
I think on the margin new theoretical alignment researchers should do less of this, as I think most deep sounding ideas just genuinely aren’t very productive to research and aren’t amenable to being proven to be unproductive to work on—often times the only evidence that a deep idea isn’t productive to work on is that nothing concrete has come of it yet.
I don’t have empirical analysis showing this—I would probably gesture to various prior alignment research projects to support this if I had to, though I worry that would devolve into arguing about what ‘success’ meant.
But if that’s not what the distribution looks like, but rather the distribution looks like a strong correlation, then it’s not a bias, it’s just following what the distribution says. Maybe to shore up / expand on your argument, you’re talking about the optimizer’s curse: https://www.lesswrong.com/posts/5gQLrJr2yhPzMCcni/the-optimizer-s-curse-and-how-to-beat-it So like, the most deep-seeming idea will tend to regress to the mean more than a random idea would regress. But this doesn’t argue to not pay attention to things that seem deep. (It argues for a portfolio approach, but there’s lots of arguments for a portfolio approach.)
Maybe another intuition you’re drawing on is information cascades. If there’s a lot of information cascades, then a lot of people are paying attention to a few very deep-seeming ideas. Which we can agree is dumb.
I think this is pretty wrong, though it seems hard to resolve. I would guess that a lot of things that are later concretely productive started with someone hearing something that struck them as deep, and then chewing on it and transforming it.