I think I disagree. This is a bandit problem, and grantmakers have tried pulling that lever a bunch of times. There hasn’t been any field-changing research (yet). They knew it had a low chance of success so it’s not a big update. But it is a small update.
Probably the optimal move isn’t cutting early-career support entirely, but having a higher bar seems correct. There are other levers that are worth trying, and we don’t have the resources to try every lever.
Also there are more grifters now that the word is out, so the EV is also declining that way.
(I feel bad saying this as someone who benefited a lot from early-career financial support).
grantmakers have tried pulling that lever a bunch of times
What do you mean by this? I can think of lots of things that seem in some broad class of pulling some lever that kinda looks like this, but most of the ones I’m aware of fall greatly short of being an appropriate attempt to leverage smart young creative motivated would-be AGI alignment insight-havers. So the update should be much smaller (or there’s a bunch of stuff I’m not aware of).
The main thing I’m referring to are upskilling or career transition grants, especially from LTFF, in the last couple of years. I don’t have stats, I’m assuming there were a lot given out because I met a lot of people who had received them. Probably there were a bunch given out by the ftx future fund also.
Also when I did MATS, many of us got grants post-MATS to continue our research. Relatively little seems to have come of these.
How are they falling short?
(I sound negative about these grants but I’m not, and I do want more stuff like that to happen. If I were grantmaking I’d probably give many more of some kinds of safety research grant. But “If a man has an idea just give him money and don’t ask questions” isn’t the right kind of change imo).
upskilling or career transition grants, especially from LTFF, in the last couple of years
Interesting; I’m less aware of these.
How are they falling short?
I’ll answer as though I know what’s going on in various private processes, but I don’t, and therefore could easily be wrong. I assume some of these are sort of done somewhere, but not enough and not together enough.
Favor insightful critiques and orientations as much as constructive ideas. If you have a large search space and little traction, a half-plane of rejects is as or more valuable than a guessed point that you knew how to even generate.
Explicitly allow acceptance by trajectory of thinking, assessed by at least a year of low-bandwidth mentorship; deemphasize agenda-ish-ness.
For initial exploration periods, give longer commitments with less required outputs; something like at least 2 years. Explicitly allow continuation of support by trajectory.
Give a path forward for financial support for out of paradigm things. (The Vitalik fellowship, for example, probably does not qualify, as the professors, when I glanced at the list, seem unlikely to support this sort of work; but I could be wrong.)
Generally emphasize judgement of experienced AGI alignment researchers, and deemphasize judgement of grantmakers.
Explicitly asking for out of paradigm things.
Do a better job of connecting people. (This one is vague but important.)
(TBC, from my full perspective this is mostly a waste because AGI alignment is too hard; you want to instead put resources toward delaying AGI, trying to talk AGI-makers down, and strongly amplifying human intelligence + wisdom.)
I agree this would be a great program to run, but I want to call it a different lever to the one I was referring to.
The only thing I would change is that I think new researchers need to understand the purpose and value of past agent foundations research. I spent too long searching for novel ideas while I still misunderstood the main constraints of alignment. I expect you’d get a lot of wasted effort if you asked for out-of-paradigm ideas. Instead it might be better to ask for people to understand and build on past agent foundations research, then gradually move away if they see other pathways after having understood the constraints. Now I see my work as mostly about trying to run into constraints for the purpose of better understand them.
Maybe that wouldn’t help though, it’s really hard to make people see the constraints.
We agree this is a crucial lever, and we agree that the bar for funding has to be in some way “high”. I’m arguing for a bar that’s differently shaped. The set of “people established enough in AGI alignment that they get 5 [fund a person for 2 years and maybe more depending how things go in low-bandwidth mentorship, no questions asked] tokens” would hopefully include many people who understand that understanding constraints is key and that past research understood some constraints.
build on past agent foundations research
I don’t really agree with this. Why do you say this?
a lot of wasted effort if you asked for out-of-paradigm ideas.
I agree with this in isolation. I think some programs do state something about OOP ideas, and I agree that the statement itself does not come close to solving the problem.
(Also I’m confused about the discourse in this thread (which is fine), because I thought we were discussing “how / how much should grantmakers let the money flow”.)
would hopefully include many people who understand that understanding constraints is key and that past research understood some constraints.
Good point, I’m convinced by this.
build on past agent foundations research
I don’t really agree with this. Why do you say this?
That’s my guess at the level of engagement required to understand something. Maybe just because when I’ve tried to use or modify some research that I thought I understood, I always realise I didn’t understand it deeply enough. I’m probably anchoring too hard on my own experience here, other people often learn faster than me.
(Also I’m confused about the discourse in this thread (which is fine), because I thought we were discussing “how / how much should grantmakers let the money flow”.)
I was thinking “should grantmakers let the money flow to unknown young people who want a chance to prove themselves.”
That’s my guess at the level of engagement required to understand something. Maybe just because when I’ve tried to use or modify some research that I thought I understood, I always realise I didn’t understand it deeply enough. I’m probably anchoring too hard on my own experience here, other people often learn faster than me.
Hm. A couple things:
Existing AF research is rooted in core questions about alignment.
Existing AF research, pound for pound / word for word, and even idea for idea, is much more unnecessary stuff than necessary stuff. (Which is to be expected.)
Existing AF research is among the best sources of compute-traces of trying to figure some of this stuff out (next to perhaps some philosophy and some other math).
Empirically, most people who set out to stuff existing AF fail to get many of the deep lessons.
There’s a key dimension of: how much are you always asking for the context? E.g.: Why did this feel like a mainline question to investigate? If we understood this, what could we then do / understand? If we don’t understand this, are we doomed / how are we doomed? Are there ways around that? What’s the argument, more clearly?
It’s more important whether people are doing that, than whether / how exactly they engage with existing AF research.
If people are doing that, they’ll usually migrate away from playing with / extending existing AF, towards the more core (more difficult) problems.
I was thinking “should grantmakers let the money flow to unknown young people who want a chance to prove themselves.”
Ah ok you’re right that that was the original claim. I mentally autosteelmanned.
I think I disagree. This is a bandit problem, and grantmakers have tried pulling that lever a bunch of times. There hasn’t been any field-changing research (yet). They knew it had a low chance of success so it’s not a big update. But it is a small update.
Probably the optimal move isn’t cutting early-career support entirely, but having a higher bar seems correct. There are other levers that are worth trying, and we don’t have the resources to try every lever.
Also there are more grifters now that the word is out, so the EV is also declining that way.
(I feel bad saying this as someone who benefited a lot from early-career financial support).
What do you mean by this? I can think of lots of things that seem in some broad class of pulling some lever that kinda looks like this, but most of the ones I’m aware of fall greatly short of being an appropriate attempt to leverage smart young creative motivated would-be AGI alignment insight-havers. So the update should be much smaller (or there’s a bunch of stuff I’m not aware of).
The main thing I’m referring to are upskilling or career transition grants, especially from LTFF, in the last couple of years. I don’t have stats, I’m assuming there were a lot given out because I met a lot of people who had received them. Probably there were a bunch given out by the ftx future fund also.
Also when I did MATS, many of us got grants post-MATS to continue our research. Relatively little seems to have come of these.
How are they falling short?
(I sound negative about these grants but I’m not, and I do want more stuff like that to happen. If I were grantmaking I’d probably give many more of some kinds of safety research grant. But “If a man has an idea just give him money and don’t ask questions” isn’t the right kind of change imo).
Interesting; I’m less aware of these.
I’ll answer as though I know what’s going on in various private processes, but I don’t, and therefore could easily be wrong. I assume some of these are sort of done somewhere, but not enough and not together enough.
Favor insightful critiques and orientations as much as constructive ideas. If you have a large search space and little traction, a half-plane of rejects is as or more valuable than a guessed point that you knew how to even generate.
Explicitly allow acceptance by trajectory of thinking, assessed by at least a year of low-bandwidth mentorship; deemphasize agenda-ish-ness.
For initial exploration periods, give longer commitments with less required outputs; something like at least 2 years. Explicitly allow continuation of support by trajectory.
Give a path forward for financial support for out of paradigm things. (The Vitalik fellowship, for example, probably does not qualify, as the professors, when I glanced at the list, seem unlikely to support this sort of work; but I could be wrong.)
Generally emphasize judgement of experienced AGI alignment researchers, and deemphasize judgement of grantmakers.
Explicitly asking for out of paradigm things.
Do a better job of connecting people. (This one is vague but important.)
(TBC, from my full perspective this is mostly a waste because AGI alignment is too hard; you want to instead put resources toward delaying AGI, trying to talk AGI-makers down, and strongly amplifying human intelligence + wisdom.)
I agree this would be a great program to run, but I want to call it a different lever to the one I was referring to.
The only thing I would change is that I think new researchers need to understand the purpose and value of past agent foundations research. I spent too long searching for novel ideas while I still misunderstood the main constraints of alignment. I expect you’d get a lot of wasted effort if you asked for out-of-paradigm ideas. Instead it might be better to ask for people to understand and build on past agent foundations research, then gradually move away if they see other pathways after having understood the constraints. Now I see my work as mostly about trying to run into constraints for the purpose of better understand them.
Maybe that wouldn’t help though, it’s really hard to make people see the constraints.
We agree this is a crucial lever, and we agree that the bar for funding has to be in some way “high”. I’m arguing for a bar that’s differently shaped. The set of “people established enough in AGI alignment that they get 5 [fund a person for 2 years and maybe more depending how things go in low-bandwidth mentorship, no questions asked] tokens” would hopefully include many people who understand that understanding constraints is key and that past research understood some constraints.
I don’t really agree with this. Why do you say this?
I agree with this in isolation. I think some programs do state something about OOP ideas, and I agree that the statement itself does not come close to solving the problem.
(Also I’m confused about the discourse in this thread (which is fine), because I thought we were discussing “how / how much should grantmakers let the money flow”.)
Good point, I’m convinced by this.
That’s my guess at the level of engagement required to understand something. Maybe just because when I’ve tried to use or modify some research that I thought I understood, I always realise I didn’t understand it deeply enough. I’m probably anchoring too hard on my own experience here, other people often learn faster than me.
I was thinking “should grantmakers let the money flow to unknown young people who want a chance to prove themselves.”
Hm. A couple things:
Existing AF research is rooted in core questions about alignment.
Existing AF research, pound for pound / word for word, and even idea for idea, is much more unnecessary stuff than necessary stuff. (Which is to be expected.)
Existing AF research is among the best sources of compute-traces of trying to figure some of this stuff out (next to perhaps some philosophy and some other math).
Empirically, most people who set out to stuff existing AF fail to get many of the deep lessons.
There’s a key dimension of: how much are you always asking for the context? E.g.: Why did this feel like a mainline question to investigate? If we understood this, what could we then do / understand? If we don’t understand this, are we doomed / how are we doomed? Are there ways around that? What’s the argument, more clearly?
It’s more important whether people are doing that, than whether / how exactly they engage with existing AF research.
If people are doing that, they’ll usually migrate away from playing with / extending existing AF, towards the more core (more difficult) problems.
Ah ok you’re right that that was the original claim. I mentally autosteelmanned.