Matt Putz

• Matt PutzJan 7, 2025, 3:07 PM

  1 point

  0

  in reply to: ryan_greenblatt’s comment on: AI Timelines

  I’ve updated toward the views Daniel expresses here and I’m now about half way between Ajeya’s views in this post and Daniel’s (in geometric mean).

  I’m curious what the biggest factors were that made you update?

• Matt PutzNov 22, 2024, 8:32 PM

  9 points

  3

  in reply to: habryka’s comment on: lcmgcd’s Shortform

  Regarding our career development and transition funding (CDTF) program:

  • The default expectation for CDTF grants is that they’re one-off grants. My impression is that this is currently clear to most CDTF grantees (e.g., I think most of them don’t reapply after the end of their grant period, and the program title explicitly says that it’s “transition funding”).

    • (When funding independent research through this program, we sometimes explicitly clarify that we’re unlikely to renew by default).

  • Most of the CDTF grants we make have grant periods that are shorter than a year (with the main exception that comes to mind being PhD programs). I think that’s reasonable (esp. given that the grantees know this when they accept the funding). I’d guess most of the people we fund through this program are able to find paid positions after <1 year.

  (I probably won’t have time to engage further.)

• Matt PutzNov 21, 2024, 10:34 PM

  24 points

  4

  in reply to: lemonhope’s comment on: lcmgcd’s Shortform

  Just wanted to flag quickly that Open Philanthropy’s GCR Capacity Building team (where I work) has a career development and transition funding program.

  The program aims to provide support—in the form of funding for graduate study, unpaid internships, self-study, career transition and exploration periods, and other activities relevant to building career capital—for individuals at any career stage who want to pursue careers that could help reduce global catastrophic risks (esp. AI risks). It’s open globally and operates on a rolling basis.

  I realize that this is quite different from what lemonhope is advocating for here, but nevertheless thought it would be useful context for this discussion (and potential applicants).

• Matt PutzNov 8, 2024, 1:16 AM

  7 points

  0

  in reply to: Austin Chen’s comment on: 5 home­grown EA pro­jects, seek­ing small donors

  Thanks for the feedback! I’ll forward it to our team.

  I think I basically agree with you that from reading the RFP page, this project doesn’t seem like a central example of the projects we’re describing (and indeed, many of the projects we do fund through this RFP are more like the examples given on the RFP page).

  Some quick reactions:

  • FWIW, our team generally makes a lot of grants that are <$100k (much more so than other Open Phil teams).

  • I agree the application would probably take most people longer than the description that Gavin gave on Manifund. That said, I think it’s still relatively lean considering the distribution of projects we fund, though I agree it’s slightly long for projects as small as this one (but I think Gavin could have filled it out in <<2 days). For reference, this is our form.

  • Regarding turnaround time, my guess is for this project, we would have taken significantly less than 3 months, especially if they had indicated that receiving a decision was time-sensitive. For reference, the form currently says:

  We expect to make most funding decisions in 3 months or less (assuming prompt responses to any follow-up questions we may have), and we may or may not be able to accommodate requests for greater time-sensitivity. Applicants asking for over $500K should expect a decision to take the full 3 months (or more, in particularly complex cases), and apply in advance accordingly. We’ll let you know as soon as we can if we anticipate a longer than 3-month decision timeline. [emphasis in original]

  • For $500k+ projects, I think a 3-month turnaround time is more defensible, though I do personally wish we generally had faster response times.

• Matt PutzNov 6, 2024, 8:21 PM

  10 points

  0

  on: 5 home­grown EA pro­jects, seek­ing small donors

  I work at Open Philanthropy, and I recently let Gavin know that Open Phil is planning to recommend a grant of $5k to Arb for the second project on your list: Overview of AI Safety in 2024 (they had already raised ~$10k by the time we came across it). Thanks for writing this post Austin — it brought the funding opportunity to our attention.

  Like other commenters on Manifund, I believe this kind of overview is a valuable reference for the field, especially for newcomers.

  I wanted to flag that this project would have been eligible for our RFP for work that builds capacity to address risks from transformative AI. I worry that not all potential applicants are aware of the RFP or its scope, so I’ll take this opportunity to mention that this RFP’s scope is quite broad, including funding for:

  • Training and mentorship programs

  • Events

  • Groups

  • Resources, media, and communications

  • Almost any other type of project that builds capacity for advanced AI risks (in the sense of increasing the number of careers devoted to these problems, supporting people doing this work, and sharing knowledge related to this work).

  More details at the link above. People might also find this page helpful, which lists all currently open application programs at Open Phil.

• Matt PutzJan 2, 2023, 10:17 PM

  1 point

  0

  on: Scor­ing Rules and Au­to­mated Mar­ket Makers

  There are two very similar pages. This one and https://​​www.lesswrong.com/​​tag/​​scoring-rules/​​

Why we’re not found­ing a hu­man-data-for-al­ign­ment org

• Matt PutzJul 22, 2022, 1:01 PM

  1 point

  0

  in reply to: Ajeya Cotra’s comment on: Without spe­cific coun­ter­mea­sures, the eas­iest path to trans­for­ma­tive AI likely leads to AI takeover

  By “refining pure human feedback”, do you mean refining RLHF ML techniques?

  I assume you still view enhancing human feedback as valuable? And also more straightforwardly just increasing the quality of the best human feedback?

• Matt PutzJun 23, 2022, 8:02 AM

  4 points

  on: An­nounc­ing the LessWrong Cu­rated Podcast

  Amazing! Thanks so much for making this happen so quickly.

  To anyone who’s trying to figure out how to get it to work on Google Podcasts, here’s what worked for me (searching the name didn’t, maybe this will change?):

  Go to the Libsyn link. Click the RSS symbol. Copy the link. Go to Google Podcasts. Click the Library tab (bottom right). Go to Subscriptions. Click symbol that looks like adding a link in the upper right. Paste link, confirm.

• Matt PutzMar 3, 2022, 4:32 PM

  1 point

  in reply to: paulfchristiano’s comment on: Late 2021 MIRI Con­ver­sa­tions: AMA /​ Discussion

  Hey Paul, thanks for taking the time to write that up, that’s very helpful!

• Matt PutzMar 3, 2022, 4:27 PM

  3 points

  in reply to: Rohin Shah’s comment on: Late 2021 MIRI Con­ver­sa­tions: AMA /​ Discussion

  Hey Rohin, thanks a lot, that’s genuinely super helpful. Drawing analogies to “normal science” seems both reasonable and like it clears the picture up a lot.

• Matt PutzMar 2, 2022, 1:44 PM

  11 points

  on: Late 2021 MIRI Con­ver­sa­tions: AMA /​ Discussion

  I would be interested to hear opinions about what fraction of people could possibly produce useful alignment work?

  Ignoring the hurdle of “knowing about AI safety at all”, i.e. assuming they took some time to engage with it (e.g. they took the AGI Safety Fundamentals course). Also assume they got some good mentorship (e.g. from one of you) and then decided to commit full-time (and got funding for that). The thing I’m trying to get at is more about having the mental horsepower + epistemics + creativity + whatever other qualities are useful, or likely being able to get there after some years of training.

  Also note that I mean direct useful work, not indirect meta things like outreach or being a PA to a good alignment researcher etc. (these can be super important, but I think it’s productive to think of them as a distinct class). E.g. I would include being a software engineer at Anthropic, but exclude doing grocery-shopping for your favorite alignment researcher.

  An answer could look like “X% of the general population” or “half the people who could get a STEM degree at Ivy League schools if they tried” or “a tenth of the people who win the Fields medal”.

  I think it’s useful to have a sense of this for many purposes, incl. questions about community growth and the value of outreach in different contexts, as well as priors about one’s own ability to contribute. Hence, I think it’s worth discussing honestly, even though it can obviously be controversial (with some possible answers implying that most current AI safety people are not being useful).

• Matt PutzFeb 27, 2022, 12:13 AM

  2 points

  in reply to: Eli Tyre’s comment on: Twit­ter thread on postrationalists

  That’s a very detailed answer, thanks! I’ll have a look at some of those tools. Currently I’m limiting my use to a particular 10-minute window per day with freedom.to + the app BlockSite. It often costs me way more than 10 minutes (checking links after, procrastinating before...) of focus though, so I might try to find an alternative.

• Matt PutzFeb 19, 2022, 8:05 PM

  4 points

  on: Twit­ter thread on postrationalists

  Sorry for the tangent, but how do you recommend engaging with Twitter, without it being net bad?

• Matt PutzJan 21, 2022, 12:29 PM

  1 point

  in reply to: paulfchristiano’s comment on: Prizes for ELK proposals

  Thanks! Great to hear that it’s going well!

• Matt PutzJan 16, 2022, 2:15 PM

  7 points

  on: Prizes for ELK proposals

  Maybe I’m being stupid here. On page 42 of the write-up, it says:
  

  In order to ensure we learned the human simulator, we would need to change the training strategy to ensure that it contains sufficiently challenging inference problems, and that doing direct translation was a cost-effective way to improve speed (i.e. that there aren’t other changes to the human simulator that would save even more time). [emphasis mine]

  Shouldn’t that be?

  In order to ensure we learned the direct translator, …

• Matt PutzJan 14, 2022, 3:37 PM

  16 points

  on: Prizes for ELK proposals

  We’re planning to evaluate submissions as we receive them, between now and the end of January; we may end the contest earlier or later if we receive more or fewer submissions than we expect.

  Just wanted to note that the “we may end the contest earlier” part here makes me significantly more hesitant about trying this. I will probably still at least have a look at it, but part of me is afraid that I’ll invest a bunch of time and then the contest will be announced to be over before I got around to submitting. And I suspect Holden’s endorsement may make that more likely. It would be easier for me to invest time spread out over the next couple of weeks, than all in one go, due to other commitments. On the other hand, if I knew there was a hard deadline next Friday, I might try to find a way to squeeze it in.
  
  I’m just pointing this out in case you hadn’t thought of it. I suspect something similar might be true for others too. Of course, it’s your prize and your rules, and if you prefer it this way, that’s totally fine.

• Matt PutzSep 4, 2021, 9:57 PM

  3 points

  in reply to: Steven Byrnes’s comment on: Re­ward Is Not Enough

  Sorry for my very late reply!

  Thanks for taking the time to answer, I now don’t endorse most of what I wrote anymore.

  I think that if the AGI has a perfect motivation system then we win, there’s no safety problem left to solve. (Well, assuming it also remains perfect over time, as it learns new things and thinks new thoughts.) (See here for the difference between motivation and reward.)

  and from the post:

  And if we get to a point where we can design reward signals that sculpt an AGI’s motivation with surgical precision, that’s fine!

  This is mostly where I went wrong. I.e. I assumed a perfect reward signal coming from some external oracle in examples where your entire point was that we didn’t have a perfect reward signal (e.g. wireheading).

  So basically, I think we agree: a perfect reward signal may be enough in principle, but in practice it will not be perfect and may not be enough. At least not a single unified reward.

• Matt PutzJun 22, 2021, 8:23 PM

  1 point

  on: Re­ward Is Not Enough

  Interesting post!

  Disclaimer: I have only read the abstract of the “Reward is enough” paper. Also, I don’t have much experience in AI safety, but I consider changing that.

  Here are a couple of my thoughts.

  Your examples haven’t entirely convinced me that reward isn’t enough. Take the bird. As I see it, something like the following is going on:

  Evolution chose to take a shortcut: maybe a bird with a very large brain and a lot of time would eventually figure out that singing is a smart thing to do if it received reward for singing well. But evolution being a ruthless optimizer with many previous generations of experience, shaped two separate rewards in the way you described. Silver et al.’s point might be that when building an AGI, we wouldn’t have to take that shortcut, at least not by handcoding it.
  Assume we have an agent that is released into the world and is trying to optimize reward. It starts out from scratch, knowing nothing, but with a lot of time and the computational capacity to learn a lot.
  Such an agent has an incentive to explore. So it tries out singing for two minutes. It notes that in the first minute it got 1 unit of reward and in the second 2 (it got better!). All in all, 3 units is very little in this world however, so maybe it moves on.
  But as it gains more experience in the world it notices that patterns like these can often be extrapolated. Maybe, with its two minutes of experience, if it sang for a third minute, it would get 3 units of reward? It tries and yes indeed. Now it has an incentive to see how far it can take this. It knows the 4 units it expects from the next try will not be worth its time on their own, but the information of whether it could eventually get a million units per minute this way is very much worth the cost!

  Something kind of analogous should be true for the spider story.
  Reward very much provides an incentive for the agent to eventually figure out that after encountering a threat, it should change its behavior, not its interpretations of the world. At the beginning it might get this wrong, but it’s unfair to compare it to a human who has had this info “handcoded in” by evolution.
  If our algorithms don’t allow you to learn to update differently in the future, because past update were unhelpful (I don’t know, pointers welcome!), then that’s not a problem with reward, it’s a problem with our algorithms!
  Maybe this is what you alluded to in your very last paragraph where you speculated that they might just mean a more sophisticated RL algorithm?

  Concerning the deceptive AGI etc., I agree problems emerge when we don’t get the reward signal exactly right and that it’s probably not a safe assumption that we will. But it might still be an interesting question how things would go assuming a perfect reward signal?
  My impression is that their answer is “it would basically work”, while yours is something like “but we really shouldn’t assume that and if we don’t, then it’s probably better to have separate reward signals etc.”. Given the bird example, I assume you also don’t agree that things would work out fine even if we did have the best possible reward signal?

  Also, I just want to mention that I agree the distinction between within-lifetime RL and intergenerational RL is useful, certainly in the case of biology and probably in machine learning too.

• Matt PutzJun 4, 2020, 4:11 PM

  1 point

  in reply to: habryka’s comment on: CFAR Par­ti­ci­pant Hand­book now available to all

  Alright, that’s helpful! Thanks!