Lukas_Gloor

• Lukas_Gloor 17 Nov 2024 0:27 UTC

  31 points

  20

  in reply to: Seth Herd’s comment on: OpenAI Email Archives (from Musk v. Alt­man)

  I agree that it sounds somewhat premature to write off Larry Page based on attitudes he had a long time ago, when AGI seemed more abstract and far away, and then not seek/​try communication with him again later on. If that were Musk’s true and only reason for founding OpenAI, then I agree that this was a communication fuckup.
  
  However, my best guess is that this story about Page was interchangeable with a number of alternative plausible criticisms of his competition on building AGI that Musk would likely have come up with in nearby worlds. People like Musk (and Altman too) tend to have a desire to do the most important thing and the belief that they can do this thing a lot better than anyone else. On that assumption, it’s not too surprising that Musk found a reason for having to step in and build AGI himself. In fact, on this view, we should expect to see surprisingly little sincere exploration of “joining someone else’s project to improve it” solutions.

  I don’t think this is necessarily a bad attitude. Sometimes people who think this way are right in the specific situation. It just means that we see the following patterns a lot:

  • Ambitious people start their own thing rather than join some existing thing.

  • Ambitious people have fallouts with each other after starting a project together where the question of “who eventually gets de facto ultimate control” wasn’t totally specified from the start.

  (Edited away a last paragraph that used to be here 50mins after posting. Wanted to express something like “Sometimes communication only prolongs the inevitable,” but that sounds maybe a bit too negative because even if you’re going to fall out eventually, probably good communication can help make it less bad.)

• Lukas_Gloor 16 Nov 2024 17:45 UTC

  75 points

  44

  in reply to: Thane Ruthenis’s comment on: OpenAI Email Archives (from Musk v. Alt­man)

  I thought the part you quoted was quite concerning, also in the context of what comes afterwards:

  Hiatus: Sam told Greg and Ilya he needs to step away for 10 days to think. Needs to figure out how much he can trust them and how much he wants to work with them. Said he will come back after that and figure out how much time he wants to spend.

  Sure, the email by Sutskever and Brockman gave some nonviolent communication vibes and maybe it isn’t “the professional thing” to air one’s feelings and perceived mistakes like that, but they seemed genuine in what they wrote and they raised incredibly important concerns that are difficult in nature to bring up. Also, with hindsight especially, it seems like they had valid reasons to be concerned about Altman’s power-seeking tendencies!

  When someone expresses legitimate-given-the-situation concerns about your alignment and your reaction is to basically gaslight them into thinking they did something wrong for finding it hard to trust you, and then you make it seem like you are the poor victim who needs 10 days off of work to figure out whether you can still trust them, that feels messed up! (It’s also a bit hypocritical because the whole “I need 10 days to figure out if I can still trust you for thinking I like being CEO a bit too much,” seems childish too.)

  (Of course, these emails are just snapshots and we might be missing things that happened in between via other channels of communication, including in-person talks.)

  Also, I find it interesting that they (Sutskever and Brockman) criticized Musk just as much as Altman (if I understood their email correctly), so this should make it easier for Altman to react with grace. I guess given Musk’s own annoyed reaction, maybe Altman was calling the others’ email childish to side with Musks’s dismissive reaction to that same email.

  Lastly, this email thread made me wonder what happened between Brockman and Sutskever in the meantime, since it now seems like Brockman no longer holds the same concerns about Altman even though recent events seem to have given a lot of new fire to them.

• Lukas_Gloor 13 Jul 2024 16:05 UTC

  3 points

  0

  on: Poker is a bad game for teach­ing epistemics. Fig­gie is a bet­ter one.

  Some of the points you make don’t apply to online poker. But I imagine that the most interesting rationality lessons from poker come from studying other players and exploiting them, rather than memorizing and developing an intuition for the pure game theory of the game.

  • If you did want to focus on the latter goal, you can play online poker (many players can >12 tables at once) and after every session, run your hand histories through a program (e.g., “GTO Wizard”) that will tell you where you made mistakes compared to optimal strategy, and how much they would cost you against an optimal-playing opponent. Then, for any mistake, you can even input the specific spot into the trainer program and practice it with similar hands 4-tabling against the computer, with immediate feedback every time on how you played the spot.

• Lukas_Gloor 3 Jul 2024 1:09 UTC

  12 points

  3

  on: An AI Race With China Can Be Bet­ter Than Not Racing

  It seems important to establish whether we are in fact going to be in a race and whether one side isn’t already far ahead.

  With racing, there’s a difference between optimizing the chance of winning vs optimizing the extent to which you beat the other party when you do win. If it’s true that China is currently pretty far behind, and if TAI timelines are fairly short so that a lead now is pretty significant, then the best version of “racing” shouldn’t be “get to the finish line as fast as possible.” Instead, it should be “use your lead to your advantage.” So, the lead time should be used to reduce risks.
  
  Not sure this is relevant to your post in particular; I could’ve made this point also in other discussions about racing. Of course, if a lead is small or non-existent, the considerations will be different.

• Lukas_Gloor 27 Jun 2024 0:47 UTC

  5 points

  0

  in reply to: Charlie Steiner’s comment on: Pop­u­la­tion ethics and the value of variety

  I wrote a long post last year saying basically that.

• Lukas_Gloor 19 Jun 2024 11:50 UTC

  9 points

  6

  in reply to: jbkjr’s comment on: Suffer­ing Is Not Pain

  Even if attaining a total and forevermore cessation of suffering is substantially more difficult/​attainable by substantially fewer people in one lifetime, I don’t think it’s unreasonable to think that most people could suffer at least 50 percent less with dedicated mindfulness practice. I’m curious as to what might feed an opposing intuition for you! I’d be quite excited about empirical research that investigates the tractability and scalability of meditation for reducing suffering, in either case.

  My sense is that existing mindfulness studies don’t show the sort of impressive results that we’d expect if this were a great solution.

  Also, I think people who would benefit most from having less day-to-day suffering often struggle with having no “free room” available for meditation practice, and that seems like an issue that’s hard to overcome even if meditation practice would indeed help them a lot.

  It’s already sign of having a decently good life when you’re able to start dedicating time for something like meditation, which I think requires a bit more mental energy than just watching series or scrolling through the internet. A lot of people have leisure time, but it’s a privilege to be mentally well off enough to do purposeful activities during your leisure time. The people who have a lot of this purposeful time probably (usually) aren’t among the ones that suffer most (whereas the people who don’t have it will struggle sticking to regular meditation practice, for good reasons).

  For instance, if someone has a chronic illness with frequent pain and nearly constant fatigue, I can see how it might be good for them to practice meditation for pain management, but higher up on their priority list are probably things like “how do I manage to do daily chores despite low energy levels?” or “how do I not get let go at work?.”

  Similarly, for other things people may struggle with (addictions, financial worries, anxieties of various sorts; other mental health issues), meditation is often something that would probably help, but it doesn’t feel like priority number one for people with problem-ridden, difficult lives. It’s pretty hard to keep up motivation for training something that you’re not fully convinced of it being your top priority, especially if you’re struggling with other things.

  I see meditation as similar to things like “eat healthier, exercise more, go to sleep on time and don’t consume distracting content or too much light in the late evenings, etc.” And these things have great benefits, but they’re also hard, so there are no low-hanging fruit and interventions in this space will have limited effectiveness (or at least limited cost-effectiveness; you could probably get quite far if you gifted people their private nutritionist cook, fitness trainer and motivator, house cleaner and personal assistant, meditation coach, give them enough money for financial independence, etc.).

  And then the people who would have enough “free room” to meditate may be well off enough to not feel like they need it? In some ways, the suffering of a person who is kind of well off in life isn’t that bad and instead of devoting 1h per day for meditation practice to reduce the little suffering that they have, maybe the well-off person would rather take Spanish lessons, or train for a marathon, etc.

  (By the way, would it be alright if I ping you privately to set up a meeting? I’ve been a fan of your writing since becoming familiar with you during my time at CLR and would love a chance to pick your brain about SFE stuff and hear about what you’ve been up to lately!)

  I’ll send you a DM!

• Lukas_Gloor 18 Jun 2024 19:11 UTC

  5 points

  0

  on: Suffer­ing Is Not Pain

  [...] I am certainly interested to know if anyone is aware of sources that make a careful distinction between suffering and pain in arguing that suffering and its reduction is what we (should) care about.

  I did so in my article on Tranquilism, so I broadly share your perspective!
  
  I wouldn’t go as far as what you’re saying in endnote 9, though. I mean, I see some chance that you’re right in the impractical sense of, “If someone gave up literally all they cared about in order to pursue ideal meditation training under ideal circumstances (and during the training they don’t get any physical illness issues or otherwise have issues crop up that prevent successfully completion of the training), then they could learn to control their mental states and avoid nearly all future sources of suffering.” But that’s pretty impractical even if true!

  It’s interesting, though, what you say about CBT. I agree it makes sense to be accurate about these distinctions, and that it could affect specific interventions (though maybe not at the largest scale of prioritization, the way I see the landscape).

• Lukas_Gloor 17 Jun 2024 10:21 UTC

  6 points

  4

  in reply to: Matthew Barnett’s comment on: Matthew Bar­nett’s Shortform

  This would be a valid rebuttal if instruction-tuned LLMs were only pretending to be benevolent as part of a long-term strategy to eventually take over the world, and execute a treacherous turn. Do you think present-day LLMs are doing that? (I don’t)

  Or that they have a sycophancy drive. Or that, next to “wanting to be helpful,” they also have a bunch of other drives that will likely win over the “wanting to be helpful” part once the system becomes better at long-term planning and orienting its shards towards consequentialist goals.

  On that latter model, the “wanting to be helpful” is a mask that the system is trained to play better and better, but it isn’t the only thing the system wants to do, and it might find that once its gets good at trying on various other masks to see how this will improve its long-term planning, it for some reason prefers a different “mask” to become its locked-in personality.

• Lukas_Gloor 1 Jun 2024 14:28 UTC

  11 points

  9

  in reply to: TurnTrout’s comment on: MIRI 2024 Com­mu­ni­ca­tions Strategy

  I thought the first paragraph and the boldened bit of your comment seemed insightful. I don’t see why what you’re saying is wrong – it seems right to me (but I’m not sure).

• Lukas_Gloor 1 Jun 2024 14:20 UTC

  11 points

  2

  in reply to: Sen’s comment on: MIRI 2024 Com­mu­ni­ca­tions Strategy

  I am not convinced MIRI has given enough evidence to support the idea that unregulated AI will kill everyone and their children.

  The way you’re expressing this feels like an unnecessarily strong bar.
  
  I think advocacy for an AI pause already seems pretty sensible to me if we accept the following premises:

  • The current AI research paradigm mostly makes progress in capabilities before progress in understanding.
    (This puts AI progress in a different reference class from most other technological progress, so any arguments with base rates from “technological progress normally doesn’t kill everyone” seem misguided.)

  • AI could very well kill most of humanity, in the sense that it seems defensible to put this at anywhere from 20-80% (we can disagree on the specifics of that range, but that’s where I’d put it looking at the landscape of experts who seem to be informed and doing careful reasoning (so not LeCun)).

  • If we can’t find a way to ensure that TAI is developed by researchers and leaders who act with a degree of responsibility proportional to the risks/​stakes, it seems better to pause.
    

  Edited to add the following:
  There’s also a sense in which whether to pause is quite independent from the default risk level. Even if the default risk were only 5%, if there were a solid and robust argument that pausing for five years will reduce it to 4%, that’s clearly very good! (It would be unfortunate for the people who will die preventable deaths in the next five years, but it still helps overall more people to pause under these assumptions.)

• Lukas_Gloor 1 Jun 2024 13:54 UTC

  19 points

  6

  in reply to: Matthew Barnett’s comment on: MIRI 2024 Com­mu­ni­ca­tions Strategy

  Would most existing people accept a gamble with 20% of chance of death in the next 5 years and 80% of life-extension and radically better technology? I concede that many would, but I think it’s far from universal, and I wouldn’t be too surprised if half of people or more think this isn’t for them.
  
  I personally wouldn’t want to take that gamble (strangely enough I’ve been quite happy lately and my life has been feeling meaningful, so the idea of dying in the next 5 years sucks).

  (Also, I want to flag that I strongly disagree with your optimism.)
  

• Lukas_Gloor 31 May 2024 0:34 UTC

  33 points

  19

  in reply to: MichaelDickens’s comment on: OpenAI: He­len Toner Speaks

  we have found Mr Altman highly forthcoming

  That’s exactly the line that made my heart sink.
  
  I find it a weird thing to choose to say/​emphasize.
  
  The issue under discussion isn’t whether Altman hid things from the new board; it’s whether he hid things to the old board a long while ago.
  
  Of course he’s going to seem forthcoming towards the new board at first. So, the new board having the impression that he was forthcoming towards them? This isn’t information that helps us much in assessing whether to side with Altman vs the old board. That makes me think: why report on it? It would be a more relevant update if Taylor or Summers were willing to stick their necks out a little further and say something stronger and more direct, something more in the direction of (hypothetically), “In all our by-now extensive interactions with Altman, we got the sense that he’s the sort of person you can trust; in fact, he had surprisingly circumspect and credible things to say about what happened, and he seems self-aware about things that he could’ve done better (and those things seem comparatively small or at least very understandable).” If they had added something like that, it would have been more interesting and surprising. (At least for those who are currently skeptical or outright negative towards Altman; but also “surprising” in terms of “nice, the new board is really invested in forming their own views here!”).
  
  By contrast, this combination of basically defending Altman (and implying pretty negative things about Toner and McCauley’s objectivity and their judgment on things that they deem fair to tell the media), but doing so without sticking their necks out, makes me worried that the board is less invested in outcomes and more invested in playing their role. By “not sticking their necks out,” I mean the outsourcing of judgment-forming to the independent investigation and the mentioning of clearly unsurprising and not-very-relevant things like whether Altman has been forthcoming to them, so far. By “less invested in outcomes and more invested in playing their role,” I mean the possibility that the new board maybe doesn’t consider it important to form opinions at the object level (on Altman’s character and his suitability for OpenAI’s mission, and generally them having a burning desire to make the best CEO-related decisions). Instead, the alternative mode they could be in would be having in mind a specific “role” that board members play, which includes things like, e.g., “check whether Altman ever gets caught doing something outrageous,” “check if he passes independent legal reviews,” or “check if Altman’s answers seem reassuring when we occasionally ask him critical questions.” And then, that’s it, job done. If that’s the case, I think that’d be super unfortunate. The more important the org, the more it matters to have a engaged/​invested board that considers itself ultimately responsible for CEO-related outcomes (“will history look back favorably on their choices regarding the CEO”).
  
  To sum up, I’d have much preferred it if their comments had either included them sticking their neck out a little more, or if I had gotten from them more of a sense of still withholding judgment. I think the latter would have been possible even in combination with still reminding the public that Altman (e.g.,) passed that independent investigation or that some of the old board members’ claims against him seem thinly supported, etc. (If that’s their impression, fair enough.) For instance, it’s perfectly possible to say something like, “In our duty as board members, we haven’t noticed anything unusual or worrisome, but we’ll continue to keep our eyes open.” That’s admittedly pretty similar, in substance, to what they actually said. Still, it would read as a lot more reassuring to me because of its different emphasis My alternative phrasing would help convey that (1) they don’t naively believe that Altman – in worlds where he is dodgy – would have likely already given things away easily in interactions towards them, and (2) that they consider themselves responsible for the outcome (and not just following of the common procedures) of whether OpenAI will be led well and in line with its mission.
  (Maybe they do in fact have these views, 1 and 2, but didn’t do a good job here at reassuring me of that.)

• Lukas_Gloor 30 May 2024 0:53 UTC

  23 points

  13

  in reply to: bhauth’s comment on: OpenAI: Fallout

  Followed immediately by:

  I too also have very strong concerns that we are putting a person whose highest stats are political maneuvering and deception, who is very high in power seeking, into this position. By all reports, you cannot trust what this man tells you.

• Lukas_Gloor 18 May 2024 16:01 UTC

  10 points

  7

  in reply to: habryka’s comment on: Stephen Fowler’s Shortform

  For me, the key question in situations when leaders made a decision with really bad consequences is, “How did they engage with criticism and opposing views?”
  
  If they did well on this front, then I don’t think it’s at all mandatory to push for leadership changes (though certainly, the worse someones track record gets, the more that speaks against them).
  
  By contrast, if leaders tried to make the opposition look stupid or if they otherwise used their influence to dampen the reach of opposing views, then being wrong later is unacceptable.

  Basically, I want to allow for a situation where someone was like, “this is a tough call and I can see reasons why others wouldn’t agree with me, but I think we should do this,” and then ends up being wrong, but I don’t want to allow situations where someone is wrong after having expressed something more like, “listen to me, I know better than you, go away.”

  In the first situation, it might still be warranted to push for leadership changes (esp. if there’s actually a better alternative), but I don’t see it as mandatory.

  The author of the original short form says we need to hold leaders accountable for bad decisions because otherwise the incentives are wrong. I agree with that, but I think it’s being too crude to tie incentives to whether a decision looks right or wrong in hindsight. We can do better and evaluate how someone went about making a decision and how they handled opposing views. (Basically, if opposing views aren’t loud enough that you’d have to actively squish them using your influence illegitimately, then the mistake isn’t just yours as the leader; it’s also that the situation wasn’t significantly obvious to others around you.) I expect that everyone who has strong opinions on things and is ambitious and agenty in a leadership position is going to make some costly mistakes. The incentives shouldn’t be such that leaders shy away from consequential interventions.

• Lukas_Gloor 15 May 2024 15:17 UTC

  10 points

  3

  in reply to: Tristan Wegner’s comment on: Ilya Sutskever and Jan Leike re­sign from OpenAI

  I agree with what you say in the first paragraph. If you’re talking about Ilya, which I think you are, I can see what you mean in the second paragraph, but I’d flag that even if he had some sort of plan here, it seems pretty costly and also just bad norms for someone with his credibility to say something that indicates that he thinks OpenAI is on track to do well at handling their great responsibility, assuming he were to not actually believe this. It’s one thing to not say negative things explicitly; it’s a different thing to say something positive that rules out the negative interpretations. I tend to take people at their word if they say things explicitly, even if I can assume that they were facing various pressures. If I were to assume that Ilya is saying positive things that he doesn’t actually believe, that wouldn’t reflect well on him, IMO.
  
  If we consider Jan Leike’s situation, I think what you’re saying applies more easily, because him leaving without comment already reflects poorly on OpenAI’s standing on safety, and maybe he just decided that saying something explicitly doesn’t really add a ton of information (esp. since maybe there are other people who might be in a better position to say things in the future). Also, I’m not sure it affects future employment prospects too much if someone leaves a company, signs a non-disparagement agreement, and goes “no comment” to indicate that there was probably dissatisfaction with some aspects of the company. There are many explanations for this and if I was making hiring decisions at some AI company, even if it’s focused on profits quite a bit, I wouldn’t necessarily interpret this as a negative signal.
  
  That said, signing non-disparagament agreements certainly feels like it has costs and constrains option value, so it seems like a tough choice.

• Lukas_Gloor 15 May 2024 13:00 UTC

  25 points

  18

  in reply to: Tyler Tracy’s comment on: Ilya Sutskever and Jan Leike re­sign from OpenAI

  It seems likely (though not certain) that they signed non-disparagement agreements, so we may not see more damning statements from them even if that’s how they feel. Also, Ilya at least said some positive things in his leaving announcement, so that indicates either that he caved in to pressure (or too high agreeableness towards former co-workers) or that he’s genuinely not particularly worried about the direction of the company and that he left more because of reasons related to his new project.

• Lukas_Gloor 14 May 2024 14:07 UTC

  6 points

  5

  in reply to: Slapstick’s comment on: Monthly Roundup #18: May 2024

  I agree: appealing to libertarianism shouldn’t automatically win someone the argument on whether it’s okay to still have factory farms.

  The fact that Zvi thought he provided enough of a pointer to an argument there feels weird, in my opinion.

  That said, maybe he was mostly focused on wanting to highlight that a large subset of people who are strongly against this ban (and may use libertarian arguments to argue for their position) are only against bans when it suits their agenda. So, maybe the point was in a way more about specific people’s hypocrisy in how they argue than the question of concern for animals.

  Either way, I continue to appreciate all these newsletters and I admit that the opinionated tone often makes them more interesting/​fun to read in cases where it’s not triggering me on issues that I see differently.

• Lukas_Gloor 12 May 2024 11:50 UTC

  4 points

  2

  in reply to: denkenberger’s comment on: We might be miss­ing some key fea­ture of AI take­off; it’ll prob­a­bly seem like “we could’ve seen this com­ing”

  I think one issue is that someone can be aware about a specific worldview’s existence and even consider it a plausible worldview, but still be quite bad at understanding what it would imply/​look like in practice if it were true.

  For me personally, it’s not that I explicitly singled out the scenario that happened and assigned it some very low probability. Instead, I think I mostly just thought about scenarios that all start from different assumptions, and that was that.

  For instance, when reading Paul’s “What failure looks like” (which I had done multiple times), I thought I understood the scenario and even explicitly assigned it significant likelihood, but as it turned out, I didn’t really understand it because I never really thought in detail about “how do we get from a 2021 world (before chat-gpt) to something like the world when things go off the rails in Paul’s description?” If I had asked myself that question, I’d probably have realized that his worldview implies that there probably isn’t a clear-cut moment of “we built the first AGI!” where AI boxing has relevance.

We might be miss­ing some key fea­ture of AI take­off; it’ll prob­a­bly seem like “we could’ve seen this com­ing”

• Lukas_Gloor 7 May 2024 0:52 UTC

  4 points

  0

  on: Rapid ca­pa­bil­ity gain around su­per­ge­nius level seems prob­a­ble even with­out in­tel­li­gence need­ing to im­prove intelligence

  I lean towards agreeing with the takeaway; I made a similar argument here and would still bet on the slope being very steep inside the human intelligence level.