jeffreycaruso

[Question] Look­ing to in­ter­view AI Safety re­searchers for a book

[Question] Dan Hendrycks and EA

• jeffreycarusoJun 5, 2024, 1:55 PM

  3 points

  −2

  on: jeffr­ey­caruso’s Shortform

  Fired from OpenAI’s Superalignment team, Aschenbrenner now runs a fund dedicated to funding AGI-focused startups, according to The Information.

  “Former OpenAI super-alignment researcher Leopold Aschenbrenner, who was fired from the company for allegedly leaking information, has started an investment firm to back startups with capital from former Github CEO Nat Friedman, investor Daniel Gross, Stripe CEO Patrick Collision and Stripe president John Collision, according to his personal website.

  In a recent podcast interview, Aschenbrenner spoke about the new firm as a cross between a hedge fund and a think tank, focused largely on AGI, or artificial general intelligence. “There’s a lot of money to be made. If AGI were priced in tomorrow, you could maybe make 100x. Probably you can make even way more than that,” he said. “Capital matters.”

  “We’re going to be betting on AGI and superintelligence before the decade is out, taking that seriously, making the bets you would make if you took that seriously. If that’s wrong, the firm is not going to do that well,” he said.”

  What happened to his concerns over safety, I wonder?

• jeffreycarusoJun 1, 2024, 1:09 PM

  3 points

  1

  in reply to: gilch’s comment on: robo’s Shortform

  Your example of the janitor interrupting the scientist is a good demonstration of my point. I’ve organized over a hundred cybersecurity events featuring over a thousand speakers and I’ve never had a single janitor interrupt a talk. On the other hand, I’ve had numerous “experts” attempt to pass off fiction as fact, draw assumptions from faulty data, and generally behave far worse than any janitor might due to their inflated egos.

  Based on my conversations with computer science and philosophy professors who aren’t EA-affiliated, and several who are, their posts are frequently down-voted simply because they represent opposite viewpoints.

  Do the moderators of this forum do regular assessments to see how they can make improvements in the online culture so that there’s more diversity in perspective?

• jeffreycarusoMay 23, 2024, 2:50 AM

  12 points

  9

  in reply to: habryka’s comment on: robo’s Shortform

  I think you’re too close to see objectively. I haven’t observed any room for policy discussions in this forum that stray from what is acceptable to the mods and active participants. If a discussion doesn’t allow for opposing viewpoints, it’s of little value. In my experience, and from what I’ve heard from others who’ve tried posting here and quit, you have not succeeded in making this a forum where people with opposing viewpoints feel welcome.

• jeffreycarusoMay 8, 2024, 5:43 PM

  1 point

  −10

  in reply to: RobertM’s comment on: RobertM’s Shortform

  Have you read this? https://​​www.politico.eu/​​article/​​rishi-sunak-ai-testing-tech-ai-safety-institute/​​

  ““You can’t have these AI companies jumping through hoops in each and every single different jurisdiction, and from our point of view of course our principal relationship is with the U.S. AI Safety Institute,” Meta’s president of global affairs Nick Clegg — a former British deputy prime minister — told POLITICO on the sidelines of an event in London this month.”

  “OpenAI and Meta are set to roll out their next batch of AI models imminently. Yet neither has granted access to the U.K.’s AI Safety Institute to do pre-release testing, according to four people close to the matter.”

  “Leading AI firm Anthropic, which rolled out its latest batch of models in March, has yet to allow the U.K. institute to test its models pre-release, though co-founder Jack Clark told POLITICO it is working with the body on how pre-deployment testing by governments might work.

  “Pre-deployment testing is a nice idea but very difficult to implement,” said Clark.”

• jeffreycarusoApr 28, 2024, 1:18 AM

  3 points

  0

  in reply to: Gunnar_Zarncke’s comment on: Ex­plor­ing the Eso­teric Path­ways to AI Sen­tience (Part One)

  Yes, I like it! Thanks for sharing that analysis, Gunnar.

• jeffreycarusoApr 27, 2024, 2:59 PM

  1 point

  0

  in reply to: Gunnar_Zarncke’s comment on: Ex­plor­ing the Eso­teric Path­ways to AI Sen­tience (Part One)

  Good list. I think I’d use a triangle to organize them. Have consciousness at the base, then sentience, then drawing from your list, phenomenal consciousness, followed by Intentionality?

• jeffreycarusoApr 27, 2024, 1:50 PM

  1 point

  0

  in reply to: Gunnar_Zarncke’s comment on: Ex­plor­ing the Eso­teric Path­ways to AI Sen­tience (Part One)

  Thank you for asking.

  To generalize across disciplines, a critical aspect of human-level artificial intelligence, requires the ability to observe and compare. This is a feature of sentience. All sentient beings are conscious of their existence. Non-sentient conscious beings exist, of course, but none who could pass a Turing test or a Coffee-making test. That requires both sentience and consciousness.

Ex­plor­ing the Eso­teric Path­ways to AI Sen­tience (Part One)

• jeffreycarusoApr 8, 2024, 5:45 PM

  3 points

  0

  on: A Shut­down Prob­lem Proposal

  What happens if you shut down power to the AWS or Azure console powering the Foundation model? Wouldn’t this be the easiest way to test various hypotheses associated with the Shutdown Problem in order to either verify it or reject it as a problem not worth sinking further resources into?

• jeffreycarusoMar 19, 2024, 3:11 AM

  1 point

  0

  in reply to: Dweomite’s comment on: State­ment on AI Ex­tinc­tion—Signed by AGI Labs, Top Aca­demics, and Many Other Notable Figures

  That’s a good example of my point. Instead of a petition, a more impactful document would be a survey of risks and their probability of occurring in the opinion of these notable public figures.

  In addition, there should be a disclaimer regarding who has accepted money from Open Philanthropy or any other EA-affiliated non-profit for research.

• jeffreycarusoMar 19, 2024, 3:04 AM

  1 point

  0

  in reply to: RHollerith’s comment on: State­ment on AI Ex­tinc­tion—Signed by AGI Labs, Top Aca­demics, and Many Other Notable Figures

  Which makes it an existential risk.

  “An existential risk is any risk that has the potential to eliminate all of humanity or, at the very least, kill large swaths of the global population.”—FLI

• jeffreycarusoMar 19, 2024, 2:29 AM

  −1 points

  0

  on: State­ment on AI Ex­tinc­tion—Signed by AGI Labs, Top Aca­demics, and Many Other Notable Figures

  What aspect of AI risk is deemed existential by these signatories? I doubt that they all agree on that point. Your publication “An Overview of Catastrophic AI Risks” lists quite a few but doesn’t differentiate between theoretical and actual.

  Perhaps if you were to create a spreadsheet with a list of each of the risks mentioned in your paper but with the further identification of each as actual or theoretical, and ask each of those 300 luminaries to rate them in terms of probability, then you’d have something a lot more useful.

• jeffreycarusoMar 14, 2024, 4:11 AM

  3 points

  0

  in reply to: Zack_M_Davis’s comment on: jeffr­ey­caruso’s Shortform

  I looked at the paper you recommended Zack. The specific section having to do with “how” AGI is developed (para 1.2) skirts around the problem.

  “We assume that AGI is developed by pretraining a single large foundation model using selfsupervised learning on (possibly multi-modal) data [Bommasani et al., 2021], and then fine-tuning it using model-free reinforcement learning (RL) with a reward function learned from human feedback [Christiano et al., 2017] on a wide range of computer-based tasks.4 This setup combines elements of the techniques used to train cutting-edge systems such as GPT-4 [OpenAI, 2023a], Sparrow [Glaese et al., 2022], and ACT-1 [Adept, 2022]; we assume, however, that 2 the resulting policy goes far beyond their current capabilities, due to improvements in architectures, scale, and training tasks. We expect a similar analysis to apply if AGI training involves related techniques such as model-based RL and planning [Sutton and Barto, 2018] (with learned reward functions), goal-conditioned sequence modeling [Chen et al., 2021, Li et al., 2022, Schmidhuber, 2020], or RL on rewards learned via inverse RL [Ng and Russell, 2000]—however, these are beyond our current scope.”

  Altman has recently said in a speech that continuing to do what has led them to GPT4 is probably not going to get to AGI. “”Let’s use the word superintelligence now, as superintelligence can’t discover novel physics, I don’t think it’s a superintelligence. Training on the data of what you know, teaching to clone the behavior of humans and human text, I don’t think that’s going to get there. So there’s this question that has been debated in the field for a long time: what do we have to do in addition to a language model to make a system that can go discover new physics?”

  https://​​the-decoder.com/​​sam-altman-on-agi-scaling-large-language-models-is-not-enough/​​

  I think it’s pretty clear that no one has a clear path to AGI, nor do we know what a superintelligence will do, yet the Longtermist ecosystem is thriving. I find that curious, to say the least.

• jeffreycarusoMar 13, 2024, 4:46 PM

  3 points

  0

  in reply to: Chris_Leong’s comment on: jeffr­ey­caruso’s Shortform

  My apologies for not being clear in my Quick Take, Chris. As Zach pointed out in his reply, I posed two issues.

  The first being an obvious parallel for me between EA and Judeo-Christian religions. You may or may not agree with me, which is fine. I’m not looking to convince anyone of my point-of-view. I was merely interested in seeing if others here had a similar POV.

  The second issue I raised was what I saw as a failure in the reasoning chain where you go from Deep Learning to Consciousness to an AI Armageddon. Why was that leap in faith so compelling to people?

  I don’t see either of those questions as not being in the interest of the “public good”, but perhaps you just said that because my first attempt wasn’t clear. Hopefully, I’ve remedied that with this answer.

• jeffreycarusoMar 13, 2024, 4:34 PM

  1 point

  −1

  in reply to: Zack_M_Davis’s comment on: jeffr­ey­caruso’s Shortform

  Thank you for the link to that paper, Zack. That’s not one that I’ve read yet.

  And you’re correct that I raised two separate issues. I’m interested in hearing any responses that members of this community would like to give to either issue.

• jeffreycarusoMar 13, 2024, 2:53 PM

  −10 points

  −27

  on: jeffr­ey­caruso’s Shortform

  It seems to me that Effective Altruism uses a theoretical negative outcome (an extinction-level event) as motivation for action in a very similar way to how Judeo-Christian religions use another theoretical negative outcome (your unsaved soul going to Hell for eternal torment) as motivation for action.

  Both have high priests who establish dogma, and legions of believers who evangelize and grow the base.

  Both spend vast amounts of money to persuade others to adopt their belief system.

  There’s nothing new there regarding how religions work, but for a philosophical belief that’s supposed to be grounded in rational decision-making, there’s a giant looming gap in the reasoning chain when it comes to AI posing an existential risk to humanity.

  Unless I’m missing something.

  Is there any proof that I haven’t read yet which demonstrates that AGI or Superintelligence will have the capability to go rogue and bring about Armagadden?

jeffr­ey­caruso’s Shortform

• jeffreycarusoMar 10, 2024, 1:23 AM

  1 point

  0

  on: Fre­quent ar­gu­ments about alignment

  Are there other forums for AI Alignment or AI Safety and Security besides this one where your article could be published for feedback from perspectives that haven’t been shaped by Rationalist thinking or EA?