I work on applied mathematics and AI at the Johns Hopkins University Applied Physics Laboratory (APL). I am also currently pursuing a PhD in Information Systems at the University of Maryland, Baltimore County (UMBC). My PhD research focuses on decision and risk analysis under extreme uncertainty, with a particular focus on potential existential risks from very advanced AI.
Aryeh Englander
Karma: 1,512
When you say that you’d give different probability estimates on different days, do you think you can represent that as you sampling on different days from a probability distribution over your “true” latent credence? If yes, do you think it would be useful to try to estimate what that distribution looks like, and then report the mean or perhaps the 90% CI or something like that? So for example, if your estimate typically ranges between 33% and 66% depending on the day with a mean of say 50%, then instead of reporting what you think today (the equivalent of taking a single random sample from the distribution), maybe you could report 50% because that’s your mean and/or report that your estimate typically ranges from 33% to 66%.
From a Facebook discussion with Scott Aaronson yesterday:
Yann: I think neither Yoshua nor Geoff believe that AI is going kill us all with any significant probability.
Scott: Well, Yoshua signed the pause letter, and wrote an accompanying statement about what he sees as the risk to civilization (I agree that there are many civilizational risks short of extinction). In his words: “No one, not even the leading AI experts, including those who developed these giant AI models, can be absolutely certain that such powerful tools now or in the future cannot be used in ways that would be catastrophic to society.”
Geoff said in a widely-shared recent video that it’s “not inconceivable” that AI will wipe out humanity, and didn’t offer any reassurances about it being vanishingly unlikely.
https://twitter.com/JMannhart/status/1641764742137016320
Yann: Scott Aaronson he is worried about catastrophic disruptions of the political, economic, and environmental systems. I don’t want to speak for him, but I doubt he worries about a Yuddite-style uncontrollable “hard takeoff”
The conversation took place in the comments section to something I posted on Facebook: https://m.facebook.com/story.php?story_fbid=pfbid0qE1PYd3ijhUXVFc9omdjnfEKBX4VNqj528eDULzoYSj34keUbUk624UwbeM4nMyNl&id=100010608396052&mibextid=Nif5oz
[Question] Steelman / Ideological Turing Test of Yann LeCun’s AI X-Risk argument?
[Cross-post] Is the Fermi Paradox due to the Flaw of Averages?
[Question] What would you expect a massive multimodal online federated learner to be capable of?
Sometimes it’s better in the long run to take a good chunk of time off to do things for fun and write or work less. Sometimes less is more. But this is very much a YMMV thing.
This is actually another related area of my research: To the extent that we cannot get people to sit down and agree on double cruxes, can we still assign some reasonable likelihoods and/or uncertainty estimates for those likelihoods? After all, we do ultimately need to make decisions here! Or if it turns out that we literally cannot use any numbers here, how do we best make decisions anyway?
Epistemic modesty and how I think about AI risk
[Link] OpenAI: Learning to Play Minecraft with Video PreTraining (VPT)
I have now posted a “Half-baked AI safety ideas thread” (LW version, EA Forum version) - let me know if that’s more or less what you had in mind.
Half-baked AI Safety ideas thread
Just putting in my vote for doing both broader and deeper explorations of these topics!
My impression—which I kind of hope is wrong—has been that it is much easier to get an EA grant the more you are an “EA insider” or have EA insider connections. The only EA connection that my professor has is me. On the other hand, I understand the reluctance to some degree in the case of AI safety because funders are concerned that researchers will take the money and go do capabilities research instead.
Honestly I suspect this is going to be the single largest benefit from paying Scott to work on the problem. Similarly, when I suggested in an earlier comment that we should pay other academics in a similar manner, in my mind the largest benefit of doing so is because that will help normalize this kind of research in the wider academic community. The more respected researchers there are working on the problem, the more other researchers start thinking about it as well, resulting (hopefully) in a snowball effect. Also, researchers often bring along their grad students!
Hopefully. I have a feeling it won’t be so easy, but we’ll see.
Yes! I actually just discussed this with one of my advisors (an expert on machine learning), and he told me that if he could get funding to do it he would definitely be interested in dedicating a good chunk of his time to researching AGI safety. (For any funders who might read this and might be interested in providing that funding, please reach out to me by email Aryeh.Englander@jhuapl.edu. I’m going to try to reach out to some potential funders next week.)
I think that there are a lot of researchers who are sympathetic to AI risk concerns, but they either lack the funding to work on it or they don’t know how they might apply their area of expertise to do so. The former can definitely be fixed if there’s an interest from funding organizations. The latter can be fixed in many cases by reaching out and talking to the researcher.
It also depends on your target audience. (Which is basically what you said, just in slightly different words.) If you want to get Serious Researchers to listen to you and they aren’t already within the sub-sub-culture that is the rationality community and its immediate neighbors, then in many (most?) cases ranting and freaking out is probably going to be actively counterproductive to your cause. Same if you’re trying to build a reputation as a Serious Researcher, with a chance that decision makers who listen to Serious Researchers might listen to you. On the other hand, if your target audience is people who already trust you or who are already in your immediate sub-sub-tribe, and you don’t mind risking being labeled a crackpot by the wider world, then I can see why visibly freaking out could be helpful.
[Also, it goes without saying that not everybody agrees with Eliezer’s probability-of-doom estimates. Depending on your relative probabilities it might make perfect sense to work in a random startup, have a 401k, not visibly freak out, etc.]
You should make this a top level post so it gets visibility. I think it’s important for people to know the caveats attached to your results and the limits on its implications in real-world dynamics.