amelia
″...this kind of transformer doesn’t seem to provide any evidence on whether we can create full-fidelity simulations in the future.”
My point wasn’t that WE would create full-fidelity simulations in the future. There’s a decent likelihood that WE will all be made extinct by AI. My point was that future AI might create full-fidelity simulations, long after we are gone.
“I currently think of the simulation hypothesis as similar to MWI in quantum mechanics—it’s a model that cannot be proven or disproven...”
Ironically, I believe many observable phenomena in quantum mechanics provide strong support (or what you might call “proof”) for the simulation hypothesis—or at least for the existence of a deeper/”information level” “under” the quantum level of our universe. Here’s a short, informal article I wrote about how one such phenomenon (wave function collapse) supports the idea of an information level (if not the entire simulation hypothesis).
[EDIT: The title of the article reflects how MWI needs a supplemental interpretation involving a “deeper/information” level. From this, you can infer my point.]
Also, the fact that something can’t currently be proven or disproven does not mean it isn’t true (and that it won’t be “proven” in the future). Such has been the case for many theories at first, including general relativity, evolution through natural selection, etc.
Thanks for sharing this! It’s so interesting how multiple people start having similar thoughts when the environment is right. It seems the simulation hypothesis and AI Risk are inextricably linked, even if for no other purpose than conducting thought experiments that help us understand both better.
To the people who upvoted this post,
Thank you very much for the support. As you maybe saw below, the restriction on my account has been lifted!
As I also mentioned below, I might not take advantage of the restored liberty in the short term. I’ve already begun consolidating all my writing on my personal website (AmeliaJones.org), with links to Medium blog posts for the writing. (The writing that was on LW would mostly be under writing...AI, or writing....physics. There are also short stories and other right-brain type stuff, but I don’t think LW folks would be too interested in that i.e. just ignore it.)
However, I might return to LW in the longer term. For now, please don’t be offended if I don’t respond to comments on this post. I don’t think I will be checking in too often. This will allow me to focus more on my new platform.
Warmest wishes to all of you, and thanks again for the support when I really needed it,
Amelia
Thanks for being curious! I’ve begun using my personal website (AmeliaJones.org) as a place for all my work. From there, I will have links to Medium blog posts. (Posts that were previously on LW would mostly be under the writing....philosophy, or writing....physics categories on the website.) I appreciate your interest!
Raemon, thank you very much for lifting the restriction on my account! I’m sure it’s extremely challenging to maintain high LW standards, while at the same time trying to promote open dialog with differing perspectives. I don’t envy your job, but I really appreciate the work you do.
In the short term, I might not take full advantage of my restored liberty. I’ve started using my personal website (www.AmeliaJones.org) for both AI art projects and all my writing (not just LW writing). The writing will have links to Medium blog posts, so people can comment as much as they choose. It’s actually turning out to be easier for me to do things this way. However, depending on whether I get any visibility or feedback via this method, I might return to LW for niche writing projects in the longer term.Thanks again for lifting the restriction, and for the important work you do.
Best wishes,
Amelia
Meta comment on LW, as it relates to this post:
So when I checked this post in the morning, it had received a negative ten (up to that point in time) in “karma.” When I hovered over the negative ten, I saw the message “3 votes,” so apparently three people strongly disapproved of (disagreed with?) this post. Five days ago, I received a negative eight in karma from two people. I asked for guidance and input, but none has been forthcoming (at least in the five days since then).
I don’t mind the “negative karma” votes, in and of themselves, but it seems like such votes would be more useful if they were accompanied by explanations. Otherwise if it’s a matter of, for example, faulty logic or faulty givens, there’s no way to know this, let alone improve. I also don’t know if it’s the same people strongly disapproving from one post to the next, or different people, each with a unique reason for disapproval.
As far as my point in this particular post (that I hope Eliezer will sit for another interview w/ Lex to cover more topics within the scope of AI safety, or at least address them in another venue like LW), I listened to the last few minutes of the interview again, and Lex said “Thank you for the fight you’re fighting. Thank you for being fearless and bold and for everything you do. I hope we get a chance to talk again and I hope you never give up.”Then Eliezer said “You’re welcome. I do worry I didn’t address a whole lot of fundamental questions I expect people have, but you know maybe we got a little bit further and made a tiny little bit of progress and I’d say like be satisfied with that, but actually no, I think one should only be satisfied with solving the entire problem.”
Then Lex said “To be continued....”
In light of what Eliezer said, I wonder whether he would also “strongly disapprove” of the hope for more discussion that I expressed in my post. I also wonder whether this is the way Eliezer imagined karma would be used on LW.
--
A side note is that millions of people listen to Lex’s podcasts, so if we really want to get the most brilliant young people working on interpretability, and AI alignment problems in general, then shouldn’t we want Lex to spend more time discussing these problems? Do the negative karma votes on this post mean people don’t want more public discussion of AI safety?
Thanks for the follow up. Yeah, that’s a great point. I was imagining the bottomless respect and love I have felt for my own parents and teachers, regardless of the heartbreaking way their intellect erodes and crumbles with age. Yet that does not translate to human society as a whole, let alone AI.
I agree that AI would be more equivalent to humans if it had designated functional areas. All analogies break down eventually, but maybe this analogy breaks down sooner rather than later. Nevertheless, I’d still love to hear Eliezer and Lex discuss it, along with whether it’s possible to add functional areas adjacent to LLMs, to improve the chances that LLMs will develop some sort of “creator respect,” or if such human-similarity functionality would have to be more integrated than adjacent.
I wish that all AI advancement work would be halted in order to work on such safety and adjacent functionality issues, but at the very least, it seems it couldn’t hurt to talk about—and thereby encourage more brilliant young people to work on—such issues.
That’s a really useful distinction. Thank you for taking the time to make it! I also think that I made it sound like “simulator” worlds allow for objective morality. In actuality, I think a supra-universal reality might allow for simulator worlds, and a supra-universal reality might allow for objective morality (by some definitions of it), but the simulator worlds and the objective morality aren’t directly related in their own right.
So within an hour, this received a downvote of negative 8, without explanation. That’s alright, but I don’t really understand how it wouldn’t be beneficial to consider AGI risk from every possible perspective and framework—including new perspectives and frameworks—given the fact that this is an existential concern.
I’m not sure if people consider the simulation hypothesis to be “fringe,” but David Chalmers isn’t exactly a “fringe” philosopher, and he has written extensively about the hypothesis. I limited my citation to one page number in Chalmers’ book, since more precise citations were requested in a response to a previous LW post, but you could argue that most of the book is related to the simulation hypothesis. (For example, chapter two is titled “What is the simulation hypothesis?” and chapter five is titled “Is it likely that we’re in a simulation?”)
Anyway, as usual, the downvotes are alright, but any explanation would be helpful (presuming the explanation doesn’t assert I suggested something that I didn’t suggest). Thanks for reading this!
Btw, I’m totally cool w/ the downvotes on this one. I probably would have downvoted it too, because it’s not at all developed or detailed.
The only time a downvote or “unfavorable impression” disturbs me is when it’s accompanied by an explanation that makes me think the downvoter is under the impression that I said something I didn’t say, or that I believe something I don’t believe. Granted, even then, the false impression can also be my fault for not having explained myself clearly in the first place.
In this particular case, I know the post was read carefully, and contemplated, because shminux asked for clarification and elaboration. That made me really happy, regardless of downvotes and upvotes. Thanks for that!
Great idea! I’ll work on that. Thx!
I was also interested in seeing the kind of reaction a philosophical physics post would get, as it pertains to a previous post I made. I really appreciate the opportunity to discuss even “far out” ideas on LW.
Thanks for the feedback! Yeah, excellent point about acceleration. In the current version, we actually don’t ever address the twin “paradox,” let alone how acceleration resolves it.
In a draft version of the book, we had addressed the twin paradox, but we got feedback that the book was way too long. There are other topics we had to cut out as well, which makes me a little disappointed in the final product. The original version was over 600 pages, and that was admittedly too long, but I feel like we went way too far in the other direction. Granted, it will be a first introduction to special and general relativity for most readers, but I know the twin paradox is going to keep them up at night, and I really wish we had addressed it.
Anyway, thanks for looking at the book, and for offering the feedback!
One of my books is “Einstein Explained: Special & General Relativity for the Masses” (physics pun intended). Yes, it’s pop-sci (“for the masses”), but I believe we convey accurate information. I really feel everyone should understand something about the nature of time, gravity and light; of course this requires passing on at least some introductory knowledge of special and general relativity. Here’s a link to the Amazon copy:
www.amazon.com/dp/B0B8ZGQ8RB
We tried to make it unique with Nietzsche-quoting dog illustrations that were supposed to be sort of like New Yorker (TM) cartoons. I don’t think the book shows any expertise, but your advice to me was to go out and “learn foundations of physics,” and I think the book demonstrates that I have at least some familiarity w/ foundations of physics.
Also, I don’t know that I have a “quest.” However, I think my goals are clearer in the post that linked back to this one. It’s at
https://www.lesswrong.com/posts/NaBfa3cYr8mk55PvT/philosophical-physics-discussion-follow-up
I could have just left it at that, but the dangling loose end of this thread was bothering me, which is why I addressed it here.
Thanks for asking! I don’t have a refined write-up. The value of LW, for me at least, is to propose, discuss, and refine ideas still in their infancy or other prepublication stages. Once I have a truly refined write-up of an idea, I think it would be more in the stage of submitting it to an academic journal. However, at that point, related ideas (and interpretation of the journal article) would be fitting on LW, and the whole cycle could start again. At least, that’s how it is for me. I’m sure other people find different value in LW.
With that said, for the definition of “encapsulating world,” we would have to start with what I meant by “world.” For that, it’s necessary to look to the literature on the many worlds interpretation. As I’m sure you already know, but just to be thorough, MWI was first proposed in a Princeton thesis paper by Hugh Everett. It’s available several places, but here’s one: https://archive.org/details/TheTheoryOfTheUniversalWaveFunction/mode/2up
(Hugh Everett, The Theory of the Universal Wave Function, Thesis, Princeton University, (1956, 1973), pp. 1–140.)
Of course there’s also the book by David Wallace, which thoroughly explains the idea of a “world” in this context: (Wallace, David (2012). The Emergent Multiverse: Quantum Theory According to the Everett Interpretation. Oxford University Press. ISBN 978-0-19-954696-1).
Also, I just came across this article from 2009. The permanent link is at https://doi.org/10.1016/j.shpsb.2008.10.002.
(Stefano Osnaghi, Fábio Freitas, Olival Freire, The origin of the Everettian heresy, Studies in History and Philosophy of Science Part B: Studies in History and Philosophy of Modern Physics, Volume 40, Issue 2, 2009, Pages 97-123, ISSN 1355-2198,https://doi.org/10.1016/j.shpsb.2008.10.002).
A tangential point about this article is that it reflects a point that I (poorly) tried to make in an earlier post, which is that there is a subset within the physics community that seems almost theocratic in the way its members regard physicists with differing (more philosophical) viewpoints. My post made a very unfavorable impression upon some LW folks, as they promptly let me know in the comments, and afterward I conceded that “theocratic” was too strong a word. I also recognized that any misunderstandings were my own fault for obviously doing a poor job explaining myself. Oddly enough, the article I just cited is full of theological-type terminology to describe the same phenomenon I tried to describe. That doesn’t justify my original post. However, it does seem to add to the wry, winking irony of the universe.As far as the “encapsulating” part of an encapsulating world interpretation, you could always check out David Chalmers’ descriptions of how quantum mechanics might be viewed as part of simulated universe. (Chalmers, D. J., & Peacock, T. (2022). Reality+: Virtual worlds and the problems of philosophy. W. W. Norton & Company.) I would give you page numbers, as the book relates to quantum mechanics, but there are many references throughout the book. You could always look in the index to read all of them.) In this case, the “encapsulating” world would be the simulator/programmer/external world.) However, in my own “bare bones” view, I’m not including any of the other assumptions that would go into the simulation hypothesis (such as the idea that the encapsulating world consists of programmers, or even the idea that there are any intentions involved).
I apologize for not giving a single sentence definition of “encapsulating world” when it comes to an encapsulating world interpretation of wave function collapse, but it’s the type of concept that requires the mathematics and diagrams of the many worlds interpretation, along with a view of quantum mechanics similar to what Chalmers describes, and then refinements from there.
As I mentioned in the original post, although the idea independently occurred to me that wave function collapse could be explained by an encapsulating world interpretation, I’m sure it occurred to millions of other people before it occurred to me. Although the idea is not unique to me, I posted it because I still think it would be worth discussing and refining on LW.
Hopefully, the discussion can be at least partially collaborative. I’m starting to realize that LW users are mostly male, and I wonder if this makes discussions slightly more competitive than collaborative (i.e. - some people seem to want to “win debates” a bit more than collaborate). I’m not saying this trend is good or bad, but I’d really appreciate some constructive, collaborative insights, in addition to critiques. (Granted, I recognize that any negative criticism is probably my own fault for poor logic or poorly-worded posts that lead to misunderstandings. Moreover, negative criticism can definitely provide room for growth. Nevertheless, a few collaborative insights that hadn’t previously occurred to me would be a much-appreciated supplement.) If you’ve read this far, thank you!
One (hopefully) final note: With respect to quantum mechanics, I believe a sort of bare-bones* “nested world interpretation” could explain wave function collapse just as well as the “many worlds interpretation” explains it. (A nested world interpretation would involve quantum behavior coming from the outer world, w/ classical physics behavior originating in, or innate to, this world.)
This belief probably does indeed make me a “crank.” The word “crank” was used by another LW user in a reply to my linked post. Based on the context, I think it was supposed to be a mildly pejorative way of referring to someone with unconventional & unprovable ideas.
I think that the “crank allegation” might have actually been correct, at least with respect to my thoughts on interpreting wave function collapse. Sorry for not recognizing that point earlier, and thanks for the discussion. It helped me to think about how my views fit in a larger social context.
As mentioned before, I’ve turned off notifications on this thread, so I might not respond to follow-ups. My silence doesn’t imply an endorsement or rebuttal of anything. It just means I have some time constraints.
--
*By “bare-bones,” I mean nothing that would arise to a simulation theory or any other embellished subset of nested world theories. It’s true that I love discussing the idea of “programmed worlds,” because such discussions engender an almost playful type of supra-universal thinking. However, while I wouldn’t bet money on a programmed world interpretation, I would go so far as to bet on a “no-strings-attached” nested world interpretation.
One of my books that has been referred to multiple times by others in this discussion is
“Einstein Explained: Special and General Relativity for the Masses” (physics pun intended).
I published it under my maiden name, which is Amy Louise Johnson. I created it in collaboration with my then-teenage sons.
I didn’t mention the book at all in my original post. Then I didn’t mention its title in the replies to the comments on the first post, the second post, or the initial replies to the comments on the second post. This is because I didn’t want to use LessWrong for marketing the book.However, I decided to mention it now, along with its title, because of the suggestions that I’m not familiar with foundational physics and that I might be a “crank” author (which I think is meant to imply that I have radical or unproven ideas). If I finally just give people the title of the book, they can at least see previews of the pages on Amazon, and decide for themselves on whether I am a “crank” or have not yet learned foundational physics. If I do have radical or crazy ideas, I guess I would probably be the last to know anyway.
I don’t blame anyone for assuming I’m a “crank” author, or that I’m not familiar with foundational physics, because I initially didn’t give any evidence to the contrary. Furthermore, I think I did a really poor job explaining my viewpoints in the initial post.
Caveats about the book:
-When the book was re-titled, it lost its reviews on Amazon, so I assume it doesn’t have any reviews at this moment.
-The first draft of the book was originally all text w/ only a few diagrams. However based on advice that we should make the book more accessible and one-of-a-kind, we limited the text to the absolute necessities, and added the philosophical, Nietzsche-quoting cartoon dogs. I know it’s a little weird.
-I’m proud of the book being “for the masses” (“popsci” I guess). Frankly, I think it is actually more challenging to explain advanced concepts in a way anyone can understand, than it is to explain advanced concepts to people who already have a specialized vocabulary and expertise. It is even more challenging to explain the concepts simply, while also not sacrificing accuracy. I think we did a decent job in meeting the challenge, but people can judge for themselves, if from nothing other than the preview pages on Amazon. If we did make any mistakes, feedback would be greatly appreciated.
-The book was converted from color to grayscale, and reformatted, and I think the result looks pretty bad.
-We received advice to use professional artwork for the cover, but I insisted on using my own drawing of a “cool Einstein.” That’s why the cover looks so unprofessional. The cover on the teens and tweens edition is also my own art.… genius....---
This is my last comment I’ll make on this thread, and I’ve turned off notifications on the thread.
If I don’t respond to future comments, it’s not because I necessarily agree or disagree with them. It’s simply because I can’t afford to spend more time on this. I’ve already spent way too long, and I have other things I should have been doing.
However, despite ending my involvement with this post and the linked one, I did find this discussion to be very productive (for me at least), and I learned a great deal from the feedback (such as how to explain myself better, the necessity of examples, and the importance of reading feedback in a realistic tone of voice, etc.) Thank you to everyone who gave feedback, good or bad, because it was all useful. Best wishes.....
Thank you for your feedback. Here’s my feedback on your feedback. My words are in bold.
Your quote: Physicist here. Your post did not make a positive impression on me, because it seems to be generally wrong.
My response: I’m really sorry my post did not make a positive impression on you. As to whether it was “generally wrong,” I’ll address that based on your points that follow. In any places where I feel you misunderstood me, that is my fault, because I obviously did a terrible job explaining myself if multiple people misunderstood (which they did). I’ll try to clarify a little bit in this reply.
--
Your quote: Your belief that there are ‘philosophical’ and ‘shut-up-and-calculate’ physicists generally agrees with my anecdotal experience.
My response: Thank you. I guess this belief (the premise on which my initial post was supposed to be based) “generally agrees” with your anecdotal experience, so we’re OK so far.
--
Your quote: However, that’s the thing: there are many physicists who are happy to think about philosophy. I think I fall into that camp. Really strange to think that there are philosophical physicists, and yet think that physicists don’t engage in philosophical discussion.
My response: My point was not supposed to be that “physicists don’t engage in philosophical discussion.” It was that the non-philosophical, self-described “shut up and calculate” physicists have a bias against philosophical discussions. Philosophical physicists definitely engage in philosophical discussions. That was supposed to be one of the two main points in my original post (that the non-philosophical physicists are biased against philosophical discussions among philosophical physicists). I think we’re actually in agreement on this, but I clearly did a poor job explaining myself, since you thought we were in disagreement. My apologies...
--
Your quote of my quote: “From a strictly materialist perspective, doesn’t it seem rather “universe-centric” to think the reality that gave rise to the Big Bang and our universe(1) only gave rise to our particular universe?”
Your response: Right, many physicists who actually have thoughts about this don’t.
My response: OK, so I think we’re alright on this part.
--
Your quote of my quote: And doesn’t it seem even more universe-centric to think of the supra-universal (“deeper”) reality as less significant than its product(s)?
Your response: I don’t think many physicists have strong opinions on what’s ‘more significant’.
My response: I think we’re OK on this. I started the post out with these questions in an attempt to set the stage for philosophical thinking about physics, before mentioning the bias. However, I think this probably wasn’t effective, as people wondered why I was asking these sort of obvious questions.
--
Your quote of my original quote: Granted, we can’t know much about the deeper reality, but it seems there could be some hints about its nature in fields like quantum mechanics, relativity physics, cosmology and philosophy.
Your response: Yes.
My response: OK, again, “Yes,” makes me think we’re OK with this.
--
Your quote of my original quote: Quantum mechanics, by dealing with the smallest “building blocks”/waves seems especially promising as a sort of island beach upon which artifacts from the surrounding sea may wash ashore.
Your response: Indeed, quantum fundamentals is a rather active field, both experimentally and theoretically.
My response: So again, it seems like you don’t think I’m “generally wrong” on this point either. I guess that’s good.
--
I actually can’t afford to spend much more time on this. The main idea is that you felt my post was “generally wrong,” but I think that, in actuality, we agree on most of the points. In the places where you believed we disagreed, I don’t actually disagree with you. Again, that’s my bad, since I communicated my viewpoints so poorly that they were misunderstood.
--
Your edit: I looked at your books too, though only the amazon preview. Congrats on the books, writing a few hundred pages is always an accomplishment. I can’t say I see any expertise beyond a pop-sci level, though. This is not a criticism, and I hope you don’t take it as such; these are pop-sci books, and don’t require more expertise than, well, a pop-sci level of expertise. They can be excellent books in their own right, I do not have the expertise to judge science communication.
My response: Thanks for the congrats! Yes, the books were intended to be “for the masses,” which is “pop-sci.” However, I consider the term “pop-sci” as a compliment.
I strongly believe that the average person deserves to know, and is capable of understanding, basic concepts of time (like time dilation) and gravity (from curving spacetime). I realize that people might not need to know about the nature of time and gravity, but somehow it seems like a shame if people don’t ever have a clue about such fundamental aspects of reality. It’s rather like the way the average person could get by without knowing the Earth is round, but I think people deserve at least the opportunity to know this. If I’ve explained advanced concepts in a way that is both correct, and easy to understand in the book, then that makes me really happy.
Also, I didn’t mention the books to demonstrate some great level of “expertise.” In my first post, I didn’t mention them at all. It’s just that the response to my first post was advice that I should go and learn some foundational physics. I mentioned the books at that point only to show that I do indeed have some familiarity w/ foundational physics. After that, someone suggested I am likely to be a “crank” author. I think that means my views would be considered radical or bizarre. I’ll just let people judge for themselves, and share the book titles in a new reply to this post. The reason I didn’t do so in the first place was because I didn’t want to use LW for free marketing. I think it might be necessary to mention the titles though, in order to defend myself from the “crank” speculation?
--
Your continued edit: However, I’m not sure how to convey this without sounding like an elitist asshole, but I’ve never had productive discussions about physics with people who don’t at least have a graduate-level understanding of physics before.
My response: I don’t think you seem like an elitist asshole at all. It just seems that your experience has been different than my experience. My experience is that it’s even possible to have productive discussions about physics with teenagers. As you maybe saw, one of my books introduces teens and tweens to special and general relativity, with that in mind. I guess it partly depends upon how you define “productive discussions.” To me, a productive discussion is one in which one or both parties comes out of the discussion with new insights, or a deeper understanding than they had before the discussion. You probably have a higher ideal for productivity.
In my mind, this LW thread was a productive discussion. I learned that I must have done a very poor job explaining myself in my initial post. That’s the most obvious reason why multiple people misunderstood the points I was trying to make.
I think that next time I should use examples of well-known people who might fall into one camp or another to make my points. The reason I didn’t do so in my original post is because sometimes name-dropping makes it seem like the name-dropped people endorse everything a writer says, which isn’t fair to the name-dropped people. Furthermore, it might seem like the writer agrees with everything the name-dropped person has ever said, which would just be incorrect. Still, as a result of this discussion, I’ve concluded it’s better to use names of people, and then maybe I could add footnotes about a lack of endorsement going either way. I also was probably too sensitive to criticism of my initial post. It’s so easy to read a response in a tone of voice that’s different than that which was intended.
So anyway, thank you for a discussion that was productive (for me at least). It must have taken some time, and I sincerely appreciate your effort.
Thanks for the feedback, but I don’t think it’s about “cognitive rewiring.” It’s more about precision of language and comprehension. You said “AI optimists think AI will go well and be helpful,” but doesn’t everyone believe that is a possibility? The bigger question is what probability you assign to the “go well and be helpful” outcome. Is there anything we can do to increase the probability? What about specific policies? You say you’re an “AI optimist,” but I still don’t know the scope of what that entails w/ specific policies. Does that mean you support open source AI? Do you oppose all AI regulations? What about an AI pause in development for safety? The terms “AI optimist” and “AI pessimist” don’t tell me much on their own.
One inspiration for my post is the now infamous exchange that went on between Yann LeCun and Yoshua Bengio.
As I’m sure you saw, Yann LeCun posted this on his Facebook page (& reposted on X):
“The heretofore silent majority of AI scientists and engineers who
- do not believe in AI extinction scenarios or
- believe we have agency in making AI powerful, reliable, and safe and
- think the best way to do so is through open source AI platforms,
NEED TO SPEAK UP !”
https://www.facebook.com/yann.lecun/posts/pfbid02We6SXvcqYkk34BETyTQwS1CFLYT7JmJ1gHg4YiFBYaW9Fppa3yMAgzfaov7zvgzWl
Yoshua Bengio replied as follows:
Let me consider your three points.
(1) It is not about ‘believing’ in specific scenarios. It is about prudence. Neither you nor anyone has given me any rational and credible argument to suggest that we would be safe with future unaligned powerful AIs and right now we do not know how to design such AIs. Furthermore, there are people like Rich Sutton who seem to want us humans to welcome our future overlords and may *give* the gift of self-preservation to future AI systems, so even if we did find a way to make safe AIs, we would still have a socio-political problem to avoid grave misuse, excessive power concentration and the emergence of entities smarter than us and with their own interests.
(2) Indeed we do have agency, but right now we invest 50 to 100x more on AI capabilities than in AI safety and governance. If we want to have a chance to solve this problem, we need major investments both from industry and governments/academia. Denying the risks is not going to help achieve that. Please realize what you are doing.
(3) Open-source is great in general and I am and have been for all my adult life a big supporter, but you have to consider other values when taking a decision. Future AI systems will definitely be more powerful and thus more dangerous in the wrong hands. Open-sourcing them would be like giving dangerous weapons to everyone. Your argument of allowing everyone to manipulate powerful AIs is like the libertarian argument that everyone should be allowed to own a machine-gun or whatever weapon they want. From memory, you disagreed with such policies. And things get worse as the power of the tool (hence of the weapons derived from it) increases. Do governments allow anyone to build nuclear bombs, manipulate dangerous pathogens, or drive passenger jets? No. These are heavily regulated by governments.
--
[I added spacing to Bengio’s post for readability.]
Media articles about this, along with commenters, have described LeGun as an “AI optimist” and Bengio as an “AI pessimist.”
Just like in how you and I communicated, I think these terms, and even the “good vs bad” dichotomy, radically simplify the nature of the situation. Meanwhile, if the general public were asked what they think the “AI optimist” (supposedly LeGun) or the “pessimist” (supposedly Bengio) believe here, I’m not sure anyone would come back with an accurate response. Thus, the terms are ineffective.
Obviously you can think of yourself with any term you like, but with respect to others, it seems the term “AI strategist” for Bengio—not to mention Eliezer—is more likely to call to mind something closer to what they actually believe.
And isn’t conveyance of accurate meaning the primary goal of communication?