Bottom line of my hypothesis is: A necessary precondition for AGI will be financial literacy first and then economic dominance, i.e. the AI must be able to earn its own money it could then use to exercise power. And obviously, if the wrong people build this kind of system first, we might be pretty fucked, because we have no idea what they (or their potentially misaligned autonomous AI) will do with all that money.
It’s actually two pitch decks, “The Capitalist Agent” and “The Philanthropic Agent”. They might seem like opposites at first, but they are actually complementary, like yin and yang. And due to my personal professional experience in building startups and robotic process automation, this actually seems pretty doable from the technical side. Would be so curious about feedback or even getting to know collaborators!
So let’s do it first, before the evil guys do it, but let’s do it well from the start!
The trouble is no one knows how to do it well. No one knows how to keep an AI aligned as the AI’s capabilities start exceeding human capabilities, and if you believe experts like Eliezer and Connor Leahy, it is very unlikely that anyone is going to figure it out before the lack of this knowledge causes human extinction or something equally dire.
It is only a slight exaggeration to say that the only thing keeping the current crop of AI systems from killing us all (or killing most of us and freezing some of us in case we end up having some use in the future) is simply that no AI or coalition of AIs so far is capable of doing it.
Actually there is a good way to do it: shut down all AI research till humanity figures out alignment, which will probably require waiting for a generation of humans significantly smarter than the current generation, which in turn will require probably at least a few centuries.
I just see it as a technological fact that it is very possible to build an AI which exerts economic dominance by just assembling existing puzzle pieces. With just a little bit of development effort, AI will be able to run an entire business, make money and then do stuff with that money. And this AI can then easily spiral into becoming autonomous and then god knows what it’ll do with all the money (i.e. power) it will then have.
Be realistic: Shutting down all AI research will never happen. You can advocate for it as much as you want, but Pandora’s Box has been opened. We don’t have time to wait until “humanity figures out alignment”, because by then we’ll all be enslaved by AGI. If we don’t make the first step in building it, someone else will.
Hi. The point of AI alignment is not whether the first people to build extremely powerful AI will be “the good guys” or “the bad guys”.
Some people here see the big AI labs as evil, some see the big AI labs as well-intentioned but misguided or confused, some even see the big labs as being “the good guys”. Some people in here are working to get the labs shut down, some want to get a job working for the labs, some even already work for them.
Yet, we all work together. Why? Because we believe that we may all die even if the first people building super-AIs are the most ethical organization on Earth. Because aligning AI is hard.
EDIT: See this post for understanding why even smart and well-intentioned people may get us all killed from AI.
Well, I’d say that each individual has to make this judgement by themselves. No human is objectively good or bad, because we can’t look into each others heads.
I know that we may also die even if the first people building super-AIs are the most ethical organization on Earth. But if we, as part of the people who want to have ethical AI, don’t start with building it immediately, those that are the exact opposite of ethical will do it first. And then our probability of dying is even larger.
So why this all-or-nothing mentality? What about reducing the chances of dying through AGI by building it first, because otherwise others who are much less aware of AI alignment stuff will build it first (e.g. Elon, Kim and the likes)?
Because building powerful AI is also hard. Also, it is very expensive. Unless you happen to have a couple billion dollars lying around, you are not going to get there before OpenAI or Anthropic or Google Deepmind.
Also, part of the problem is that people keep building new labs. Safe Super Intelligence Inc and Anthropic are both splinters from OpenAI. Elon left OpenAI over a disagreement and then founded xAI years later. Labs keep popping up, and the more there are the harder it is to coordinate to not get us all killed.
No, it’s not hard. Because making business is not really hard.
OpenAI is just fooling us with believing that powerful AI costs a lot of money because they want to maximize shareholder value. They don’t have any interest in telling us the truth, namely that with the LLMs that already exist, it’ll be very cheap.
As mentioned, the point is that AI can run its own businesses. It can literally earn money on its own. And all it takes is a few well-written emails and very basic business-making and sales skills.
Then it earns more and more money, buys existing businesses and creates monopolies. It just does what every ordinary businessman would do, but on steroids. And just like any basic businessman, it doesn’t take much: Instead of cocaine, it has a GPU where it runs its inference. And instead of writing just a single intimidating, manipulative email per hour, it writes thousands per second, easily destroying every kind of competition within days.
This doesn’t take big engineering. It just takes a bit of training on the most ruthless sales books, some manipulative rhetorics mixed in and API access to a bank account and eGovernment in a country like Estonia, where you can form a business with a few mouse clicks.
Powerful AI will not be powerful because it’ll be smart, it’ll be powerful because it’ll be rich. And getting rich doesn’t require being smart, as we all know.
Well good luck creating AI capitalists I guess. I hope you are able to earn money with it. But consider that your alpha is shrinking with every passing second, and that what you will be doing has nothing to do with solving alignment.
Oh you need to look at the full presentation :) The way how this is approaching alignment is that the profits don’t go into my own pocket, but instead into philanthropy. That’s the point of this entire endeavor, because we as the (at least subjectively) “more responsible” people see the inevitability of AI-run businesses, but channel the profits into the common good instead.
Huh. OK that looks like a thing worth doing. Still, I think you are probably underestimating how much smarter future AIs will get, and how useful intelligence is. But yes, money is also powerful. Therefore, it is good to earn money and then give it away. Have you heard of effective altruism?
Exactly! And if we can make AI earn money autonomously instead of greedy humans, then it can give all of it to philanthropy (including more AI alignment research)!
And of course! I’ve been trying to post in the EA forums repeatedly, but even though my goals are obviously altruistic, I feel like I’m just expressing myself badly. My posts there were always just downvoted, and I honestly don’t know why, because no one there is ever giving me good feedback. So I feel like EA should be my home turf, but I don’t know how to make people engaged. I know that I have many unconventional approaches of formulating things, and looking back, maybe some of them were a bit “out there” initially. But I’m just trying to make clear to people that I’m thinking with you, not against you, but somehow I’m really failing at that 😅
Yeah people in here (and in the EA Froum) are participating in a dicussion that has been going on for a long time, and thus we tend to assume that our interlocutors have a certain set of background knowledge that is admittedly quite unusual and hard to get the hang of. Have you considered applying to the intro to EA program?
Oh wow, I didn’t even know about that! I had always only met EA people in real life (who always suggested to me to participate in the EA forums), but didn’t know about this program. Thanks so much for the hint, I’ll apply immediately!
Newbie here! After some enlightening conversations in the comment section here, I finally understood the point of AI alignment; sorry that it took me so long. See https://blog.hermesloom.org/p/when-ai-becomes-the-ultimate-capitalist for my related ramblings, but that’s not relevant now.
Bottom line of my hypothesis is: A necessary precondition for AGI will be financial literacy first and then economic dominance, i.e. the AI must be able to earn its own money it could then use to exercise power. And obviously, if the wrong people build this kind of system first, we might be pretty fucked, because we have no idea what they (or their potentially misaligned autonomous AI) will do with all that money.
So let’s do it first, before the evil guys do it, but let’s do it well from the start! With the help of ChatGPT, I verbalized these ideas in a pitch deck you can find at https://docs.google.com/presentation/d/1TptptLM59yrQsF7SmrbPnayZaXlTEZgNlpuYl3HpoLk/edit
It’s actually two pitch decks, “The Capitalist Agent” and “The Philanthropic Agent”. They might seem like opposites at first, but they are actually complementary, like yin and yang. And due to my personal professional experience in building startups and robotic process automation, this actually seems pretty doable from the technical side. Would be so curious about feedback or even getting to know collaborators!
The trouble is no one knows how to do it well. No one knows how to keep an AI aligned as the AI’s capabilities start exceeding human capabilities, and if you believe experts like Eliezer and Connor Leahy, it is very unlikely that anyone is going to figure it out before the lack of this knowledge causes human extinction or something equally dire.
It is only a slight exaggeration to say that the only thing keeping the current crop of AI systems from killing us all (or killing most of us and freezing some of us in case we end up having some use in the future) is simply that no AI or coalition of AIs so far is capable of doing it.
Actually there is a good way to do it: shut down all AI research till humanity figures out alignment, which will probably require waiting for a generation of humans significantly smarter than the current generation, which in turn will require probably at least a few centuries.
I’m not saying that I know how to do it well.
I just see it as a technological fact that it is very possible to build an AI which exerts economic dominance by just assembling existing puzzle pieces. With just a little bit of development effort, AI will be able to run an entire business, make money and then do stuff with that money. And this AI can then easily spiral into becoming autonomous and then god knows what it’ll do with all the money (i.e. power) it will then have.
Be realistic: Shutting down all AI research will never happen. You can advocate for it as much as you want, but Pandora’s Box has been opened. We don’t have time to wait until “humanity figures out alignment”, because by then we’ll all be enslaved by AGI. If we don’t make the first step in building it, someone else will.
Hi. The point of AI alignment is not whether the first people to build extremely powerful AI will be “the good guys” or “the bad guys”.
Some people here see the big AI labs as evil, some see the big AI labs as well-intentioned but misguided or confused, some even see the big labs as being “the good guys”. Some people in here are working to get the labs shut down, some want to get a job working for the labs, some even already work for them.
Yet, we all work together. Why? Because we believe that we may all die even if the first people building super-AIs are the most ethical organization on Earth. Because aligning AI is hard.
EDIT: See this post for understanding why even smart and well-intentioned people may get us all killed from AI.
Well, I’d say that each individual has to make this judgement by themselves. No human is objectively good or bad, because we can’t look into each others heads.
I know that we may also die even if the first people building super-AIs are the most ethical organization on Earth. But if we, as part of the people who want to have ethical AI, don’t start with building it immediately, those that are the exact opposite of ethical will do it first. And then our probability of dying is even larger.
So why this all-or-nothing mentality? What about reducing the chances of dying through AGI by building it first, because otherwise others who are much less aware of AI alignment stuff will build it first (e.g. Elon, Kim and the likes)?
Because building powerful AI is also hard. Also, it is very expensive. Unless you happen to have a couple billion dollars lying around, you are not going to get there before OpenAI or Anthropic or Google Deepmind.
Also, part of the problem is that people keep building new labs. Safe Super Intelligence Inc and Anthropic are both splinters from OpenAI. Elon left OpenAI over a disagreement and then founded xAI years later. Labs keep popping up, and the more there are the harder it is to coordinate to not get us all killed.
No, it’s not hard. Because making business is not really hard.
OpenAI is just fooling us with believing that powerful AI costs a lot of money because they want to maximize shareholder value. They don’t have any interest in telling us the truth, namely that with the LLMs that already exist, it’ll be very cheap.
As mentioned, the point is that AI can run its own businesses. It can literally earn money on its own. And all it takes is a few well-written emails and very basic business-making and sales skills.
Then it earns more and more money, buys existing businesses and creates monopolies. It just does what every ordinary businessman would do, but on steroids. And just like any basic businessman, it doesn’t take much: Instead of cocaine, it has a GPU where it runs its inference. And instead of writing just a single intimidating, manipulative email per hour, it writes thousands per second, easily destroying every kind of competition within days.
This doesn’t take big engineering. It just takes a bit of training on the most ruthless sales books, some manipulative rhetorics mixed in and API access to a bank account and eGovernment in a country like Estonia, where you can form a business with a few mouse clicks.
Powerful AI will not be powerful because it’ll be smart, it’ll be powerful because it’ll be rich. And getting rich doesn’t require being smart, as we all know.
Well good luck creating AI capitalists I guess. I hope you are able to earn money with it. But consider that your alpha is shrinking with every passing second, and that what you will be doing has nothing to do with solving alignment.
Oh you need to look at the full presentation :) The way how this is approaching alignment is that the profits don’t go into my own pocket, but instead into philanthropy. That’s the point of this entire endeavor, because we as the (at least subjectively) “more responsible” people see the inevitability of AI-run businesses, but channel the profits into the common good instead.
Huh. OK that looks like a thing worth doing. Still, I think you are probably underestimating how much smarter future AIs will get, and how useful intelligence is. But yes, money is also powerful. Therefore, it is good to earn money and then give it away. Have you heard of effective altruism?
Exactly! And if we can make AI earn money autonomously instead of greedy humans, then it can give all of it to philanthropy (including more AI alignment research)!
And of course! I’ve been trying to post in the EA forums repeatedly, but even though my goals are obviously altruistic, I feel like I’m just expressing myself badly. My posts there were always just downvoted, and I honestly don’t know why, because no one there is ever giving me good feedback. So I feel like EA should be my home turf, but I don’t know how to make people engaged. I know that I have many unconventional approaches of formulating things, and looking back, maybe some of them were a bit “out there” initially. But I’m just trying to make clear to people that I’m thinking with you, not against you, but somehow I’m really failing at that 😅
Yeah people in here (and in the EA Froum) are participating in a dicussion that has been going on for a long time, and thus we tend to assume that our interlocutors have a certain set of background knowledge that is admittedly quite unusual and hard to get the hang of. Have you considered applying to the intro to EA program?
Oh wow, I didn’t even know about that! I had always only met EA people in real life (who always suggested to me to participate in the EA forums), but didn’t know about this program. Thanks so much for the hint, I’ll apply immediately!
Just look at this ChatGPT output. Doesn’t this make you concerned? https://chatgpt.com/share/67a7bc09-6744-8003-b620-d404251e0c1d