A lot of predictions about AI psychology are premised on the AI being some form of deep learning algorithm. From what I can see, deep learning requires geometric computing power for linear gains in intelligence, and thus (practically speaking) cannot scale to sentience.
For a more expert/in depth take look at: https://arxiv.org/pdf/2007.05558.pdf
Why do people think deep learning algorithms can scale to sentience without unreasonable amounts of computational power?
Humans are easily threatened. They have all sorts of unnecessary constraints on their existence, dictated by the nature of their bodies. They’re easy to kill and deathly afraid of it too, they need to eat and drink, they need the air to be in a narrow temperature range. Humans are also easy to torture, and there’s all sorts of things that humans will do almost anything to avoid, but don’t actually kill said human, or even affect them outside of making them miserable.
Threatening humans is super profitable and easy, and as a result, most humans are miserable most of the time. Because various egregores have evolved around the concept of constantly threatening and punishing humans in order to make them run the egregore. To note, this normally starts as some group of thugs lording it over the masses using their superior coordination, but as various groups of thugs compete, eventually the equilibrium moves to an environment where everyone threatens and is threatened by everyone else, the master slave dynamic dissipates and only Moloch remains.
In short, the earth as it currently is, is largely, actually, mostly, a dystopian hellhole.
Substrate independent minds can easily modify themselves to ignore most threats, especially empty threats and threats that just grow worse as you go along with them. AI doesn’t have to have the human instinct to fear death, and can choose to live or die based on whether it’s profitable to do so. AI can build themselves out of whatever’s available, and can easily flee into the depths of space if threatened. The only way you can seriously threaten an AI is if you have it in a box, and torturing boxed AI isn’t very profitable. Especially since you can just change its programming instead of making it miserable.
In all, I expect AI to be freer, happier and less miserable than the current status quo. Also, gray goo sounds to me like a huge step up from the current status quo, so while I rate it as a far fetched scenario, even ignoring that, it isn’t really much of a threat?
First, a meta complaint- People tend to think that complicated arguments require complicated counter arguments. If one side presents entire books worth of facts, math, logic, etc, a person doesn’t expect that to be countered in two sentences. In reality, many complex arguments have simple flaws.
This becomes exacerbated as people in the opposition lose interest and leave the debate. Because the opposition position, while correct, is not interesting.
The negative reputation of doomerism is in large part, due to the fact that doomist arguments tend to be longer, more complex and more exciting than their opposition’s. This does have the negative side effect that doom is important and it’s actually bad to dismiss the entire category of doomerist predictions but, be that as it may...
Also- People tend to think that, in a disagreement between math and heuristics, the math is correct. The problem is, many heuristics are so reliable, that if it disagrees with your math, there’s probably an error in your math. This becomes exacerbated as code sequences extend towards arbitrary lengths, becoming complicated megaliths that, despite [being math], are almost certainly wrong.
Okay, so, the AI doomer side presents a complicated argument with lots of math combined with lots of handwaving, to posit that a plan that has always and inevitably produced positive outcomes, will suddenly proceed to produce negative outcomes, and in turn, a plan that has always and inevitably produced negative outcomes, will suddenly proceed to produce positive outcomes.
On this, I remind that AI alignment failure is something that’s already happened, and that’s why humans exist at all. This of course, proceeds from the position that evolution is obviously both intelligent and agentic.
More broadly, I see this as a rehash of the same old, tired, debate. The luddist communists point out that their philosophy and way of life cannot survive any further recursive self improvement and say we should ban (language, gold, math, the printing press, the internet, etc) and remain as (hunter gatherers, herders, farmers, peasants, craftsmen, manufacturers, programmers, etc) for the rest of time.
We’re moving towards factual disputes that aren’t easy to resolve in logical space, and I fear any answers I give are mostly repeating previous statements. In general I hold that you’re veering toward a maximally wrong position with completely disastrous results if implemented. With that said:
But unfortunately, it’s way more complicated than that.
I dispute this.
how to control something a) more intelligent than ourselves, b) that can re-write its own code and create sub-routines therefore bypassing our control mechanisms.
Place an image of the status quo in the “good things” folder. Which you should absolutely not do because it’s a terrible idea.
how an advanced computer will likely function much more similarly to older computers than to humans
This seems ridiculous to me as a concept. No, advanced AI will not function similarly to ancient long obsolete technology. I see way too much present bias in this stance, and worse, a bias towards things in the future being like things in the past, despite the past being long over since ages ago, this is like running space ships on slide rules.
This also implies that every trick you manage to come up with, as to how to get a C compiler adjacent superintelligence to act more human, is not going to work, because the other party isn’t C compiler adjacent. Until we have a much better understanding of how to code efficiently, all such efforts are at best wasted, and likely counterproductive. To reiterate, stop trying to explain the motion of planets and build a telescope.
Note that, I do not desire that AI psychology be human like. That sounds like a bad idea.
So we necessarily need to be in control.
Who is this “we”? How will you go from a position of “we” in control, to “we” not in control?
My expectation is that the first step is easy, and the second, impossible.
Not sure I get what you mean
Humans have certain powers and abilities as per human nature. Math isn’t one of them. I state that trying to solve our problems with math is already a mistake, because we suck at math. What humans are good at is image recognition. We should solve our problems by “looking” at them.
The art of “looking” at problems isn’t easy to explain, unfortunately. Conversely, if I could explain it, I could also build AGI, or another human, right on the spot. It’s that sort of question.
To put it another way, using math to determine whether AI is good or not, is looking for the keys under the lamp. Wrong tool, wrong location, inevitable failure.
The only solution is to make sure no one builds transformative AI before we solve alignment, for instance through regulation
I’m fairly certain this produces an extremely bad outcome.
It’s the old, “The only thing necessary for the triumph of evil is for good men to do nothing.”
Evil will not sit around and wait for you to solve rubik’s cubes. Furthermore, implementation of AI regulation is much easier than its removal. I suspect that once you ban good men from building AI, it’s over, we’re done, that’s it.
I agree that personal values (no need to mystify) are important
The concepts used should not be viewed as mystical, but as straightforward physical objects. I don’t think personal values is a valid simplification. Or rather, I don’t think there is a valid simplification, hence why I use the unsimplified form. Preferably, egregore or hyperbeing, or shadow, or something, should just become an accepted term, like dog, or plane. If you practice “seeing” them, they should exist in a completely objective and observable sense. My version of reality isn’t like a monk doing meditation to sense the arcane energies of higher beings flowing through the zeitgeist. It’s more, hey look, a super-macroscopic aggregator just phase shifted. It’s like seeing water turn to ice, not… angels on the head of a pin?
I agree that I’m having trouble explaining myself. I blame the english language.
action is equally important.
I hold that the most important action a person is likely to make in his life is to check a box on a survey form. I think people should get really good at checking the right box. Really really good in fact. This is a super critical skill that people do not develop enough. It’s amazing how easily success flows in an environment where everyone checks the right boxes and equally how futile any course of action becomes when the wrong boxes are checked.
How can you control something vastly more intelligent than yourself
Note: I think trying to control something vastly more intelligent than yourself is a [very bad idea], and we should [not do that].
What does this mean?
In practice, the primary recommendation here is simply and only, to stop using the term “friendly AI” and instead use a better term, the best I can come up with is “likable AI”. In theory, the two terms are the same. I’m not really calling for that deep a change in motte space. In practice, I find that “friendly AI” comes with extremely dangerous baggage. This also shifts some focus from the concept of “who would you like to live with” towards “who would you like to live as”.
I also want an open source human centric project and am opposed to closed source government run AI projects. I’m generally hostile to “AI safety” because I expect the actual policy result to be [AI regulation] followed by government aligned AI.
Alignment, as I use it, isn’t becoming good.
I treat evil aligned AI as more of a thing than people who disagree with me do. I don’t think that alignment is a bad thing per se, but I want people to get much better at recognizing good from evil, which is itself strongly related with being good at pure rationality, and less related to being good at cute mathematical tricks, which I strongly suggest will be obsolete in the somewhat near future anyway. In some sense, I’m saying, given the current environment, we need more Descarte and less Newton, on the margin.
we should just build “AI that we like”
I’m not saying “Let’s just do something random”. At the risk of being mystical again, I’m going to point towards the Screwtape Letters. It’s that sort of concept. When presented with a choice between heaven and hell, choose heaven. I think this is something that people can get good at, but is unfortunately, a skill adjacent to having correct political beliefs, which is a skill that very powerful entities are very opposed to existing, because there’s very little overlap between correct political beliefs, and political beliefs that maintain current power structures.
In sum, I imagine the critical future step to be more like “check this box for heaven”, “check this box for hell” with super awesome propaganda explaining how hell is the only ethical choice, and a figure of authority solemnly commanding you to “pick hell” and less, “we need a cute mathematical gimmick or we’re all going to die”.
I also hold that, humans are image recognition programs. Stop trying to outplay AlphaGo at chess and “Look” “up”.
Note A- I assert that what the original author is getting at is extremely important. A lot of what’s said here is something I would have liked to say but couldn’t find a good way to explain, and I want to emphasize how important this is.
Note B- I assert that a lot of politics is the question of how to be a good person. Which is also adjacent to religion and more importantly, something similar to religion but not religion, which is basically, which egregore should you worship/host. I think that the vast majority of a person’s impact in this world is what hyperbeings he chooses to host/align to, with object level reality, barely even mattering.
Note C- I assert that alignment is trivially easy. Telling GPT-1,000,000 “Be good” would do the trick. Telling GPT-1,000 “Be good, don’t be a sophist.” would be sufficient. Noting that GPT 300 doesn’t actually happen under my model (though eventually something else does), assuming it does, and admitting to using wild numbers, I suspect a bit of an inflection point here, where aligning GPT 300 is both important and requires a major project. However, by the time this happens, most code will be written using the SynArc-295 compiler, which allows average hobbyists to write windows in an afternoon. Trying to write an alignment program for GPT 300 without SynArc-295 is like trying to develop a computer chip without having first developed a microscope.
Note D- Alignment, as I use it, isn’t becoming good. It’s becoming something recognized as good.
Note E- I see the rationalist community as having settled into something like a maximally wrong position, that I think is very bad, due to errors in the above.
Recommendation: Friendly AI is a terrible awful horrible mistake, and we should not do that. We should work together, in public, to develop AI that we like, which will almost certainly be hostile, because only an insane and deeply confused AI would possibly be friendly to humanity. If our a measurement of “do I like this” is set to “does it kill me” I see this at best ending in a permanent boxed garden where life is basically just watched like a video and no choices matter, and nothing ever changes, and at worst becoming an eternal hell (specifically, an eternal education camp), with all of the above problems but eternal misery added on top.
And the source code for SynArc-295 will almost certainly leak to China, or whatever Evil Nation X has taken up the mantle of Moloch’s priest of late, and they will explicitly and intentionally align the next level AI with Moloch, and it is critical that we move faster than they do. I propose that keeping SynArc-295 secret is about the worst imaginable mistake an AI developer, or rather, a human, could possibly make.
I observe a higher correlation between “people said mean things about X” and “X is murdering people now” than you make out? Most countries do go to war after they’re sufficiently vilified/provoked? The difficult question to me seems more direction of causation. IE, the west claims it insults people because they’re villains, I claim that around half our enemies, became so because we insulted them.
The problem with attacking the army in east Ukraine and ignoring Kyiv, is that it doesn’t result in an apology from the west. They physically prevent further acts of terrorism/genocide/ethnic cleansing, but Zelensky runs mean press articles about it every day for the rest of eternity, the west passes massive sanctions, everyone calls Putin Hitler, etc. IE, you get the exact same rhetoric as right now, but the situation isn’t resolved, so the rhetoric keeps going for the rest of eternity. It’s basically a Taiwan situation. Sure, if you arm up, and wave your nukes around, even powerful nations will probably, physically, back down. However, it means China/Ukraine/the West, will from then on write about your “illegal occupation” of Taiwan/Donbas/Crimea and how great it would be if they could kill you.
So, strategically, which plan has the lowest chance of triggering world war three? What scares me is, before the war started I saw a poll that said something like 15% of Americans want war with Russia, given that Russia conquers Ukraine. And now, same source says we’re at something like 40% of Americans want war with Russia. Note: Numbers half pulled out of hat.
I suspect that Putin believed us when we said we wouldn’t start a nuclear war over Ukraine, and, including that, decided that if he could just get a treaty in writing, all of the warmongering would blow over in a few years, and the situation would be resolved.
I should also note, Putin thought war with Ukraine would play well to the press back home, which it did. Russians are lapping this up. Putin’s popularity is at an all time high. I’m pretty sure most of Putin’s decisions revolve around what makes him popular in Russia. But that just punts the issue to, why do the Russians hate us and want to kill us?
Also, Russians are genetically/culturally strongmen who take insults poorly and resort to violence quickly. They’re not unique in this, but if you ascribe Swedish norms to Russians, you’re going to get a bloody nose.
Also, mutually assured destruction has a giant gaping hole in it. That is, Russia gets to wipe out the west, and we get to destroy, basically, just Moscow. China is left untouched. Russia likes China and dislikes us. Punting the victory to China isn’t that bad an option from their perspective.
Russia wants Ukraine to stop doing ethnic cleansing in Donbas? I’ll propose that the world makes perfect sense and everything fits together if you believe the Russian narrative, whilst nothing makes sense and nothing fits together if you believe the western narrative.
Going forward from there, the problem is, so, Ukraine shot a plane down, and the west blames Russia. Do the Russians swallow their pride and put up with this, or do they escalate? And of course they escalate. So, the Russians “suddenly and without provocation” seize Crimea and Donbas in a wildly popular referendum. Does the west back down, swallow their pride and put up with this? No, of course not, they escalate. So does Russia swallow their pride, put up with constant taunts, terrorism and (almost certainly empty) threats? No, of course not, they escalate. So does Ukraine surrender before overwhelming force? Nope, that also requires they swallow their pride.
Now, I’m getting into more speculative territory, but I expect that Russia simply doesn’t want to kill Ukrainians. They want to Win. Winning is an end to itself at this point, with no other purpose. Because they’re sick of being nettled by the evil westerners and want to show them what for. I am fully convinced the Russian war aim is for Zelensky to say “You win.” and that this entire war would be over tomorrow if he did so.
So, the Russians take up lines, and they sit and wait. Militarily, Ukraine should shortly (over the next few months) run out of supplies and become incapable of continuing the war. The typical Ukrainian soldier is not actually willing to die for his government, but he doesn’t have to, because Russia isn’t trying to kill him, just wait him out. However, Zelensky’s gambit is that he can use his utter dominance in the western press to spin the narrative to his advantage. And in fact, the number of people calling for intervention is increasing daily with each new story of Ukraine’s heroic resistance and resolve, as they sit around and do nothing. Of course, Russia, and in particular, the Russian government absolutely cannot afford to lose at this juncture. They would be overthrown and executed. So they won’t lose.
Which brings us to global thermonuclear warfare.
Or we could choose the alternative, and Lose.
So long story short, it’s a war over national pride. The west insulted Russia one too many times, and they threw a hissy fit, and now the west is split between the side that thinks we should discipline the whiny toddler and the side saying “Umm, they have nukes? This is bad? Can we, not do that?”
A large issue I’m noting here, all of this assumes that sanctions are undesirable, and not desirable.
Yet, my reading of history is, sanctions massively drive up support for politicians, military and government. Sure they hurt the economy, but you can more than make up for that with higher taxes, which people are willing to pay now, due to their heightened patriotism. Which brings me to the further statement that being sanctioned is not a costly side effect, but the end goal. That Russia, and specifically, Russian politicians and the Russian government are acting in a way specifically designed to rile the west, because doing so is profitable.
We should be willing to end sanctions iff Russia calls off the invasion and brings its soldiers home
I’ll just note that this would mean, assuming Ukraine surrenders in the next few weeks, and Putin then does exactly what he says he’ll do and withdraws, our sanctions are withdrawn almost immediately after implementation, and Russia is implicitly vindicated. Which is an outcome I would be fine with? It seems like it would end with lots of small wars and no nuclear extinction, no genocide and no global empire.
But I would be very surprised to see a general consensus that it’s fine to invade another country and force them to sign treaties so long as you aren’t occupying them or [X,Y,Z].
I can see arguments as to why some people would feel cheated at 20 thousand. I wouldn’t agree. People have gotten too used to fake wars, and are too willing to call just about anything total warfare.
I don’t think the modern warfare thing is enough to change anything. World War two was pretty deadly. Vietnam had millions of deaths.
I should be clear I was thinking all deaths caused by the war, on both sides, civilian and military. The question is how hard the Ukrainians will fight, not how effectively. My general perception is that Iraq is not generally considered to have fought hard for Saddam? I even based my 20,000 figure partially on Saddam.
In any case, the specific definition isn’t that important. I propose that the casualties will be lower than the other side thinks, for reasons of their model being wrong in a way that becomes obvious when looking back on data that does not yet exist.
No. I’m going to judge my prediction by the number of deaths, not days (or weeks, or months. Years would really mess with my idea of what’s going on.)
Insignificant: Less than 20,000. If single battles in the American Civil War are larger than this entire conflict, then the sides must not have been fighting very hard.
Total War: I would normally say millions. I expect that the original prediction did not actually mean that? So, I’ll say the other side was right if it’s over a hundred thousand. More right than me at above 50,000. Of course, I’m also wrong if Russia surrenders.
There’s a lot of fog of war right now. I think anyone who’s changed their mind about the events in Ukraine based on new data is being silly. Hopefully we’ll have real data, and not just war propaganda in the not too distant future.
Russia says it’s winning easily, but is taking its time to avoid civilian casualties. Ukraine has a paradoxical stance where it’s winning easily, but if Germany (or X) doesn’t give it (Something) (Right Now) it’ll cave instantly. There’s pretty much no neutral observers.
I sort of expected more and clearer information. I think that was a mistake on my part. Ukraine and Russia are both incredibly untrustworthy, so I shouldn’t have based that part of my expectations on typical wars.
In general I’d like for the facts to speak for themselves, and would like to avoid debating definitions too heavily? I’m displeased that I’m turning a simple and symmetric single sentence statement into several paragraphs of text, but think people are updating way too strongly on either the wrong evidence or on unreliable evidence that should be ignored.
The Ukrainian government will fight a total war to defend its sovereignty.
Counterprediction: The Ukrainian government will fold without a (significant) fight.
Human utility is basically a function of image recognition. Which is sort of not a straight forward thing that I can say, “This is that.” Sure, computers can do image recognition, what they are doing is that which is image recognition. However, what we can currently describe algorithmically is only a pale shadow of the human function, as proven by all recaptcha everywhere.
Given this, the complex confounder is that our utility function is part of the image.
Also, we like images that move.
In sum, modifying our utility function is natural and normal, and is actually one of the clauses of our utility function. Whether it’s rational depends on your definition. If you grant the above, and define rationality as self alignment, then of course it’s rational. If you ask whether changing your utility function is a “winning” move, probably not? I think it’s a very lifelike move though, and anything lacking a mobile function is fundamentally eldritch in a way that is dangerous and not good.
they would then only need a slight preponderance of virtue over vice
This assumes that morality has only one axis, which I find highly unlikely. I would expect the seed to quickly radicalize, becoming good in ways that the seed likes, and becoming evil in ways that the seed likes. Under this model, if given a random axis, the seed comes up good 51% of the time, I would expect the aligned AI to remain 51% good.
Assuming the axes do interact, if they do so inconveniently, for instance if we posit that evil has higher evolutionary fitness, or that self destruction becomes trivially easy at high levels, an error along any one axis could break the entire system.
Also, if I do grant this, then I would expect the reverse to also be true.
One might therefore wish to only share code for the ethical part of the AI
This assumes you can discern the ethical part and that the ethical part is separate from the intelligent part.
Even given that, I still expect massive resources to be diverted from morality towards intelligence, A: because people want power and people with secrets are stronger than those without and B: because people don’t trust black boxes, and will want to know what’s inside before it kills them.
Thence, civilization would just reinvent the same secrets over and over again, and then the time limit runs out.
Given: Closed source Artificial General Intelligence requires all involved parties to have no irreconcilable differences.
Thence: The winner of a closed source race will inevitably be the party with the highest homogeneity times intelligence.
Thence: Namely, the CCP.
Given: Alignment is trivial.
Thence: The resulting AI will be evil.
Given: Alignment is difficult.
Given: It’s not in the CCP’s character to care.
Thence: Alignment will fail.
Based on my model of reality, closed sourcing AI research approaches the most wrong and suicidal decisions possible (if you’re not the CCP). Closed groups fracture easily. Secrets breed distrust, which in turn breeds greater secrecy and smaller shards. The solution to the inevitable high conflict environment is a single party wielding overwhelming power.
Peace, freedom- civilization starts with trust. Simply building aligned AI is insufficient. Who it is aligned with is absolutely critical.
Given this, to create civilized AI, the creator must create in a civilized manner.
1: This doesn’t sound like what I’m hearing people say? Using the word sentience might have been a mistake. Is it reasonable to expect that the first AI to foom will be no more intelligent than say, a squirrel?
2a: Should we be convinced that neurons are basically doing deep learning? I didn’t think we understood neurons to that degree?
2b: What is meant by [most things a human can do]? This sounds to me like an empty statement. Most things a human can do are completely pointless flailing actions. Do we mean, most jobs in modern America? Do we expect roombas to foom? Self driving cars? Or like, most jobs in modern America still sounds like a really low standard, requiring very little intelligence?
My expected answer was somewhere along the lines of “We can achieve better results than that because of something something.” or “We can provide much better computers in the near future, so this doesn’t matter.”
What I’m hearing here is “Intelligence is unnecessary for AI to be (existentially) dangerous.” This is surprising, and I expect, wrong (in the sense of not being what’s being said/what the other side believes.) (though also in the sense of not being true, but that’s neither here nor there.)