Seems to me the “killer app” is not superintelligence per se, but superintelligence plus self-modification. With highly intelligent people that problem is often that they can’t modify themselves, so in addition to their high intelligence they also have some problems they can’t rid of.
Maybe Kasparov plays chess because he genuinely believes that playing chess is the most useful thing he could ever do. But more likely there are other reasons; for example he probably enjoys playing chess, and the more useful things are boring to him. Or maybe he is emotionally insecure and wants to stay in an area he is already good at, because he couldn’t emotionally bear giving up the glory and starting something as a total newbie, even if a few years later it would pay off. (This is just a random idea; I have no idea what Kasparov really is like. Also, maybe he is doing other things too, we just don’t know about them.)
Imagine a person who would be able to edit their own mind. For example, if they would realize that their plans would advance further if they had better social skills, they would (a) temporarily self-modify to enjoy scientifically studying social skills, then (b) do a research about social skills and find out what is evidence-based, and finally (c) self-modify to have those traits that reliably work. And then they would approach every obstacle in the same way. Procrastinating too much? Find the parts of your mind that make you do so, and edit them out. Want to stay fit, but you hate exercising? Modify yourself to enjoy exercising, but only for a limited time every day. Want to study finance, but have a childhood emotional trauma related to money? Remove the trauma.
You would get a person who, upon seeing an optimal way, would start following that way. They would probably soon realize that their time is a limited resource, and start employing other people to do some work for them. They would use various possible computer tools, maybe even employ people to improve those tools for them. They would research possibilities of self-improvement, use them for themselves, and also trade the knowledge with other people for resources or loyalty. After a while, they would have an “army” of loyal improved followers, and would also interact a lot with the rest of the world. And that would be just a beginning.
Maybe for a human there would be some obstacle, for example that the average human life is too short to research and implement immortality. Or maybe they would reach the escape velocity and become something transhuman.
Most savants simply don’t care about changing the world
I would guess they still find some things frustrating (e.g. when someone or something is interrupting them from their hobby), they are just no strategic enough to remove all sources of frustration. Either they do not bother to plan long-term, or they don’t believe such long-term planning could work.
the 190 IQ son of a waiter is probably a cook
Let’s imagine a country with a strict caste system, where the IQ 190 person born in the lower caste must remain there. If the society is really protected against any attempts to break the system, for example if the people from the lower caste are forbidden all education and internet, and are under constant surveillance, there is probably not much he could do. But if it’s a society more or less like ours, only it is illegal and socially frowned upon hiring people to do other caste’s work, a strategic person could start exploring possibilites to cheat—for example, could they fake a different, higher-caste identity? Or perhaps move to a country without the caste system. Or if there are exceptions where the change of caste is allowed, they would try one, and could try to cheat to get the exception easier. (For example, if a higher-caste person can “adopt” you into their caste, you could try to make a deal with one, or blackmail one, or maybe somehow fake the whole process of being adopted.) They could also try to somehow undermine the caste system; create a community that ignores the caste rules; etc.
Hm, that is a better point, it seems then most of my objections are just to the wording. Most intelligent people are also shy etc. and that is why they end up being math researchers instead of being Steve Jobs. If an intelligent person could edit in his mind courage, dedication, charme… that would be powerful.
But I think self-modification would be powerful even without very high IQ, 120 would already make one pretty succesful.
Or is it more IQ being necessary for efficient self-modification?
My point is, this sounds like a powerful combination, but probably not the intelligence explosion kind.
The caste stuff: really elegant steelmanning, congrats. But I think kind of missing the point, probably I explained myself wrong. Basically an IQ-meritocracy requires a market based system, exchanged based one, where what you get is very roughly proportional to what you give to others. However most of the planet is not exchange based but power based. This is where intelligence is less useful. Imagine trying to compete with Stalin for the job of being Lenin’s successor. What traits you need for it? First of all, mountains of courage, that guys is scary. Of course if you can self-edit, that is indeed extremely helpful in it… I did not factor that in. But broadly speaking, you don’t just outsmart him. Power requires other traits. And of course it can very well be that you don’t want power, you want to be a researcher… but in that situation you are forced to take orders from him so you may still want to topple the big boss or something.
Now of course if we see intelligence as simply the amount of sophistication put into self-editing, so seeing a higher intelligence as something that can self-edit e.g. charisma better than a lower intelligence… then these possibilities are indeed there. I am just saying, still no intelligence explosion, more like everything explosion, or maybe everything else but intelligence explosion. Charisma explosion, and so on… but I do agree that this combined can be very powerful.
Or is it more IQ being necessary for efficient self-modification?
Sounds like false dilemma if IQ is one of those things that can be modified. :D
To unpack the word, IQ approximately means how complex concepts can you “juggle” in your head. Without enough IQ, even if you had an easy computer interface to modify your own brain, you wouldn’t understand what exactly you should do to achieve your goals (because you wouldn’t sufficiently understand the concepts and the possible consequences of the changes). That means, you would be making those changes blindly… and you could get lucky and hit the path where your IQ increases so then you can start making reliably the right steps, or you could set yourself on a way towards some self-destructive attractor.
As a simple example, a stupid person could choose to press a button that activates their reward center… and would keep doing that until they die from starvation. Another example would be self-modification where you lose your original goals, or lose the will to further self-improve, etc. This does not have to happen in one obvious step, but could be a subtle cumulative consequence of many seemingly innocent steps. For example, a person could decide that being fit is instrumentally useful for their goals, so they would self-modify to enjoy exercise, but would make a mistake of modifying themselves too much, so now they only want to exercise all day long, and no longer care about their original goals. Then they would either stop self-modifying, or self-modify merely to exercise better.
It also depends on how complex would be the “user interface to modify your own brain”. Maybe IQ 120 would not be enough to understand it. Maybe even IQ 200 wouldn’t. You could just see a huge network of nodes, each connected to hundreds of other nodes, each one with insanely abstract description… and either give up, or start pushing random buttons and most likely hurt yourself.
So basicly the lowest necessary starting IQ is the IQ you need to self-modify to safely enough increase your IQ. This is a very simple model which assumes that if IQ N allows you to get to IQ N+1, then IQ N+1 probably allows you to get to IQ N+2. The way does not have to be this smooth; there may be a smooth increase up to some level, which then requires a radical change to overcome; or maybe the intelligence gains at each step will decrease.
Imagine trying to compete with Stalin for the job of being Lenin’s successor. What traits you need for it?
Courage, social skills, ability to understand how politics really works. You should probably start in some position in army or secret service, some place where you can start building your own network of loyal people, without being noticed by Stalin. Or maybe you should start as a crime boss in hiding, I don’t know.
To unpack the word, IQ approximately means how complex concepts can you “juggle” in your head
I think I agree with this, this is why I don’t understand how can intelligence be defined as goal-achieving ability. When I am struggling with the hardest exercises on the Raven test, what I wish I had more is not some goal-achieving power but something far simpler, something akin to short term memory. So when I wish for more intelligene, I just wish for a bit more RAM in short-term memory, so more detailed, more granular ideas can be uploaded into tumble space. No idea why should it mean goal-achieving or optimizing ability. And for AI IQ sounds like entirely hardware...
Basically an IQ-meritocracy requires a market based system, exchanged based one, where what you get is very roughly proportional to what you give to others. However most of the planet is not exchange based but power based. This is where intelligence is less useful.
Intelligence is very useful in conflicts. If you can calculate beforehand whether you will win or lose a battle you don’t have to fight the battle if you think you will lose it.
In our modern world great intelligence means the ability to hack computers. Getting information and being able to alter email message that person A sends person B. That’s power.
Getting money because you are smart enough to predict stock market movements is another way to get power.
And of course it can very well be that you don’t want power, you want to be a researcher… but in that situation you are forced to take orders from him so you may still want to topple the big boss or something.
Stalin was likely powerful but a lot of today’s political leaders are less powerful.
Peter Thiel made the point that Obama probably didn’t even knew that the US was wiretapping Angela Merkel. It’s something that the nerds in the NSA decided in their Star Trek bridge lookalike.
Seems to me the “killer app” is not superintelligence per se, but superintelligence plus self-modification. With highly intelligent people that problem is often that they can’t modify themselves, so in addition to their high intelligence they also have some problems they can’t rid of.
Maybe Kasparov plays chess because he genuinely believes that playing chess is the most useful thing he could ever do. But more likely there are other reasons; for example he probably enjoys playing chess, and the more useful things are boring to him. Or maybe he is emotionally insecure and wants to stay in an area he is already good at, because he couldn’t emotionally bear giving up the glory and starting something as a total newbie, even if a few years later it would pay off. (This is just a random idea; I have no idea what Kasparov really is like. Also, maybe he is doing other things too, we just don’t know about them.)
Imagine a person who would be able to edit their own mind. For example, if they would realize that their plans would advance further if they had better social skills, they would (a) temporarily self-modify to enjoy scientifically studying social skills, then (b) do a research about social skills and find out what is evidence-based, and finally (c) self-modify to have those traits that reliably work. And then they would approach every obstacle in the same way. Procrastinating too much? Find the parts of your mind that make you do so, and edit them out. Want to stay fit, but you hate exercising? Modify yourself to enjoy exercising, but only for a limited time every day. Want to study finance, but have a childhood emotional trauma related to money? Remove the trauma.
You would get a person who, upon seeing an optimal way, would start following that way. They would probably soon realize that their time is a limited resource, and start employing other people to do some work for them. They would use various possible computer tools, maybe even employ people to improve those tools for them. They would research possibilities of self-improvement, use them for themselves, and also trade the knowledge with other people for resources or loyalty. After a while, they would have an “army” of loyal improved followers, and would also interact a lot with the rest of the world. And that would be just a beginning.
Maybe for a human there would be some obstacle, for example that the average human life is too short to research and implement immortality. Or maybe they would reach the escape velocity and become something transhuman.
I would guess they still find some things frustrating (e.g. when someone or something is interrupting them from their hobby), they are just no strategic enough to remove all sources of frustration. Either they do not bother to plan long-term, or they don’t believe such long-term planning could work.
Let’s imagine a country with a strict caste system, where the IQ 190 person born in the lower caste must remain there. If the society is really protected against any attempts to break the system, for example if the people from the lower caste are forbidden all education and internet, and are under constant surveillance, there is probably not much he could do. But if it’s a society more or less like ours, only it is illegal and socially frowned upon hiring people to do other caste’s work, a strategic person could start exploring possibilites to cheat—for example, could they fake a different, higher-caste identity? Or perhaps move to a country without the caste system. Or if there are exceptions where the change of caste is allowed, they would try one, and could try to cheat to get the exception easier. (For example, if a higher-caste person can “adopt” you into their caste, you could try to make a deal with one, or blackmail one, or maybe somehow fake the whole process of being adopted.) They could also try to somehow undermine the caste system; create a community that ignores the caste rules; etc.
Hm, that is a better point, it seems then most of my objections are just to the wording. Most intelligent people are also shy etc. and that is why they end up being math researchers instead of being Steve Jobs. If an intelligent person could edit in his mind courage, dedication, charme… that would be powerful.
But I think self-modification would be powerful even without very high IQ, 120 would already make one pretty succesful.
Or is it more IQ being necessary for efficient self-modification?
My point is, this sounds like a powerful combination, but probably not the intelligence explosion kind.
The caste stuff: really elegant steelmanning, congrats. But I think kind of missing the point, probably I explained myself wrong. Basically an IQ-meritocracy requires a market based system, exchanged based one, where what you get is very roughly proportional to what you give to others. However most of the planet is not exchange based but power based. This is where intelligence is less useful. Imagine trying to compete with Stalin for the job of being Lenin’s successor. What traits you need for it? First of all, mountains of courage, that guys is scary. Of course if you can self-edit, that is indeed extremely helpful in it… I did not factor that in. But broadly speaking, you don’t just outsmart him. Power requires other traits. And of course it can very well be that you don’t want power, you want to be a researcher… but in that situation you are forced to take orders from him so you may still want to topple the big boss or something.
Now of course if we see intelligence as simply the amount of sophistication put into self-editing, so seeing a higher intelligence as something that can self-edit e.g. charisma better than a lower intelligence… then these possibilities are indeed there. I am just saying, still no intelligence explosion, more like everything explosion, or maybe everything else but intelligence explosion. Charisma explosion, and so on… but I do agree that this combined can be very powerful.
Sounds like false dilemma if IQ is one of those things that can be modified. :D
To unpack the word, IQ approximately means how complex concepts can you “juggle” in your head. Without enough IQ, even if you had an easy computer interface to modify your own brain, you wouldn’t understand what exactly you should do to achieve your goals (because you wouldn’t sufficiently understand the concepts and the possible consequences of the changes). That means, you would be making those changes blindly… and you could get lucky and hit the path where your IQ increases so then you can start making reliably the right steps, or you could set yourself on a way towards some self-destructive attractor.
As a simple example, a stupid person could choose to press a button that activates their reward center… and would keep doing that until they die from starvation. Another example would be self-modification where you lose your original goals, or lose the will to further self-improve, etc. This does not have to happen in one obvious step, but could be a subtle cumulative consequence of many seemingly innocent steps. For example, a person could decide that being fit is instrumentally useful for their goals, so they would self-modify to enjoy exercise, but would make a mistake of modifying themselves too much, so now they only want to exercise all day long, and no longer care about their original goals. Then they would either stop self-modifying, or self-modify merely to exercise better.
It also depends on how complex would be the “user interface to modify your own brain”. Maybe IQ 120 would not be enough to understand it. Maybe even IQ 200 wouldn’t. You could just see a huge network of nodes, each connected to hundreds of other nodes, each one with insanely abstract description… and either give up, or start pushing random buttons and most likely hurt yourself.
So basicly the lowest necessary starting IQ is the IQ you need to self-modify to safely enough increase your IQ. This is a very simple model which assumes that if IQ N allows you to get to IQ N+1, then IQ N+1 probably allows you to get to IQ N+2. The way does not have to be this smooth; there may be a smooth increase up to some level, which then requires a radical change to overcome; or maybe the intelligence gains at each step will decrease.
Courage, social skills, ability to understand how politics really works. You should probably start in some position in army or secret service, some place where you can start building your own network of loyal people, without being noticed by Stalin. Or maybe you should start as a crime boss in hiding, I don’t know.
I think I agree with this, this is why I don’t understand how can intelligence be defined as goal-achieving ability. When I am struggling with the hardest exercises on the Raven test, what I wish I had more is not some goal-achieving power but something far simpler, something akin to short term memory. So when I wish for more intelligene, I just wish for a bit more RAM in short-term memory, so more detailed, more granular ideas can be uploaded into tumble space. No idea why should it mean goal-achieving or optimizing ability. And for AI IQ sounds like entirely hardware...
Intelligence is very useful in conflicts. If you can calculate beforehand whether you will win or lose a battle you don’t have to fight the battle if you think you will lose it.
In our modern world great intelligence means the ability to hack computers. Getting information and being able to alter email message that person A sends person B. That’s power.
Getting money because you are smart enough to predict stock market movements is another way to get power.
Stalin was likely powerful but a lot of today’s political leaders are less powerful.
Peter Thiel made the point that Obama probably didn’t even knew that the US was wiretapping Angela Merkel. It’s something that the nerds in the NSA decided in their Star Trek bridge lookalike.