That makes sense to me but to make any argument about the “general game of life” seems very hard. Actions in the real world are made under great uncertainty and aggregate in a smooth way. Acting in the world is trying to control (what physicists call) chaos.
In such a situation, great uncertainty means that an intelligence advantage only matters “on average over a very long time”. It might not matter for a given limited contest, such as a struggle for world domination. For example, you might be much smarter than me and a meteorologist, but you’d find it hard to predict the weather in a year’s time better than me if it’s a single-shot-contest. How much “smarter” would you need to be in order to have a big advantage? Pretty much regardless of your computational ability and knowledge of physics, you’d need such an amount of absurdly precise knowledge about the world that it might still take (both you and even much less intelligent actors) less resources to actively control the entire planet’s weather than predict it a year in advance.
The way that states of the world are influenced by our actions is usually in some sense smooth. For any optimal action, there are usually lots of similar “nearby actions”. These may or may not be near-optimal but in practice only plans that have a sufficiently high margin for error are feasible. The margin of error depends on the resources that allow finely controlled actions and thus increase the space of feasible plans. This doesn’t have a good analogy in chess: chess is much further from smooth than most games in the real world.
Maybe RTS games are a slightly better analogy. They have “some smoothness of action-result mapping” and high amounts of uncertainty. Based on AlphaStar’s success in StarCraft, I would expect we can currently build super-human AIs for such games. They are superior to humans both in their ability to quickly and precisely perform many actions, as well as find better strategies. An interesting restriction is to limit the numbers of actions the AI may take to below what a human can to see the effect of these abilities individually. Restricting the precision and frequency of actions reduces the space of viable plans, at which point the intelligence advantage might matter much less.
All in all, what I’m trying to say is that the question “how much does what intelligence imbalance matter in the world” is hard. The question is not independent of access to information and access to resources or ability to act on the world. To make use of a very high intelligence, you might need a lot more information and also a lot more ability to take precise actions. The question for some system “taking over” is whether its initial intelligence, information and ability to take actions is sufficient to bootstrap quickly enough.
These are just some more reasons you can’t predict the result just by saying “something much smarter is unbeatable at any sufficiently complex game”.
Maybe an analogy which seems closer to the “real world” situation—let’s say you and someone like Sam Altman both tried to start new companies. How much more time and starting capital do you think you’d need to have a better shot of success than him?
I don’t quite see what you’re trying to tell me. That one (which?) of my two analogies (weather or RTS) is bad? That you agree or disagree with my main claim that “evaluating the relative value of an intelligence advantage is probably hard in real life”?
Your analogy doesn’t really speak to me because I’ve never tried to start a company and have no idea what leads to success, or what resources/time/information/intelligence helps how much.
For example, you might be much smarter than me and a meteorologist, but you’d find it hard to predict the weather in a year’s time better than me if it’s a single-shot-contest.
Sure, but I’d presumably be quite a lot better at predicting the weather in two days time.
What point are you trying to make? I’m not sure how that relates to what I was trying to illustrate with the weather example. Assuming for the moment that you didn’t understand my point.
The “game” I was referring to was one where it’s literally all-or-nothing “predict the weather a year from now”, you get no extra points for tomorrow’s weather. This might be artificial but I chose it because it’s a common example of the interesting fact that chaos can be easier to control than simulate.
Another example. You’re trying to win an election and “plan long-term to make the best use of your intelligence advantage”, you need to plan and predict a year ahead. Intelligence doesn’t give you a big advantage in predicting tomorrow’s polls given today’s polls. I can do that reasonably well, too. In this contest, resources and information might matter a lot more than intelligence. Of course, you can use intelligence to obtain information and resources. But this bootstrapping takes time and it’s hard to tell how much depending where you start off.
That makes sense to me but to make any argument about the “general game of life” seems very hard. Actions in the real world are made under great uncertainty and aggregate in a smooth way. Acting in the world is trying to control (what physicists call) chaos.
In such a situation, great uncertainty means that an intelligence advantage only matters “on average over a very long time”. It might not matter for a given limited contest, such as a struggle for world domination. For example, you might be much smarter than me and a meteorologist, but you’d find it hard to predict the weather in a year’s time better than me if it’s a single-shot-contest. How much “smarter” would you need to be in order to have a big advantage? Pretty much regardless of your computational ability and knowledge of physics, you’d need such an amount of absurdly precise knowledge about the world that it might still take (both you and even much less intelligent actors) less resources to actively control the entire planet’s weather than predict it a year in advance.
The way that states of the world are influenced by our actions is usually in some sense smooth. For any optimal action, there are usually lots of similar “nearby actions”. These may or may not be near-optimal but in practice only plans that have a sufficiently high margin for error are feasible. The margin of error depends on the resources that allow finely controlled actions and thus increase the space of feasible plans. This doesn’t have a good analogy in chess: chess is much further from smooth than most games in the real world.
Maybe RTS games are a slightly better analogy. They have “some smoothness of action-result mapping” and high amounts of uncertainty. Based on AlphaStar’s success in StarCraft, I would expect we can currently build super-human AIs for such games. They are superior to humans both in their ability to quickly and precisely perform many actions, as well as find better strategies. An interesting restriction is to limit the numbers of actions the AI may take to below what a human can to see the effect of these abilities individually. Restricting the precision and frequency of actions reduces the space of viable plans, at which point the intelligence advantage might matter much less.
All in all, what I’m trying to say is that the question “how much does what intelligence imbalance matter in the world” is hard. The question is not independent of access to information and access to resources or ability to act on the world. To make use of a very high intelligence, you might need a lot more information and also a lot more ability to take precise actions. The question for some system “taking over” is whether its initial intelligence, information and ability to take actions is sufficient to bootstrap quickly enough.
These are just some more reasons you can’t predict the result just by saying “something much smarter is unbeatable at any sufficiently complex game”.
Maybe an analogy which seems closer to the “real world” situation—let’s say you and someone like Sam Altman both tried to start new companies. How much more time and starting capital do you think you’d need to have a better shot of success than him?
I really have no idea, probably a lot?
I don’t quite see what you’re trying to tell me. That one (which?) of my two analogies (weather or RTS) is bad? That you agree or disagree with my main claim that “evaluating the relative value of an intelligence advantage is probably hard in real life”?
Your analogy doesn’t really speak to me because I’ve never tried to start a company and have no idea what leads to success, or what resources/time/information/intelligence helps how much.
Sure, but I’d presumably be quite a lot better at predicting the weather in two days time.
What point are you trying to make? I’m not sure how that relates to what I was trying to illustrate with the weather example. Assuming for the moment that you didn’t understand my point.
The “game” I was referring to was one where it’s literally all-or-nothing “predict the weather a year from now”, you get no extra points for tomorrow’s weather. This might be artificial but I chose it because it’s a common example of the interesting fact that chaos can be easier to control than simulate.
Another example. You’re trying to win an election and “plan long-term to make the best use of your intelligence advantage”, you need to plan and predict a year ahead. Intelligence doesn’t give you a big advantage in predicting tomorrow’s polls given today’s polls. I can do that reasonably well, too. In this contest, resources and information might matter a lot more than intelligence. Of course, you can use intelligence to obtain information and resources. But this bootstrapping takes time and it’s hard to tell how much depending where you start off.