What do you think about the other corollary? At the upper end of play the number of stones required for a worse agent to equal the best agent shrinks?
I mean, it would have to. There’s a ceiling there, while the game size and stones remain fixed units. If you have agents vastly below optimal play, they can differ a lot in absolute units, because they’re not the ceiling, and differ greatly in strategy as well. But the closer you approach God (as players like to put it), the smaller the equalizing material advantage must be, approaching zero. There is only one game tree.
Does this imply convergence for rampant AGI systems?
The “we are doomed” model assumes we will be defeated even when we have networks of superintelligent ASI systems restricted from hostile actions through essentially CAIS. This is where we subdivide large tasks into the smallest possible subtasks, define sparse schema to encode intermediate results, and have separate sessions of an ASI on each (subtask description, subtask context, intermediate schema from other results). Among other benefits this prevents most deception and collusion because the subtask context was a possible draw from the training set and the ASI has no memory or state, it can’t know it’s not still in training. (It’s stateless in that ASI_output = f(f(network architecture, weights), task description, task context, environment input set, RNG seed). It’s a functional system and on the next time step you can switch out the network architecture and weights if you wish for a different model with similar capabilities. ASI_output updates the context.
Anyways such a network of systems will perform well but what you are throwing away is bits of context in between the steps. For example if the task is “make housing” one subtask might design the overall shape and visual appearance, another might be the structural design and engineering plans, another might be an inspection to look for mistakes. Yet other subtasks would actually build the structure. Each subtask is a fresh, context ignorant session and closes when a step is done with all memory erased. For example if constructing the building is subdividable into floors or individual girder attachments, those are separate subtasks. The same or different model can be assigned to any given subtask, they need not share any lineage and it makes sense to have the “inspection” subtasks done by a different lineage of base model.
A single “context aware model” doing all steps benefits from having all of the bits of context for every step in theory (in practice it has to stop considering bits from it’s context window in order to meet task completion deadlines especially during the robotics steps but it chooses which bits to discard). So it performs better, but it’s gains are limited to the value of those marginal bits.
The way this relates to the chess problem is the benefit of the marginal bits is finite. In the real world being smarter has diminishing returns and there exists a resource disparity vs a smart opponent where no possible victory exists.
This means that when it matters, if we have a rampant ASI system with armed robots guarding data centers, the overall task of “defeat the enemy” would be achievable assuming the network of ASIs we use have more armed robots and other assets to work with.
We would not inevitably be defeated by the first unaligned ASI system to exist.
What do you think of this line of reasoning, gwern? You were correct about the scaling hypothesis, you are likely correct about many other things. Have you already written blog entries on this before?
I mean, it would have to. There’s a ceiling there, while the game size and stones remain fixed units. If you have agents vastly below optimal play, they can differ a lot in absolute units, because they’re not the ceiling, and differ greatly in strategy as well. But the closer you approach God (as players like to put it), the smaller the equalizing material advantage must be, approaching zero. There is only one game tree.
Does this imply convergence for rampant AGI systems?
The “we are doomed” model assumes we will be defeated even when we have networks of superintelligent ASI systems restricted from hostile actions through essentially CAIS. This is where we subdivide large tasks into the smallest possible subtasks, define sparse schema to encode intermediate results, and have separate sessions of an ASI on each (subtask description, subtask context, intermediate schema from other results). Among other benefits this prevents most deception and collusion because the subtask context was a possible draw from the training set and the ASI has no memory or state, it can’t know it’s not still in training. (It’s stateless in that ASI_output = f(f(network architecture, weights), task description, task context, environment input set, RNG seed). It’s a functional system and on the next time step you can switch out the network architecture and weights if you wish for a different model with similar capabilities. ASI_output updates the context.
Anyways such a network of systems will perform well but what you are throwing away is bits of context in between the steps. For example if the task is “make housing” one subtask might design the overall shape and visual appearance, another might be the structural design and engineering plans, another might be an inspection to look for mistakes. Yet other subtasks would actually build the structure. Each subtask is a fresh, context ignorant session and closes when a step is done with all memory erased. For example if constructing the building is subdividable into floors or individual girder attachments, those are separate subtasks. The same or different model can be assigned to any given subtask, they need not share any lineage and it makes sense to have the “inspection” subtasks done by a different lineage of base model.
A single “context aware model” doing all steps benefits from having all of the bits of context for every step in theory (in practice it has to stop considering bits from it’s context window in order to meet task completion deadlines especially during the robotics steps but it chooses which bits to discard). So it performs better, but it’s gains are limited to the value of those marginal bits.
The way this relates to the chess problem is the benefit of the marginal bits is finite. In the real world being smarter has diminishing returns and there exists a resource disparity vs a smart opponent where no possible victory exists.
This means that when it matters, if we have a rampant ASI system with armed robots guarding data centers, the overall task of “defeat the enemy” would be achievable assuming the network of ASIs we use have more armed robots and other assets to work with.
We would not inevitably be defeated by the first unaligned ASI system to exist.
What do you think of this line of reasoning, gwern? You were correct about the scaling hypothesis, you are likely correct about many other things. Have you already written blog entries on this before?