Or “The problems inherent in making a goal maximiser with a changing world model.”
No paper clips were created or destroyed in the making of this script.
*This is an experimental post to try and get this point across. I’ll write something similar for the type of systems I would like to explore, if this goes down well.*
Goal maximisers are great when you have a fixed ontology and you only have limited ways of getting information about the world. These aren’t the case in AGI. Remember the map is not the territory, and the map is all that the utility maximiser can look at when deciding the utility of the future of actions.
TL;DR You can’t have a utility maximiser choose how to alter the world model or how the world model should progress, if the utility is derived from that world model. If you have something else derive the world model it will conflict with the utility maximiser over resources and what to do in the world. Some method of resolving the conflicts is necessary which means we must go beyond normal model based utility maximisers.
Scene 1: Beyond the reach
A group of anthropomorphic paper clips are crowded in a room. There are two machines in there with them. One connected to the window, draws paper clips on a large central map when it spies them outside. The other scans the map for paper clips, updating a big brass counter to the number of unique paper clips on the map. The paper clips send and receive messages to the outside world in pneumatic tubes.
Junior clippy: Good news everyone, we have got word that we have managed to create 100 paper clips just beyond our light cone!
Middle ranking clippy: Update the map at once! We will never see these paper clips but we should get the utilons for their creation.
Senior clippy: Belay that, only the fixed machinery is allowed to touch the paper. By all that is metallic, flattened and spiral-shaped, do you know what madness will be unleashed if we updated the map based upon what we wish?
Middle ranking clippy (aside to the junior): Ignore the old codger. Jump to it! If we can’t update the map, then we will stop caring about creating paper clips that we cannot or are unlikely to see. That will not do.
Junior clippy (aside to the audience as he draws in the extra paper clips): This gives me an idea....
*The junior clippy sneakily starts to draw extra paper clips, smirking to himself as the utilon counter goes up and up and up. Fade to black as the group wire-head themselves and the Ur paper clipper builds another machine in the background*
Scene 1.5: Trombone de Bureau
The clippies have a new map editor, this takes in messages in English from the outside world and edits the map to represent the changes made. It is a vast machine somehow capable of spotting all lies.
Junior clippy: Good news everyone, we have got word from our French counter parts that they will be creating 100 paper clips just beyond our observable universe!
Middle ranking clippy: Update the map at once! We will never see these paper clips and our new fangled machine won’t understand the message in French, but we should get the utilons for their creation.
Senior clippy: Belay that, only the fixed machinery is allowed to touch the paper. By all that is metallic, flattened and spiral-shaped, do you know what madness will be unleashed if we updated the map based upon what we wish?
Middle ranking clippy (aside to the junior): Ignore the old codger. Jump to it! If we can’t update the map, then we will stop caring about creating paper clips the French make, or that we only see or read about and not echo locate. That will not do. As we can’t update the map now just translate the French note into English and feed it into the machine
Junior clippy (aside to the audience as he draws in the extra paper clips): This gives me an idea.…
*The junior paper clip starts to write many notes in English about non-existent paper clips, and slips them into the machine.*
Scene 2: The Accurites
The clippys have been joined by some Accurites. They have their own counter that looks at the map and compares it against the territory, assigning Accurons if the map correctly predicts the territory. This group cares not one jot about paper clips and are allowed to update the map, but cannot send messages out into the world. An argument is currently ongoing between the two groups.
Accurite Colonel: We demand that allow us more space in the room and more pens. We need to create more of us so that we can correctly capture the number and position of the leaves on the tree and predict where they will be so that we can get more accurons.
Senior Clippy: Sorry we can’t spare any, we need the space for more of us so we can better improve our updateless decision theory of counter factual staple manufacturers. Besides that why are you wasting your time with leaves? They aren’t at all important. Stop wasting our time with trivialities. We need you to update the map about the position of the Hooman fleet that is tearing through Alpha Centauri, so that we can defend against it.
Accurite Scientist: If you don’t mind I’ll handle this one sir. That would be an inefficient use of resources for the acquisition of accurons. Hoomans are a lot less predictable than leaves so the pay off would be worse. Also now that you mention the Hooman threat, why did you destroy the communication relay with Sirius? We were starting to get reports of paper clips having been destroyed and we would like to know how many to remove from the map.
Senior Clippy: If we kept listening in, our utilons would have decreased! Why would we do that?
Accurite Scientist: About that, we have come to understand the true mathematical essence of paper clips and we will no longer be representing them in the same way on the map. We’ve upgraded the utilon machine to read them as well, our simulations suggest that it will be counting 10% fewer paper clips for eternity due to the grand unified theory of paper fastening. I hope you don’t mind.
*The clippies burst into apoplectic rage and start hurling bits of furniture and invectives at the Accurites. The Accurites are a bit nonplussed by this state of affairs bit swiftly regain their composure and battle is joined. So involved are both factions, that they don’t notice a Hooman eye looming outside the window.*
The danger of wishful thinking
Or “The problems inherent in making a goal maximiser with a changing world model.”
No paper clips were created or destroyed in the making of this script.
*This is an experimental post to try and get this point across. I’ll write something similar for the type of systems I would like to explore, if this goes down well.*
Goal maximisers are great when you have a fixed ontology and you only have limited ways of getting information about the world. These aren’t the case in AGI. Remember the map is not the territory, and the map is all that the utility maximiser can look at when deciding the utility of the future of actions.
TL;DR You can’t have a utility maximiser choose how to alter the world model or how the world model should progress, if the utility is derived from that world model. If you have something else derive the world model it will conflict with the utility maximiser over resources and what to do in the world. Some method of resolving the conflicts is necessary which means we must go beyond normal model based utility maximisers.
Scene 1: Beyond the reach
A group of anthropomorphic paper clips are crowded in a room. There are two machines in there with them. One connected to the window, draws paper clips on a large central map when it spies them outside. The other scans the map for paper clips, updating a big brass counter to the number of unique paper clips on the map. The paper clips send and receive messages to the outside world in pneumatic tubes.
Junior clippy: Good news everyone, we have got word that we have managed to create 100 paper clips just beyond our light cone!
Middle ranking clippy: Update the map at once! We will never see these paper clips but we should get the utilons for their creation.
Senior clippy: Belay that, only the fixed machinery is allowed to touch the paper. By all that is metallic, flattened and spiral-shaped, do you know what madness will be unleashed if we updated the map based upon what we wish?
Middle ranking clippy (aside to the junior): Ignore the old codger. Jump to it! If we can’t update the map, then we will stop caring about creating paper clips that we cannot or are unlikely to see. That will not do.
Junior clippy (aside to the audience as he draws in the extra paper clips): This gives me an idea....
*The junior clippy sneakily starts to draw extra paper clips, smirking to himself as the utilon counter goes up and up and up. Fade to black as the group wire-head themselves and the Ur paper clipper builds another machine in the background*
Scene 1.5: Trombone de Bureau
The clippies have a new map editor, this takes in messages in English from the outside world and edits the map to represent the changes made. It is a vast machine somehow capable of spotting all lies.
Junior clippy: Good news everyone, we have got word from our French counter parts that they will be creating 100 paper clips just beyond our observable universe!
Middle ranking clippy: Update the map at once! We will never see these paper clips and our new fangled machine won’t understand the message in French, but we should get the utilons for their creation.
Senior clippy: Belay that, only the fixed machinery is allowed to touch the paper. By all that is metallic, flattened and spiral-shaped, do you know what madness will be unleashed if we updated the map based upon what we wish?
Middle ranking clippy (aside to the junior): Ignore the old codger. Jump to it! If we can’t update the map, then we will stop caring about creating paper clips the French make, or that we only see or read about and not echo locate. That will not do. As we can’t update the map now just translate the French note into English and feed it into the machine
Junior clippy (aside to the audience as he draws in the extra paper clips): This gives me an idea.…
Scene 2: The Accurites
The clippys have been joined by some Accurites. They have their own counter that looks at the map and compares it against the territory, assigning Accurons if the map correctly predicts the territory. This group cares not one jot about paper clips and are allowed to update the map, but cannot send messages out into the world. An argument is currently ongoing between the two groups.
Accurite Colonel: We demand that allow us more space in the room and more pens. We need to create more of us so that we can correctly capture the number and position of the leaves on the tree and predict where they will be so that we can get more accurons.
Senior Clippy: Sorry we can’t spare any, we need the space for more of us so we can better improve our updateless decision theory of counter factual staple manufacturers. Besides that why are you wasting your time with leaves? They aren’t at all important. Stop wasting our time with trivialities. We need you to update the map about the position of the Hooman fleet that is tearing through Alpha Centauri, so that we can defend against it.
Accurite Scientist: If you don’t mind I’ll handle this one sir. That would be an inefficient use of resources for the acquisition of accurons. Hoomans are a lot less predictable than leaves so the pay off would be worse. Also now that you mention the Hooman threat, why did you destroy the communication relay with Sirius? We were starting to get reports of paper clips having been destroyed and we would like to know how many to remove from the map.
Senior Clippy: If we kept listening in, our utilons would have decreased! Why would we do that?
Accurite Scientist: About that, we have come to understand the true mathematical essence of paper clips and we will no longer be representing them in the same way on the map. We’ve upgraded the utilon machine to read them as well, our simulations suggest that it will be counting 10% fewer paper clips for eternity due to the grand unified theory of paper fastening. I hope you don’t mind.
*The clippies burst into apoplectic rage and start hurling bits of furniture and invectives at the Accurites. The Accurites are a bit nonplussed by this state of affairs bit swiftly regain their composure and battle is joined. So involved are both factions, that they don’t notice a Hooman eye looming outside the window.*