The one problem I had with Yudkowsky’s TDT paper (which I didn’t read very attentively, mind you, so correct me if I’m wrong) was the part where he staged a dramatic encounter where a one-boxer was pleading with a wistful two-boxing agent who wished he was a one-boxer to change his algorithm to choose just one box. It occurred to me that even if the two-boxer listened to her, then his algorithm would have been altered by totally external factors. For the superintelligence setting up the problem to have predicted his change of mind, he would have had to simulate not a single agent, but the whole two agent system in order to predict this particular scenario correctly. Future settings might be different.
PS. One-boxers tend to make errors like this quite often, actually, illegitimately reducing the whole problem to one algorithm run by a single agent. (Actually a lot of decision theorists do that IMO. I’ve just been introduced to the world of decision theory by one-boxers.)
The one problem I had with Yudkowsky’s TDT paper (which I didn’t read very attentively, mind you, so correct me if I’m wrong) was the part where he staged a dramatic encounter where a one-boxer was pleading with a wistful two-boxing agent who wished he was a one-boxer to change his algorithm to choose just one box. It occurred to me that even if the two-boxer listened to her, then his algorithm would have been altered by totally external factors. For the superintelligence setting up the problem to have predicted his change of mind, he would have had to simulate not a single agent, but the whole two agent system in order to predict this particular scenario correctly. Future settings might be different.
PS. One-boxers tend to make errors like this quite often, actually, illegitimately reducing the whole problem to one algorithm run by a single agent. (Actually a lot of decision theorists do that IMO. I’ve just been introduced to the world of decision theory by one-boxers.)