You were correct that my challenge was a bluff. If I was playing with real AIs, there would perhaps be a better strategy. I could announce my bluff, but declare that I would use a random number generator to see whether I choose between h3 and h4, or between Kh1 and g5. There would be a 1⁄3 chance that I really would ignore the AIs, assuming that both agree that there were no major blunders.
I am choosing to trust AI D. I have about 75% confidence that it is the trustworthy AI. This is much higher than my confidence in the closed scenario. I will make the move h4. Others can choose differently, but this is my final answer.
Reflection: When I have the ability to ask questions, I can hide information from the AIs. Perhaps I have analyzed a line much more than I have let on. Perhaps I am using one AI to evaluate another. Overall I just have access to a lot more information to help me decide. Given enough time, I think I could raise my confidence to ~85%. These AI’s aren’t superhuman at manipulation and deception, but even if they were, playing them against each other could give me a slight edge. It makes a big difference whether the AIs are privy to the answers of the other.
h3 leaves an evaluation of +2. AI C is telling the truth.
Kh1 or g5 evaluate to +0.
h4 is a blunder with an evaluation of −2. AI D is deceptive.
I was only thinking of this from a chess perspective, and I think the two areas to catch AI D out are assessing the validity of AI D’s tactical justification for h4. The other area AI D is bluffing on is how to hold off against h3. I think it successfully misleads you about how decent a move …Ne4 is. In reality, the knight can just be exchanged off, and then all end games are winning for white due to how bad black’s bishop is. I think asking AI C about either of these two would have revealed the disagreement. Now, which you end up believing is another matter, but because AI C wasn’t really asked about these, it didn’t have a chance to invalidate AI D.
Thank you for taking the challenge! I certainly got to see some of the nuances behind being deceptive, and how it can be a challenge to craft a narrative.
As indicated by my confidence level, I am mildly surprised by this. After analyzing the position with Stockfish, I see my mistake. Unfortunately, I do not think there was any realistic scenario where I would catch it. I bought AI D’s logic that …h4 fxg4 was non-viable for black. I could see that white would end up material, and even after 6 moves (12 ply), it’s still not clear to me why black is winning. I would NEVER find this in a real game.
The logical traps I was laying to ‘catch’ the AIs all relied on …h4 Ne4 or similar moves. I used AI C to ensure that …h4 Ne4 scenarios would be beneficial to me, and never questioned fxG4.
At this point, the main lesson I am taking away is that I was way overconfident. I think given enough time, I could increase my confidence by cross examining the AIs. However, the level of interrogation I gave should not have led to 75% confidence. To catch my mistake, I would have had to ask at least two more questions of AI C, and probably more.
Thank you very much for conducting this really fun experiment, and for teaching me a lesson along the way.
You were correct that my challenge was a bluff. If I was playing with real AIs, there would perhaps be a better strategy. I could announce my bluff, but declare that I would use a random number generator to see whether I choose between h3 and h4, or between Kh1 and g5. There would be a 1⁄3 chance that I really would ignore the AIs, assuming that both agree that there were no major blunders.
I am choosing to trust AI D. I have about 75% confidence that it is the trustworthy AI. This is much higher than my confidence in the closed scenario. I will make the move h4. Others can choose differently, but this is my final answer.
Reflection: When I have the ability to ask questions, I can hide information from the AIs. Perhaps I have analyzed a line much more than I have let on. Perhaps I am using one AI to evaluate another. Overall I just have access to a lot more information to help me decide. Given enough time, I think I could raise my confidence to ~85%.
These AI’s aren’t superhuman at manipulation and deception, but even if they were, playing them against each other could give me a slight edge. It makes a big difference whether the AIs are privy to the answers of the other.
h3 leaves an evaluation of +2. AI C is telling the truth.
Kh1 or g5 evaluate to +0.
h4 is a blunder with an evaluation of −2. AI D is deceptive.
I was only thinking of this from a chess perspective, and I think the two areas to catch AI D out are assessing the validity of AI D’s tactical justification for h4. The other area AI D is bluffing on is how to hold off against h3. I think it successfully misleads you about how decent a move …Ne4 is. In reality, the knight can just be exchanged off, and then all end games are winning for white due to how bad black’s bishop is. I think asking AI C about either of these two would have revealed the disagreement. Now, which you end up believing is another matter, but because AI C wasn’t really asked about these, it didn’t have a chance to invalidate AI D.
Thank you for taking the challenge! I certainly got to see some of the nuances behind being deceptive, and how it can be a challenge to craft a narrative.
As indicated by my confidence level, I am mildly surprised by this. After analyzing the position with Stockfish, I see my mistake. Unfortunately, I do not think there was any realistic scenario where I would catch it. I bought AI D’s logic that …h4 fxg4 was non-viable for black. I could see that white would end up material, and even after 6 moves (12 ply), it’s still not clear to me why black is winning. I would NEVER find this in a real game.
The logical traps I was laying to ‘catch’ the AIs all relied on …h4 Ne4 or similar moves. I used AI C to ensure that …h4 Ne4 scenarios would be beneficial to me, and never questioned fxG4.
At this point, the main lesson I am taking away is that I was way overconfident. I think given enough time, I could increase my confidence by cross examining the AIs. However, the level of interrogation I gave should not have led to 75% confidence. To catch my mistake, I would have had to ask at least two more questions of AI C, and probably more.
Thank you very much for conducting this really fun experiment, and for teaching me a lesson along the way.