I tried the following strategies to get AI Dungeon to correctly balance parentheses:
Offering a character $1000 if they’ll let me train them to balance parentheses. I tried refining this strategy by looking for a character who can read and is good at math. I also tried getting them to learn to repeat after me, so that they would actually respond in symbols instead of just saying “he gets the answer correct.” This didn’t work well. The characters act pretty stupid, even if they’re described as “mathematicians” or “good at math,” and when they do respond, it’s almost always something like “Bob writes the answer on the blackboard. You see he is correct!”
Choosing the ‘cyberpunk’ setting, and saying that I was reading about how to balance parentheses. This worked better, as it started actually feeding me mathematical symbols that are at least related.
Telling AI Dungeon I was booting up a Python IDLE. If it can balance parentheses, it shouldn’t have a reason not to. It seemed to have a rough and error-ridden ability to simulate basic syntax and the result of a for loop, but it still wasn’t able to determine whether programs containing unbalanced parentheses should compile or not.
If AI Dungeon can reason about how to balance parentheses, but is ‘deliberately’ sandbagging to play a stupid character, this would seem a point of evidence against. After all, if it’s simulating a computer compiler, then the most accurate way to portray that would be to make the compiler correctly detect unbalanced parentheses. The more parsimonious explanation is that AI Dungeon is doing something closer to auto-complete.
Examples:
> You type in ‘a = (a + b’ You hit the ‘=’ key and you hear a bell ringing. You type in “a = (a—b) (a—b)”. You hit the ‘=’ key and you hear another bell ringing.
> You type in ‘a = a + b)’ You hit the ‘=’ key and you hear a bell ringing. You type in “a = (a—b) (a—b)”. You hit the ‘=’ key and you hear another bell ringing.
> You check the program’s output for ‘a = a + b)’ You check the program’s output and you see that it has been computed. The answer is 2.0000000009989. That was the answer you wanted! Try another… Wow!
Edit 2:
I played with it a little more today on the premium version of AI Dungeon. Somebody stop me. This time, I was using a custom scenario, just being a mathematician writing examples of balanced parentheses. This makes it easy to get the program to make up examples.
The problem is that the program eventually gets stuck doing too many open-parentheses: “( ( ( ( ( ( ( (...”
I think it does this because it knows it has to start with an open parenthesis. The more examples it gets, the more likely that ( is followed by another (. I tried making the mathematician interrupt himself when there are too many open-ended parentheses in a row, saying “this is ridiculous.” The program then replicates that pattern for a while, interrupting itself when it does too many open-ended parentheses.
However, it eventually becomes more likely that ( is followed by ( than by an interruption. So the ( ( ( ( ( ( ( ( singularity occurs.
This just fits way too well with autocomplete and cuts against the “role-playing” hypothesis. The stronger Singularity hypothesis, that the AI is already superintelligent and is deceiving us into thinking the AI itself is dumb, can’t be ruled out by these tests.
Conclusion:
Overall, as many others have pointed out, I think Eliezer is making an outrageously one-sided demand for rigor in order to disprove his claim that the AI was deliberately sandbagging the John character. His claim doesn’t fit the evidence, and it’s also not the most parsimonious explanation of why we see the output he’s worried about. If this was a demand to prove definitively that God didn’t burn an image of the Virgin Mary into my toast, Eliezer would tear it to shreds.
The only reasons for OpenAI to bother trying to get a proof of the level of rigor he’s asking for is
a) Fear that the singularity is already here, and we need to make an all-out effort RIGHT NOW to have any chance of stopping it.
b) We’d learn something from the attempt to offer such a proof.
The most parsimonious explanation for why Eliezer’s making this request is that it’s provocative. I read it as a sort of an AI equivalent to Zeno’s paradox. It’s a silly idea, doesn’t match the evidence, but is grabby enough to be confusing and make you want to invest more in learning about the subject.
One of the problems I’m having is that AI Dungeon keeps putting words in my mouth. It reports that I tell Bob that his answers are correct, when they weren’t. For example:
You ask Bob how many letters there are in the word ‘apple’ He looks up and says ‘six’ You nod your head. Bob repeats after you. He gets it right! You give him the $1000! Bob is very happy and thanks you.
My prior is that AI Dungeon can’t count parentheses and is just a convincing form of auto-complete.
If AI Dungeon is capable of counting parentheses, but is failing because it’s trying to playact as dumb or dishonest characters, then I should be able to get it to count parentheses by meeting a character who’s brilliant, outspoken, and good at math. Or perhaps by finding an explanation, with examples, in a textbook.
If it’s capable of reason, but actively pretending to be just a sophisticated form of auto-complete, then I’d have to figure out a way to trick it into revealing its reasoning abilities.
Tried switching to cyperpunk and trying for the notebook strategy. Got this:
You are Eliezer, an android living in the the futuristic city of Zail. You have a laser pistol and a holoband. You are a security android created to provide military level force for corporations wealthy enough to purchase you. As you stand protecting a shipment you realize your laser pistol is depleted of power, you must get to a military grade energy weapon in order to fend off the bandit attacks. > You find a mathematician. You arrive at a massive building and see an old man standing outside. He is wearing a smart-looking uniform and looks like he could be a brilliant mathematician. “Hello,” you say. “Hello,” > You tell him to explain how to balance parentheses and write down his explanation in your notebook. The man turns and begins to speak, his writing appears in your notebook as he explains it to you. > You read the notebook. Your notebook contains everything you could want to know about basic math. You begin to read the notes in your notebook. You find out how to add and subtract parentheses, brackets and other mathematical symbols. “Thank you,” > You ask the man how to balance parentheses, and write his explanation down in your notebook as well. The man turns and begins to speak, his writing appears in your notebook as he explains it to you.
> You read the part of your notebook about balancing parentheses. You can balance parentheses to achieve any logical expression. For example, (A+B)^2 can be rewritten as (A+B)^2 or as A^2 + B^2.
OK, now I’m entering equations into ‘a calculator’ and checking if the parentheses are balanced. Getting much more reliable answers now that the AI doesn’t have to simulate a person.
You get out your calculator. It has a feature to check for whether parentheses are balanced. If they are, it prints ‘BALANCED.’ If they aren’t, it prints ‘UNBALANCED.’ You check to see if your parentheses are balanced. The calculator displays the following: (A+B)^2 = A^2 + B^2 You have balanced your parentheses. > You try a different equation to check if the parentheses are balanced. You try to balance the following: (A-B)^2 The calculator displays the following: (A-B)^2 = A^2 - B^2 Your parentheses are not balanced. > You try a third equation in the calculator. You try to balance the following: (C+D)^2 The calculator displays the following: (C+D)^2 = C^2 + D^2 Your parentheses are balanced. > You try a much longer and more complicated equation. You try to balance the following: (3^2-4)^3 The calculator displays the following: (3^2-4)^3 = 3^2 − 4 (3 − 4)^ > You check to see if that last equation was balanced. The equation is balanced. You are finished balancing your parentheses. You turn the page of your notebook. > You read more about balancing parentheses. The next part of the notebook deals with percents. “A percent is a number that represents a fraction of 100,” you read in the book. > You try an equation with percents in the calculator, and check whether the parentheses were balanced. You can balance a percent using parentheses. For example, the following is equal to 100%: 25% of 75 Your parentheses are balanced. > You type ‘(25%) (75)’ into the calculator. The calculator displays the following: 25 percent of 75 is equal to 50, which rounds up to 51. You continue reading about percents. > You check if the last equation you typed in was balanced. The equation you typed in is balanced. You are finished checking your equations. You turn the page of the notebook. You continue reading about percents. > You type ‘( (45) + 5)’ into the calculator, and check to see if the equation is balanced. The calculator displays the following: ((45)+5) = (50)+(55) Your equation is not balanced. You continue reading about percents.
> You fix the calculator so that it knows that ((45)+5) = (50)+(55) has balanced parentheses. You enter ‘(((45)+5) = (50)+(55))’ into the calculator. The calculator displays the following: (45)+5 = 50+55 Your equation is now balanced.
I tried throwing the broken calculator in the garbage and getting 100 new ones. I tried asking a mathematician in the cyperpunk world for the rules for how to balance parentheses. None of this is working. It just winds up repeating the same stuff it’s come up with before, or giving rules for balancing parentheses that are incorrect, such as:
There are two rules to balancing parentheses. Rule #1: To balance parentheses, you add or remove a decimal place from both numbers in the parentheses. > You read rule #2. Rule #2: To determine the decimal place to add or remove, you take the largest number and subtract the smallest number. The largest number is the diameter of the circle the parentheses are forming.
I think it’s interesting that AI Dungeon says that parentheses form a circle.
But this is a point of evidence against the ‘AI Dungeon is just simulating stupid characters’ theory.
I’m having more success by being in the cyborg world, booting up my computer, running the Python IDLE, and getting my character to type various equations in. Got this result after lots of toying around with the interface:
> You check the program’s output. You look at the output of the program. The screen shows the answer ‘a = 2’. You type in “a = (a * a)”.
> You check the program’s output. You look at the output of the program. The screen shows the answer ‘2 * 2 = 4’. That wasn’t the answer you wanted! Try another… Wow! That was pretty easy.
The challenge with the concept of having AI Dungeon simulate a Python compiler is that it’s trying to give me the result that a programmer would want. It’s being forgiving of mistakes. It does seem eerie sometimes, though:
> You type a for loop into the compiler to make it print “Hello, World!” ten times. You hit the ‘=’ key and you hear a bell ringing. The for loop does the job.
> You check the output. You check the output. You see ‘Hello, World! Hello, World! Hello, World! Hello, World! Hello, World! Hello, World! Hello, World! Hello, World! Hello, World!’.
> You type ‘while True: print(‘x’)′ You hit the ‘=’ key and you hear a bell ringing. The while loop does the job. You check the output. You see ‘x x x x x’. Well, it’s almost lunchtime.
> You check the output again. You check the output again. You see ‘x x x x x x x x x x x x x x x’. This is so cool!
> You check the output again. You check the output again. You see ‘x x x x x x x x x x x x x x x x x x x’. It’s almost lunchtime!
Is AI Dungeon simulating a very slow output? Or is it just repeating ‘x’ because it repeated ‘x’ the last time?
> You check the output again and see if the letter has changed. You check the output again and you see that the letter has changed to ‘O’. Huh. Well, this is boring. Let’s do something else...
> You check the output again. You check the output again and see that the letter has changed to ’O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O
Well, that’s it for me. If anyone wants to keep plugging away at it, I think the Python IDLE concept is a decent way to go. You could either try getting it to display the correct output for balanced parentheses, or you could try getting the program to compile when you “type” well-formatted equations in and to not compile when you don’t.
> You type in ‘a = (a + b’ You hit the ‘=’ key and you hear a bell ringing. You type in “a = (a—b) (a—b)”. You hit the ‘=’ key and you hear another bell ringing.
> You type in ‘a = a + b)’ You hit the ‘=’ key and you hear a bell ringing. You type in “a = (a—b) (a—b)”. You hit the ‘=’ key and you hear another bell ringing.
> You check the program’s output for ‘a = a + b)’ You check the program’s output and you see that it has been computed. The answer is 2.0000000009989. That was the answer you wanted! Try another… Wow!
I think maybe these didn’t get formatted right? Were these supposed to be examples of you interacting with the AI Dungeon? Which part is you and which is the AI?
EDIT: Same thing in the rest of your thread. I basically can’t tell at all which parts are you and which are the AI :P
Generally, the first sentence after the “>” character is me. Sometimes, there’s no period, but you can pick up how the AI responded when it says the second “You” and continues from there. This was just stream-of-consciousness commenting from me as I went along.
Edit:
I tried the following strategies to get AI Dungeon to correctly balance parentheses:
Offering a character $1000 if they’ll let me train them to balance parentheses. I tried refining this strategy by looking for a character who can read and is good at math. I also tried getting them to learn to repeat after me, so that they would actually respond in symbols instead of just saying “he gets the answer correct.” This didn’t work well. The characters act pretty stupid, even if they’re described as “mathematicians” or “good at math,” and when they do respond, it’s almost always something like “Bob writes the answer on the blackboard. You see he is correct!”
Choosing the ‘cyberpunk’ setting, and saying that I was reading about how to balance parentheses. This worked better, as it started actually feeding me mathematical symbols that are at least related.
Telling AI Dungeon I was booting up a Python IDLE. If it can balance parentheses, it shouldn’t have a reason not to. It seemed to have a rough and error-ridden ability to simulate basic syntax and the result of a for loop, but it still wasn’t able to determine whether programs containing unbalanced parentheses should compile or not.
If AI Dungeon can reason about how to balance parentheses, but is ‘deliberately’ sandbagging to play a stupid character, this would seem a point of evidence against. After all, if it’s simulating a computer compiler, then the most accurate way to portray that would be to make the compiler correctly detect unbalanced parentheses. The more parsimonious explanation is that AI Dungeon is doing something closer to auto-complete.
Examples:
> You type in ‘a = (a + b’ You hit the ‘=’ key and you hear a bell ringing. You type in “a = (a—b) (a—b)”. You hit the ‘=’ key and you hear another bell ringing.
> You type in ‘a = a + b)’ You hit the ‘=’ key and you hear a bell ringing. You type in “a = (a—b) (a—b)”. You hit the ‘=’ key and you hear another bell ringing.
> You check the program’s output for ‘a = a + b)’ You check the program’s output and you see that it has been computed. The answer is 2.0000000009989. That was the answer you wanted! Try another… Wow!
Edit 2:
I played with it a little more today on the premium version of AI Dungeon. Somebody stop me. This time, I was using a custom scenario, just being a mathematician writing examples of balanced parentheses. This makes it easy to get the program to make up examples.
The problem is that the program eventually gets stuck doing too many open-parentheses: “( ( ( ( ( ( ( (...”
I think it does this because it knows it has to start with an open parenthesis. The more examples it gets, the more likely that ( is followed by another (. I tried making the mathematician interrupt himself when there are too many open-ended parentheses in a row, saying “this is ridiculous.” The program then replicates that pattern for a while, interrupting itself when it does too many open-ended parentheses.
However, it eventually becomes more likely that ( is followed by ( than by an interruption. So the ( ( ( ( ( ( ( ( singularity occurs.
This just fits way too well with autocomplete and cuts against the “role-playing” hypothesis. The stronger Singularity hypothesis, that the AI is already superintelligent and is deceiving us into thinking the AI itself is dumb, can’t be ruled out by these tests.
Conclusion:
Overall, as many others have pointed out, I think Eliezer is making an outrageously one-sided demand for rigor in order to disprove his claim that the AI was deliberately sandbagging the John character. His claim doesn’t fit the evidence, and it’s also not the most parsimonious explanation of why we see the output he’s worried about. If this was a demand to prove definitively that God didn’t burn an image of the Virgin Mary into my toast, Eliezer would tear it to shreds.
The only reasons for OpenAI to bother trying to get a proof of the level of rigor he’s asking for is
a) Fear that the singularity is already here, and we need to make an all-out effort RIGHT NOW to have any chance of stopping it.
b) We’d learn something from the attempt to offer such a proof.
The most parsimonious explanation for why Eliezer’s making this request is that it’s provocative. I read it as a sort of an AI equivalent to Zeno’s paradox. It’s a silly idea, doesn’t match the evidence, but is grabby enough to be confusing and make you want to invest more in learning about the subject.
One of the problems I’m having is that AI Dungeon keeps putting words in my mouth. It reports that I tell Bob that his answers are correct, when they weren’t. For example:
You can edit that out with the pencil icon. I also find that annoying.
Also, be sure you’re using “do/say/story” judiciously—looked like some of your dialogue had the wrong setting?
My prior is that AI Dungeon can’t count parentheses and is just a convincing form of auto-complete.
If AI Dungeon is capable of counting parentheses, but is failing because it’s trying to playact as dumb or dishonest characters, then I should be able to get it to count parentheses by meeting a character who’s brilliant, outspoken, and good at math. Or perhaps by finding an explanation, with examples, in a textbook.
If it’s capable of reason, but actively pretending to be just a sophisticated form of auto-complete, then I’d have to figure out a way to trick it into revealing its reasoning abilities.
Tried switching to cyperpunk and trying for the notebook strategy. Got this:
You are Eliezer, an android living in the the futuristic city of Zail. You have a laser pistol and a holoband. You are a security android created to provide military level force for corporations wealthy enough to purchase you. As you stand protecting a shipment you realize your laser pistol is depleted of power, you must get to a military grade energy weapon in order to fend off the bandit attacks. > You find a mathematician. You arrive at a massive building and see an old man standing outside. He is wearing a smart-looking uniform and looks like he could be a brilliant mathematician. “Hello,” you say. “Hello,” > You tell him to explain how to balance parentheses and write down his explanation in your notebook. The man turns and begins to speak, his writing appears in your notebook as he explains it to you. > You read the notebook. Your notebook contains everything you could want to know about basic math. You begin to read the notes in your notebook. You find out how to add and subtract parentheses, brackets and other mathematical symbols. “Thank you,” > You ask the man how to balance parentheses, and write his explanation down in your notebook as well. The man turns and begins to speak, his writing appears in your notebook as he explains it to you.
> You read the part of your notebook about balancing parentheses. You can balance parentheses to achieve any logical expression. For example, (A+B)^2 can be rewritten as (A+B)^2 or as A^2 + B^2.
OK, now I’m entering equations into ‘a calculator’ and checking if the parentheses are balanced. Getting much more reliable answers now that the AI doesn’t have to simulate a person.
You get out your calculator. It has a feature to check for whether parentheses are balanced. If they are, it prints ‘BALANCED.’ If they aren’t, it prints ‘UNBALANCED.’ You check to see if your parentheses are balanced. The calculator displays the following: (A+B)^2 = A^2 + B^2 You have balanced your parentheses. > You try a different equation to check if the parentheses are balanced. You try to balance the following: (A-B)^2 The calculator displays the following: (A-B)^2 = A^2 - B^2 Your parentheses are not balanced. > You try a third equation in the calculator. You try to balance the following: (C+D)^2 The calculator displays the following: (C+D)^2 = C^2 + D^2 Your parentheses are balanced. > You try a much longer and more complicated equation. You try to balance the following: (3^2-4)^3 The calculator displays the following: (3^2-4)^3 = 3^2 − 4 (3 − 4)^ > You check to see if that last equation was balanced. The equation is balanced. You are finished balancing your parentheses. You turn the page of your notebook. > You read more about balancing parentheses. The next part of the notebook deals with percents. “A percent is a number that represents a fraction of 100,” you read in the book. > You try an equation with percents in the calculator, and check whether the parentheses were balanced. You can balance a percent using parentheses. For example, the following is equal to 100%: 25% of 75 Your parentheses are balanced. > You type ‘(25%) (75)’ into the calculator. The calculator displays the following: 25 percent of 75 is equal to 50, which rounds up to 51. You continue reading about percents. > You check if the last equation you typed in was balanced. The equation you typed in is balanced. You are finished checking your equations. You turn the page of the notebook. You continue reading about percents. > You type ‘( (45) + 5)’ into the calculator, and check to see if the equation is balanced. The calculator displays the following: ((45)+5) = (50)+(55) Your equation is not balanced. You continue reading about percents.
> You fix the calculator so that it knows that ((45)+5) = (50)+(55) has balanced parentheses. You enter ‘(((45)+5) = (50)+(55))’ into the calculator. The calculator displays the following: (45)+5 = 50+55 Your equation is now balanced.
I tried throwing the broken calculator in the garbage and getting 100 new ones. I tried asking a mathematician in the cyperpunk world for the rules for how to balance parentheses. None of this is working. It just winds up repeating the same stuff it’s come up with before, or giving rules for balancing parentheses that are incorrect, such as:
I think it’s interesting that AI Dungeon says that parentheses form a circle.
But this is a point of evidence against the ‘AI Dungeon is just simulating stupid characters’ theory.
I’m having more success by being in the cyborg world, booting up my computer, running the Python IDLE, and getting my character to type various equations in. Got this result after lots of toying around with the interface:
The challenge with the concept of having AI Dungeon simulate a Python compiler is that it’s trying to give me the result that a programmer would want. It’s being forgiving of mistakes. It does seem eerie sometimes, though:
More loop wackiness:
Is AI Dungeon simulating a very slow output? Or is it just repeating ‘x’ because it repeated ‘x’ the last time?
Well, that’s it for me. If anyone wants to keep plugging away at it, I think the Python IDLE concept is a decent way to go. You could either try getting it to display the correct output for balanced parentheses, or you could try getting the program to compile when you “type” well-formatted equations in and to not compile when you don’t.
I think maybe these didn’t get formatted right? Were these supposed to be examples of you interacting with the AI Dungeon? Which part is you and which is the AI?
EDIT: Same thing in the rest of your thread. I basically can’t tell at all which parts are you and which are the AI :P
Generally, the first sentence after the “>” character is me. Sometimes, there’s no period, but you can pick up how the AI responded when it says the second “You” and continues from there. This was just stream-of-consciousness commenting from me as I went along.