A fun lesson on the scientific method: A few days ago, OpenAI announced a rollout of “Advanced Voice Mode”. Confused by the messaging, I ran a test to see if their phone app could respond to an interruption:
Nisan: Can you count to 10, and then I’ll interrupt you in the middle and see how that works.
[Nisan interrupts at some point, I forgot exactly where.]
ChatGPT 4o: —sixteen, seventeen, eighteen, nineteen, twenty, twenty-one, twenty-two, twenty-three, twenty-four, twenty-five, twenty-six, twenty-seven, twenty-eight, twenty-nine, thirty. Looks like I made it to the end! Did you mean to interrupt somewhere?
Nisan: Yeah, I did. It must not have worked. Let’s try that again. [Generates a random number.] Please count up to 30.
I tried to replicate. At 20 it went on to 25, and I explained what it got wrong. I tried again. I interrupted at 6 and it stopped at 7, saying “Gotcha, stopped right at eleven!”. I explained what happened and it said something like “Good job, you found the horrible, marrow cricket” (these last 3 words are verbatim) and then broke.
A fun lesson on the scientific method: A few days ago, OpenAI announced a rollout of “Advanced Voice Mode”. Confused by the messaging, I ran a test to see if their phone app could respond to an interruption:
Nisan: Can you count to 10, and then I’ll interrupt you in the middle and see how that works.
ChatGPT 4o: Sure! I’ll start counting now: One, two, three, four, fiv—
Nisan: Stop. Interruption.
ChatGPT 4o: How’d the interruption go?
Nisan: Oh, that went great. You stopped at five.
ChatGPT 4o: Awesome! Glad it worked smoothly. Do you want to try another test or experiment?
Nisan: Yeah, let’s do another interruption test. This time, please count to 20.
ChatGPT 4o: Got it! Starting now: One, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, thirt—
Nisan: Stop. Interruption.
ChatGPT 4o: How was that?
Nisan: Wow. [Thinks: It seemed to stop slightly before I interrupted it!]
ChatGPT 4o: Glad that impressed you! Want to try something else?
Nisan: Uh, hold on.
ChatGPT 4o: Take your time!
Nisan: [Generates a random number.] Okay, please count to 30, and I’ll interrupt at a random point.
ChatGPT 4o: Alright, starting now: One, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, thirteen, fourteen, fifteen—
[Nisan interrupts at some point, I forgot exactly where.]
ChatGPT 4o: —sixteen, seventeen, eighteen, nineteen, twenty, twenty-one, twenty-two, twenty-three, twenty-four, twenty-five, twenty-six, twenty-seven, twenty-eight, twenty-nine, thirty. Looks like I made it to the end! Did you mean to interrupt somewhere?
Nisan: Yeah, I did. It must not have worked. Let’s try that again. [Generates a random number.] Please count up to 30.
ChatGPT 4o: No problem, let’s try again! Starting now: One, two, three, four, five, six, seven, eight, nine, ten, eleven, twelve, thirteen, fourteen, fifteen—
[Nisan interrupts at some point.]
ChatGPT 4o: —sixteen, seventeen, eighteen, nineteen, twenty, twenty-one, twenty-two, twenty-three, twenty-four, twenty-fiv— Did it work this time?
Nisan: Uh, no.
I believe ChatGPT can’t hear me when it’s talking. But it correctly guessed when I would interrupt on the first two trials!
I tried to replicate. At 20 it went on to 25, and I explained what it got wrong. I tried again. I interrupted at 6 and it stopped at 7, saying “Gotcha, stopped right at eleven!”. I explained what happened and it said something like “Good job, you found the horrible, marrow cricket” (these last 3 words are verbatim) and then broke.