I consider it unlikely that my interactions with ChatGPT 3.5 (that led to my original post) played a significant role as training data that helped make ChatGPT 4 so much better.
Why do you consider it unlikely?
If I was good at memorization (model parameter size) but bad at reasoning then your original post showing up in my training data would help me.
I’m also curious how much of the improvement comes from improvements to the hidden and secret prompt that Openai adds to your Chatgpt interactions. Unfortunately we can’t test that.
First, my original post didn’t provide the correct answers for most questions, only what was wrong with ChatGPT. Going from knowing what was wrong to actually giving correct answers seems like quite a leap. Further, ChatGPT changed its answers to better ones (including more rigorous explanations) even in cases where its original answers were correct.
Second, ChatGPT’s self-reported training data cutoff date at the time it was asked these questions was September 2021 or January 2022. To my knowledge, Issa didn’t ask it this question, but sources like https://www.reddit.com/r/ChatGPT/comments/16m6yc7/gpt4_training_cutoff_date_is_now_january_2022/ suggest that it was September 2021 at the time of his sessions, then became January 2022. So, the blog post, published in December 2022, should not have been part of its training data.
With that said, the sessions themselves (not the blog post about them) might have been part of the feedback loop, but in most cases I did not tell ChatGPT what was wrong with its answers within the session itself.
Why do you consider it unlikely?
If I was good at memorization (model parameter size) but bad at reasoning then your original post showing up in my training data would help me.
I’m also curious how much of the improvement comes from improvements to the hidden and secret prompt that Openai adds to your Chatgpt interactions. Unfortunately we can’t test that.
Good question!
First, my original post didn’t provide the correct answers for most questions, only what was wrong with ChatGPT. Going from knowing what was wrong to actually giving correct answers seems like quite a leap. Further, ChatGPT changed its answers to better ones (including more rigorous explanations) even in cases where its original answers were correct.
Second, ChatGPT’s self-reported training data cutoff date at the time it was asked these questions was September 2021 or January 2022. To my knowledge, Issa didn’t ask it this question, but sources like https://www.reddit.com/r/ChatGPT/comments/16m6yc7/gpt4_training_cutoff_date_is_now_january_2022/ suggest that it was September 2021 at the time of his sessions, then became January 2022. So, the blog post, published in December 2022, should not have been part of its training data.
With that said, the sessions themselves (not the blog post about them) might have been part of the feedback loop, but in most cases I did not tell ChatGPT what was wrong with its answers within the session itself.