Did you have to prompt it in any special ways to get it to do this?
I’ve tried this same experiment several times in the past because I have decades of writing that must be in the training set, but each time I didn’t make progress because the fine tuning refused to recognize that I was a person it knew about and could make writing sound like, even though if prompted differently could give me back unique claims that I made in posts.
I’ve not tried again with the latest models. Maybe they’ll do it now?
Yes. I have been iterating on the prompt for a while. Here are a few techniques that make it sound more like me.
I tell it to describe “lsusr”. In particular, what makes me different from other writers similar to me. Then I tell it to emphasize those things. I also say “lsusr” many times and use it as an adjective. I don’t know if this works but my intuition says it is natural for an LLM to understand.
I have it write a draft, then I tell it to tell me how it missed the mark, and to fix those mistakes. This prevents overfitting on my words. If I tell it to be “bold”, for example, it will overfit on “bold” instead of copying me along many dimensions. More generally, I don’t describe myself to ChatGPT. That results in ChatGPT copying my description of me instead of actual me. I let ChatGPT describe me, and then tell ChatGPT to write like it just described, but more so.
Often something ChatGPT writes will use a word like “Bayesian” that is associated with writers like me but which I don’t use much. Telling ChatGPT not to use specific words seems to improve its output without causing distortive side-effects.
Next step would be to try it on Claude, and on o1-mini/preview (the iterative revising should work for both, like it did with my Rubik’s Cube exercise). If you are in the base model adequately, then you should be in Llama-3-405b-base as well, and that’s available through a few APIs now, I believe, and you may find it to work a lot better if you can get the prompt right—several of your complaints like unsettlingness, groupthink, jokes, or indirection are characteristic of mode-collapsed tuned models but not base models.
Did you have to prompt it in any special ways to get it to do this?
I’ve tried this same experiment several times in the past because I have decades of writing that must be in the training set, but each time I didn’t make progress because the fine tuning refused to recognize that I was a person it knew about and could make writing sound like, even though if prompted differently could give me back unique claims that I made in posts.
I’ve not tried again with the latest models. Maybe they’ll do it now?
Yes. I have been iterating on the prompt for a while. Here are a few techniques that make it sound more like me.
I tell it to describe “lsusr”. In particular, what makes me different from other writers similar to me. Then I tell it to emphasize those things. I also say “lsusr” many times and use it as an adjective. I don’t know if this works but my intuition says it is natural for an LLM to understand.
I have it write a draft, then I tell it to tell me how it missed the mark, and to fix those mistakes. This prevents overfitting on my words. If I tell it to be “bold”, for example, it will overfit on “bold” instead of copying me along many dimensions. More generally, I don’t describe myself to ChatGPT. That results in ChatGPT copying my description of me instead of actual me. I let ChatGPT describe me, and then tell ChatGPT to write like it just described, but more so.
Often something ChatGPT writes will use a word like “Bayesian” that is associated with writers like me but which I don’t use much. Telling ChatGPT not to use specific words seems to improve its output without causing distortive side-effects.
Next step would be to try it on Claude, and on o1-mini/preview (the iterative revising should work for both, like it did with my Rubik’s Cube exercise). If you are in the base model adequately, then you should be in Llama-3-405b-base as well, and that’s available through a few APIs now, I believe, and you may find it to work a lot better if you can get the prompt right—several of your complaints like unsettlingness, groupthink, jokes, or indirection are characteristic of mode-collapsed tuned models but not base models.