I think the name “prompt tuning” is a little misleading in this context, because the prompts in that setting aren’t actually fixed tokens from the model’s vocabulary and we can’t interpret them as saying “this token in the text string is fixed”. In particular, prompt tuning seems much closer to model fine-tuning in terms of what’s actually happening (gradient descent on the embeddings of some auxiliary tokens).
Yeah, but they’re still operating on the same channel. I guess what I don’t get is how removing the limitation that they have to be tokens semantically sensible to us would allow it to access conditionals that were qualitatively beyond reach before.
I think the name “prompt tuning” is a little misleading in this context, because the prompts in that setting aren’t actually fixed tokens from the model’s vocabulary and we can’t interpret them as saying “this token in the text string is fixed”. In particular, prompt tuning seems much closer to model fine-tuning in terms of what’s actually happening (gradient descent on the embeddings of some auxiliary tokens).
Yeah, but they’re still operating on the same channel. I guess what I don’t get is how removing the limitation that they have to be tokens semantically sensible to us would allow it to access conditionals that were qualitatively beyond reach before.