Yes, I also realized that “ideas” being a thing is due to bounded rationality—specifically they are the outputs of AI search. “Proofs” are weirder though, and I haven’t seen them distinguished very often. I wonder if this is a reasonable analogy to make:
I think only particular reward functions, such as in multi-agent/co-operative environments (agents can include humans, like in RLHF) or in actually interactive proving environments?
Yes, I also realized that “ideas” being a thing is due to bounded rationality—specifically they are the outputs of AI search. “Proofs” are weirder though, and I haven’t seen them distinguished very often. I wonder if this is a reasonable analogy to make:
Ideas : search
Answers : inference
Proofs: alignment
Ideas come from unsupervised training, answers from supervised training and proofs from RL on a specified reward function.
I think only particular reward functions, such as in multi-agent/co-operative environments (agents can include humans, like in RLHF) or in actually interactive proving environments?