Rohin Shah comments on Answering questions honestly instead of predicting human answers: lots of problems and some solutions

[Rohin Shah]

I think I might be missing a change you made to the algorithm. Can ${\theta }_1$ write an arbitrary program for $f_{?}$? In that case, what prevents you from getting

def M_theta_1_plus(theta_2, x, q):
    axioms = world_model(theta_2=theta_2)(x)
    deduced_stmts = deduction(axioms)
    return {
        "f": f_minus(q, deduced_stmts),
        "f?": True,
    }

It seems like this should be lower complexity than the intended result, since True has much lower complexity than H_understands?

It seemed to me like you were using it in a way such that $f_1$ shared no weights with $f_2$

I mean, I would still have said this because I interpret a “head” $f_1$ as “the part after the shared layers”, but I’m also happy to instead treat $f_1$ as the entire function $X\times Q\to A$ for which the first head forms part of the implementation.

[evhub]

Can ${\theta }_1$ write an arbitrary program for $f_{?}$?

Yes—at least that’s the assumption I’m working under.

It seems like this should be lower complexity than the intended result, since True has much lower complexity than H_understands?

I agree that the ${\theta }_1$ you’ve described has lower complexity than the intended ${\theta }_1$—but the ${\theta }_2$ in this case has higher complexity, since ${\theta }_2$ is no longer getting any of its complexity for free from conditioning on the $f_{?}$ condition. And in fact what you’ve just described is precisely the unintended model—what I call $M^{-}$—that I’m trying to compete against, with the hope being that the savings that $M^{+}$ gives you in ${\theta }_2$ are sufficient to compensate for the loss in having to specify $f^{+}$ and H_understands in ${\theta }_1$.

If we calculate the complexity of your proposal, we get $$\begin{array}{cc}& \text{complexity}\left(M^{-}\right)\\ & =\text{complexity}\left({\theta }_1^{-}\right)+\text{complexity}\left({\theta }_2^{-}|M^{-}{|}_{f_{?}}\right)\\ & =\text{complexity}(W-H)+\text{complexity}\left(f^{-}\right)+\text{complexity}\left(H|\text{True}\right)\\ & =\text{complexity}(W-H)+\text{complexity}\left(f^{-}\right)+\text{complexity}\left(H\right)\\ & \approx \text{complexity}\left(W\right)\end{array}$$ whereas, if we calculate the complexity of the intended $M^{+}$, we get $$\begin{array}{cc}& \text{complexity}\left(M^{+}\right)\\ & =\text{complexity}\left({\theta }_1^{+}\right)+\text{complexity}\left({\theta }_2^{+}|M^{+}{|}_{f_{?}}\right)\\ & =\text{complexity}(W-H)+\text{complexity}\left(f^{-}\right)+\text{complexity}\left(f^{+}\right)+\text{complexity}\left(\text{H\_understands}\right)+\text{complexity}(H|\text{H\_understands}\to f^{+}=f^{-})\\ & \approx \text{complexity}(W-H)+\text{complexity}\left(f^{+}\right)+\text{complexity}\left(\text{H\_understands}\right)+\text{complexity}\left(H\right)-\underset{{\theta }_2}{min}\left\{\text{complexity}\right({\theta }_2)|{\text{H\_understands}}_{H={\theta }_2}\to f_{H={\theta }_2}^{+}=f_{H={\theta }_2}^{-}\}\\ & \approx \text{complexity}\left(W\right)+\text{complexity}\left(f^{+}\right)+\text{complexity}\left(\text{H\_understands}\right)-\underset{{\theta }_2}{min}\left\{\text{complexity}\right({\theta }_2)|{\text{H\_understands}}_{H={\theta }_2}\to f_{H={\theta }_2}^{+}=f_{H={\theta }_2}^{-}\}\end{array}$$ such that you can see that the question of which one wins is precisely dependent on whether the savings from conditioning on $\text{H\_understands}\to f^{+}=f^{-}$ offsets the cost of having to specify $f^{+}$ and $\text{H\_understands}$.

[Rohin Shah]

such that you can see that the question of which one wins is precisely dependent on whether the savings from conditioning on $\text{H\_understands}\to f^{+}=f^{-}$ offsets the cost of having to specify $f^{+}$ and $\text{H\_understands}$.

Yeah, that makes sense. I guess I don’t really see the intuition about why this should be true, but fair enough to leave that as an open question.