The intuition becomes a little clearer when I take the following alternative derivation:
Let us look at the change in expected value when I increase my capabilities. From the expected value stemming from worlds where I win, we have (p∗q)′=p′∗q+p∗q′. For the other actor, their probability of winning decreases at a rate that matches my increase in probability of winning. Also, their probability of deploying a safe AI doesn’t change. So the change in expected value stemming fro m worlds where they win is −p′∗r∗q.
We should be indifferent to increasing capabilities when these sum to 0, so p′∗q+p∗q′=p′∗r∗q.
Let’s choose our units so km=1. Then, using the expressions for q′ from your comment, we have rq0p′0=p′0q0+p0q0(r−1).
Dividing through by q0 we get rp′0=p′0+p0(r−1). Collecting like terms we have (r−1)∗p′0=p0∗(r−1) and thus p′0=p0. Substituting for p′0 we have 12−p0=p0 and thus p0=14
Oh wait, yeah, this is just an example of the general principle “when you’re optimizing for xy, and you have a limited budget with linear costs on x and y, the optimal allocation is to spend equal amounts on both.”
Formally, you can show this via Lagrange-multiplier optimization, using the Lagrangian L(x,y)=xy−λ(ax+by−M). Setting the partials equal to zero gets you λ=y/a=x/b, and you recover the linear constraint function ax+by=M. So ax=by=M/2. (Alternatively, just optimizing xM−axb works, but I like Lagrange multipliers.)
In this case, we want to maximize pq+(1−p)rq0=p(q−rq0)−rq0, which is equivalent to optimizing p∗(q−rq0). Let’s define w=q−rq0, so we’re optimizing p∗w.
Our constraint function is defined by the tradeoff between p and w. p(k)=(.5−p0)k+p0, so k=p−p0.5−p0. w(k)=(r−1)q0k+q0−rq0=(r−1)q0(k−1), so k=−w(1−r)q0+1=p−p0.5−p0 .
Rearranging gives the constraint function .5−p0(1−r)q0w+p=.5. This is indeed linear, with a total ‘budget’ M of .5 and a p-coefficient b of 1. So by the above theorem we should have 1∗p=.5/2=.25.
Yes, you’re quite right!
The intuition becomes a little clearer when I take the following alternative derivation:
Let us look at the change in expected value when I increase my capabilities. From the expected value stemming from worlds where I win, we have (p∗q)′=p′∗q+p∗q′. For the other actor, their probability of winning decreases at a rate that matches my increase in probability of winning. Also, their probability of deploying a safe AI doesn’t change. So the change in expected value stemming fro m worlds where they win is −p′∗r∗q.
We should be indifferent to increasing capabilities when these sum to 0, so p′∗q+p∗q′=p′∗r∗q.
Let’s choose our units so km=1. Then, using the expressions for q′ from your comment, we have rq0p′0=p′0q0+p0q0(r−1).
Dividing through by q0 we get rp′0=p′0+p0(r−1). Collecting like terms we have (r−1)∗p′0=p0∗(r−1) and thus p′0=p0. Substituting for p′0 we have 12−p0=p0 and thus p0=14
Oh wait, yeah, this is just an example of the general principle “when you’re optimizing for xy, and you have a limited budget with linear costs on x and y, the optimal allocation is to spend equal amounts on both.”
Formally, you can show this via Lagrange-multiplier optimization, using the Lagrangian L(x,y)=xy−λ(ax+by−M). Setting the partials equal to zero gets you λ=y/a=x/b, and you recover the linear constraint function ax+by=M. So ax=by=M/2. (Alternatively, just optimizing xM−axb works, but I like Lagrange multipliers.)
In this case, we want to maximize pq+(1−p)rq0=p(q−rq0)−rq0, which is equivalent to optimizing p∗(q−rq0). Let’s define w = q−rq0, so we’re optimizing p∗w.
Our constraint function is defined by the tradeoff between p and w. p(k)=(.5−p0)k+p0, so k=p−p0.5−p0. w(k)=(r−1)q0k+q0−rq0=(r−1)q0(k−1), so k=−w(1−r)q0+1=p−p0.5−p0 .
Rearranging gives the constraint function .5−p0(1−r)q0w+p=.5. This is indeed linear, with a total ‘budget’ M of .5 and a p-coefficient b of 1. So by the above theorem we should have 1∗p=.5/2=.25.