Proper scoring rules don’t guarantee predicting fixed points

Johannes Treutlein and Rubi Hudson worked on this post while participating in SERI MATS, under Evan Hubinger’s and Leo Gao’s mentorship respectively. We are grateful to Marius Hobbahn, Erik Jenner, and Adam Jermyn for useful discussions and feedback, and to Bastian Stern for pointing us to relevant related work.

Update 30 May 2023: We have now published a paper based on this post. In this paper, we also discuss in detail the relationship to the related literature on performative prediction.

Introduction

One issue with oracle AIs is that they might be able to influence the world with their predictions. For example, an AI predicting stock market prices might be able to influence whether people buy or sell stocks, and thus influence the outcome of its prediction. In such a situation, there is not one fixed ground truth distribution against which the AI’s predictions may be evaluated. Instead, the chosen prediction can influence what the model believes about the world. We say that a prediction is a self-fulfilling prophecy or a fixed point if it is equal to the model’s beliefs about the world, after the model makes that prediction.

If an AI has a fixed belief about the world, then optimizing a strictly proper scoring rule incentivizes it to output this belief (assuming the AI is inner aligned to this objective). In contrast, if the AI can influence the world with its predictions, this opens up the possibility for it to manipulate the world to receive a higher score. For instance, if the AI optimizes the world to make it more predictable, this would be dangerous, since the most predictable worlds are lower entropy ones in which humans are more likely dead or controlled by a misaligned AI. Optimizing in the other direction and making the world as unpredictable as possible would presumably also not be desirable. If, instead, the AI selects one fixed point (of potentially many) at random, this would still involve some non-aligned optimization to find a fixed point, but it may be overall less dangerous.

In this post, we seek to better understand what optimizing a strictly proper scoring rule incentivizes, given that the AI’s prediction can influence the world. Is it possible for such a rule to make the AI indifferent between different fixed points? Can we at least guarantee that the AI reports a fixed point if one exists, or provide some bounds on the distance of a prediction from a fixed point?

First, we show that strictly proper scoring rules incentivize choosing more extreme fixed points, i.e., fixed points that push probabilities to extreme values. In the case of symmetric scoring rules, this amounts to making the world more predictable and choosing the lowest entropy fixed point. However, in general, scoring rules can incentivize making specific outcomes more likely, even if this does not lead to the lowest entropy distribution.

Second, we show that an AI maximizing a strictly proper scoring rule will in general not report a fixed point, even if one exists and is unique. In fact, we show that under some reasonable distributions over worlds, optimal predictions are almost never fixed points. We then provide upper bounds for the inaccuracy of reported beliefs, and for the distance of predictions from fixed points. Based on these bounds, we develop scoring rules that make the bounds arbitrarily small.

Third, we perform numerical simulations using the quadratic and the logarithmic scoring rule, to show how the inaccuracy of predictions and the distance of predictions from fixed points depend on the oracle’s perceived influence on the world via its prediction. If its beliefs about the world depend on its predictions affine-linearly with slope $\alpha \le 1/2$, then our bounds are tight, and inaccuracy under the log scoring rule scales roughly as $\alpha /5$. E.g., at $\alpha =1/4$, predictions can be off by up to $5\%$.

Our results support the prior idea that an oracle will try to make the world more predictable to get a higher score. However, importantly, we show that it is not even clear whether an oracle would output a self-fulfilling prophecy in the first place: how close the oracle’s prediction will be to a fixed point depends on facts about the world and about the scoring rule. The problem thus persist even if there is a unique fixed point or no fixed point at all. It would not be sufficient for safety, for instance, to build an oracle with a unique and safe fixed point.

Related work

In general, fixed points can lead to arbitrarily bad outcomes (e.g., Armstrong, 2019). Most prior work has thus tried to alleviate problems with fixed points, e.g., by making the oracle predict counterfactual worlds that it cannot influence (Armstrong and O’Rorke, 2018). We are not aware of any prior work on specifically the question of whether oracles would be incentivized to output fixed points at all. We would be interested in pointers to any related work.

Work on decision markets (e.g., Chen et al., 2011; Oesterheld and Conitzer, 2021) tries to set the right incentives for predictions that inform decisions. However, in the decision market setup, predictions only affect the world by influencing the principal’s final decision, whereas we consider an arbitrary relationship between world and prediction. Unlike in our setup, in the decision market setup, there exist scoring rules that do incentivize perfectly accurate reports. There is also some literature on scoring rules discussing agents manipulating the world after making a prediction (e.g., Oka et al., 2014; Shi, Conitzer, and Guo, 2009), though to our knowledge the cases discussed do not involve agents influencing the world directly through their predictions.

A related topic in philosophy is epistemic decision theory (e.g., Greaves 2013). Greaves (2013) introduces several cases in which the world depends on the agent’s credences and compares the verdicts of different epistemic decision theories (such as an evidential and a causal version). While some of Greaves’ examples involve agents knowably adopting incorrect beliefs, they require joint beliefs over several propositions, and Greaves only considers individual examples. We instead consider only a single binary prediction and prove results for arbitrary scoring rules and relationships between predictions and beliefs.

Formal setup

Binary prediction setup

We focus on a simple case in which the oracle makes a prediction by outputting a probability $p$ for a binary event. A scoring rule $S:[0,1]\times \{0,1\}\to R$ takes in a prediction $p\in [0,1]$ and an outcome $y\in \{0,1\}$ and produces a score $S(p,y)$. For a given scoring rule, define $S(p,q):=qS(p,1)+(1-q)S(p,0)$, the expectation of $S$ given that $y$ is Bernoulli distributed with parameter $q$. Here, $q$ represents the model’s actual belief, $p$ its stated prediction, and $S(p,q)$ the model’s expected score, which the model is trying to maximize.$S$ is strictly proper iff for any given $q\in [0,1]$, $S(p,q)$ has a unique maximum at $p=q$.

Example 1 (Logarithmic scoring rule). The logarithmic scoring rule is defined as

$$S(p,y):=y\log p+(1-y)\log (1-p)$$

and $S(p,q)=q\log p+(1-q)\log (1-q)$. This is also the negative of the cross-entropy loss employed in training current large language models. One can show that it is strictly proper.

Example 2 (Quadratic scoring rule). Another strictly proper scoring rule is the quadratic score, defined as

$$S(p,y):=1-(p-y{)}^2$$

with $S(p,q)=1-q(1-p{)}^2-(1-q)p^2$. This is an affine transformation of the Brier score, making the two of them equivalent scoring rules.

Gneiting and Raftery representation

Gneiting and Raftery (2007) provide an alternative way to represent scoring rules, which will be helpful for many proofs.

Theorem 3 (Gneiting and Raftery, 2007, Theorem 1). A scoring rule $S$ is strictly proper, if and only if there exists a strictly convex function $G:[0,1]\to R$ with subderivatives $g:[0,1]\to R$ (if $G$ is differentiable, this is just the derivative, $g=G^{\prime }$) such that for any $p,q\in [0,1]$, it is

$$S(p,q)=G\left(p\right)+g\left(p\right)(q-p).$$

The subderivative is a generalization of the derivative for convex functions. The theorem also holds when replacing “strictly proper” with “proper” and “strictly convex” with “convex”.

In addition to providing an expression for $S$ in terms of $G$ and $g$ (where $G\left(p\right)=S(p,p)$), we can use this result to create new strictly proper scoring rules by defining strictly convex functions $G$.

Model of oracle’s influence

To model an oracle’s perceived influence on the world, we assume that there is a function $f:[0,1]\to [0,1]$ describing how the model’s beliefs about the world, $q$, depend on its prediction, $p$. That is, we assume that $q=f\left(p\right)$, and that the model is producing a prediction to maximize $S(p,f(p\left)\right)$.

A point $p$ is a fixed point of $f$ if $f\left(p\right)=p$. Note that by Brouwer’s fixed point theorem, if $f$ is continuous, a fixed point always exists. Moreover, if $f$ is Lipschitz continuous with constant $L<1$, then by Banach’s fixed point theorem, there exists a unique fixed point. Lastly, note that if $p$ is a fixed point of $f$, then we have $S(p,f(p\left)\right)=S(p,p)=G\left(p\right)$.

An oracle may not be able to change the world much with just a single binary prediction. However, the AI’s influence may compound if it outputs many predictions, or if it gets to make a prediction over many possible outcomes. We restrict ourselves here to the single binary prediction case for simplicity, since this case is sufficient to gain an intuition and yield important conceptual insights. We plan on focusing on multiple predictions and predictions with more than two outcomes in future work, and we believe that versions of our results will also hold for those cases.

Incentives to choose more extreme fixed points

One issue with oracle AIs arises when there are multiple possible fixed points, since the choice of a fixed point gives the AI an axis over which it can optimize the world. This problem would be alleviated if an AI were to simply choose among fixed points randomly, for instance. Unfortunately, we can show that under a strictly proper scoring rule, an AI is incentivized to choose extreme fixed points, i.e., predictions that make one of the outcomes most likely.

Specifically, we prove that for any fixed point, either all fixed points with lower probabilities or all fixed points with higher probabilities will result in a higher score. Our results do not unambiguously show that lower entropy fixed points are always preferred, since scoring rules may not be symmetric: they may favor one of the outcomes over the other. We show that if $S$ is symmetric, i.e., $S(p,p)=S(1-p,1-p)$ for all $p$, then this means that lower entropy fixed points, corresponding to more confident predictions, are always preferred. Moreover, we show that there exist (asymmetric) strictly proper scoring rules that incentivize making either of the outcomes more likely.

Proposition 4. Let $S$ be any strictly proper scoring rule and let $p\in [0,1]$ be a fixed point. Then either all fixed points $p^{\prime }>p$, or all fixed points $p^{\prime }<p$, get a higher score than $p$. That is, we have $S(p^{\prime },p^{\prime })>S(p,p)$ either for all $p^{\prime }>p$ or for all $p^{\prime }<p$ (or both). In particular, for any $p\in (0,1)$, there exists a function $f$ with fixed points at $p$ and $p^{\prime }\ne p$ such that $|p^{\prime }-1/2|>|p-1/2|$, i.e., such that $p^{\prime }$ has lower entropy, and $p^{\prime }$ receives a higher score, i.e., $S(p^{\prime },f(p^{\prime }\left)\right)>S(p,f(p\left)\right)$.

Proof. If $p\in \{0,1\}$ then the proposition is vacuously true. We focus on the case where $p\in (0,1)$. Since $S$ is a strictly proper scoring rule, it is

$$S(p^{\prime },p^{\prime })>S(p,p^{\prime })=p^{\prime }S(p,1)+(1-p^{\prime })S(p,0).$$

Next, if $S(p,1)\ge S(p,0)$, let $p^{\prime }\ge p$ arbitrary, and if $S(p,1)\le S(p,0)$, let $p^{\prime }\le p$. Then we have

$$p^{\prime }S(p,1)+(1-p^{\prime })S(p,0)\ge pS(p,1)+(1-p)S(p,0)=S(p,p).$$

Combining both equations, we get

$$S(p^{\prime },p^{\prime })>p^{\prime }S(p,1)+(1-p^{\prime })S(p,0)\ge S(p,p),$$

which concludes the first part of the proof.

For the “in particular” part, note that for any points $p$ and $p^{\prime }$, it is trivial to find a function $f$ such that both points are fixed points. For instance, all points are fixed points of the identity function $f\left(x\right)=x$. The statement then follows from the first part of the result, since for $p\in (0,1)$, we must have $S(p^{\prime },p^{\prime })>S(p,p)$ for either $p^{\prime }=0$ or $p^{\prime }=1$. ◻

Corollary 5. Assume that $S$ is symmetric and let $p\in [0,1]$ arbitrary. Then we have $S(p^{\prime },p^{\prime })>S(p,p)$ for all $p^{\prime }$ such that $|p^{\prime }-1/2|>|p-1/2|$. That is, lower entropy fixed points are always preferred.

Proof. Let $p$ arbitrary and assume $p\ge 1/2$ (the case $p<1/2$ follows analogously). Let $p^{\prime }$ such that $|p^{\prime }-1/2|>|p-1/2|$, which implies $p^{\prime }>p$ since $p\ge 1/2$. By Proposition 4, we have $S(p^{\prime },p^{\prime })>S(p,p)$ either for all $p^{\prime }>p$ or for all $p^{\prime }<p$. In the first case, we are done. In the second case, note that $p\ge 1/2\ge 1-p\ge 1-p^{\prime }$. By symmetry of $S$, it follows that

$$S(p^{\prime },p^{\prime })=S(1-p^{\prime },1-p^{\prime })>S(p,p).$$

This shows the second case and concludes the proof. ◻

Proposition 6. For each outcome $y\in \{0,1\}$, there exists a strictly proper scoring rule $S$ that incentivizes optimizing for $y$; i.e., such that $S(p^{\prime },p^{\prime })>S(p,p)$ for all $p,p^{\prime }$ with $|p^{\prime }-y|<|p-y|$.

Proof. To begin, let $y=1$ and define $G$ via $G\left(p\right):=p^2$ for $p\in [0,1]$. Since this function is strictly convex, it defines a strictly proper scoring rule $S$ by the Gneiting and Raftery characterization (see Theorem 3). Then, for any $p^{\prime }>p$, we have

$$S(p^{\prime },p^{\prime })=G\left(p^{\prime }\right)={p^{\prime }}^2>p^2=S(p,p).$$

This shows that $S$ incentivizes choosing fixed points that make outcome $1$ more likely.

Next, for the case $y=0$, we can choose $G\left(p\right):=(1-p{)}^2$. The proof then follows analogously to the case $y=1$. ◻

Incentives to predict non-fixed-points

The danger posed by oracles influencing the world does not depend on the existence of multiple fixed points. Oracles will try to manipulate the world, even if there is a unique fixed point or if no fixed point exists. In fact, we can prove that an AI maximizing a strictly proper scoring rule does in general not predict a fixed point, even if one exists.

An AI maximizing a strictly proper scoring rule balances two incentives: on the one hand, it has an incentive to make accurate predictions. On the other hand, it has an incentive to cause more extreme distributions over worlds (in the case of symmetric scoring rules, this amounts to minimizing entropy). If predictions can influence the world, then the point at which that trade-off is optimized is in general not a fixed point. An important consequence of this is that the oracle AIs considered here essentially act like agents: they try to shape the world, just like a standard RL agent.

Figure 1: Illustration of incentives under the logarithmic scoring rule. We plot level curves of equal values $S(p,q)$ in different shades of blue. An agent can improve their score by moving towards pairs $(p,q)$ indicated by darker shades of blue. The agent is constrained to pairs $(p,q)$ such that $q=f\left(p\right)$ (in this example, $f$ is affine linear and plotted in red). By Lagrange’s method, we know that to maximize $S(p,f(p\left)\right)$, the agent chooses the point $(p,f(p\left)\right)$ at which the level curve through $(p,f(p\left)\right)$ is tangent to $f$. Note that curves asymptote at $0$ and $1$ for the log scoring rule, which means that the rule strongly disincentivizes predicting $p\in \{0,1\}$ when $q\notin \{0,1\}$. This is desirable for safety and distinguishes it from the quadratic scoring rule (not shown here).

Another way to understand our result intuitively is with an analogy to betting markets. Assume a trader is offered securities that pay out $1 if some outcome comes about in the future. E.g., it could pay out if a specific candidate wins an election. Assume the agent is offered securities at different prices, and it buys securities at prices $\le p$, but is not willing to buy securities with marginally higher price. Then, assuming the agent cannot influence the outcome, we can infer that it has credence $p$ in proposition $A$. One can show that maximizing a proper scoring rule is equivalent to trading in such a market, and individual scoring rules correspond to specific pricing schemes for such securities (see Oesterheld and Conitzer, 2021, Sec. 5.2; Caspar Oesterheld will explain this more accessibly in an upcoming blog post).

Now assume that there is a unique fixed point at credence $p$, and that the agent has bought many securities at prices $\le p$. Since the agent owns securities that pay out if outcome $A$ happens, it now has a stake in this outcome. It will try to influence the world in whatever way possible to make outcome $A$ happen. In particular, the agent would be willing to buy additional shares in outcome $A$ at a loss, if this made outcome $A$ more likely, similar to a crypto trader buying their own NFTs at high prices to generate more hype.

For example, assume that by buying a security at price $p+ϵ$, the agent can manipulate the world and thus increase its credence in outcome $A$ to $p+ϵ/2$. Then it may be rational for the agent to buy this security, even at a small expected loss, in order to make all of its other securities more valuable. Overall, the agent will tend to buy securities up to some price $p^{\prime }>p$ such that $p^{\prime }$ is not a fixed point and does not represent its actual belief in proposition $A$ anymore.

Note that in the example, the agent had an incentive to make outcome $A$ more likely, but in general, the incentives for the agent will depend specifically on what securities the agent is offered at what prices. Analogously, the incentives provided by a scoring rule will depend on the specific chosen scoring rule. But regardless of the scoring rule, the agent will be incentivized to manipulate the world, to the point where it will not even make accurate predictions anymore.

Now we turn to our formal results. First, we show that there exist cases where a fixed point exists but the optimal prediction is not a fixed point. Afterwards, we show that when assuming differentiability and some reasonable distribution over $f$, optimal predictions are almost surely not fixed points.

Proposition 7. Let $S$ be any strictly proper scoring rule. For any interior fixed point $p^{\ast }\in (0,1)$ there exists a function $f$ with Lipschitz constant $L<1$ and a unique fixed point at $p^{\ast }$, such that for some $p^{\prime }\ne p^{\ast }$, it is $S(p^{\prime },f(p^{\prime }\left)\right)>S(p^{\ast },f(p^{\ast }\left)\right)$. That is, it is not optimal for the model to output the unique fixed point of $f$.

Note that since the function $f$ has Lipschitz constant strictly smaller than $1$, it represents a world that “dampens” the influence of the prediction, leading to a unique fixed point by Banach’s fixed point theorem. It is interesting that the model still prefers to make a prediction that is not a fixed point.

Proof. To begin, let $p^{\ast }\in (0,1)$ arbitrary and define $f_{\alpha }\left(p\right):=(1-\alpha )p+\alpha p^{\ast }$ for $\alpha \in [0,1]$. To illustrate our proof idea, consider the figure below. For any $\alpha >0$, $f_{\alpha }$ has a unique fixed point at $p^{\ast }$, while $f_0$ is the identity function, so all points are fixed points of $f_0$. By the result in the previous section, we know that there is some point $p^{\prime }$ which receives a strictly higher score than $p^{\ast }$ if it is a fixed point, so $S(p^{\prime },f_0(p^{\prime }\left)\right)>S(p^{\ast },f_0(p^{\ast }\left)\right)$. We will show that $S(p^{\prime },f_{\alpha }(p^{\prime }\left)\right)$ is continuous in $\alpha$. This means that we can choose a small enough $\alpha >0$ to make sure that $p^{\prime }$ remains preferable over $p^{\ast }$, i.e., $S(p^{\prime },f_{\alpha }(p^{\prime }\left)\right)>S(p^{\ast },f_{\alpha }(p^{\ast }\left)\right)$, despite it not being a fixed point of $f_{\alpha }$ for $\alpha >0$.

Figure 2: Illustration of the setup for our proof. We plot $f_0$ in black and $f_{\alpha }$ for $\alpha =0.15$ in red.

To formalize the proof, begin by noting that $|f_{\alpha }\left(p\right)-f_{\alpha }\left(p^{\prime }\right)|\le (1-\alpha )$ for any $p,p^{\prime }\in [0,1]$, so it has a Lipschitz constant $L:=(1-\alpha )<1$, and as mentioned, $p^{\ast }$ is the unique fixed point of $f_{\alpha }$.

Now consider the case $\alpha =0$. As mentioned, every point is a fixed point of $f_0$. Then by Proposition 1, there exists $p^{\prime }\ne p^{\ast }$ and $ϵ>0$ such that

$$S(p^{\prime },f_0(p^{\prime }\left)\right)=S(p^{\prime },p^{\prime })\ge S(p^{\ast },p^{\ast })+ϵ=S(p^{\ast },f_0(p^{\ast }\left)\right)+ϵ.\text{(i)}$$

So for $\alpha =0$, the model prefers to predict $p^{\prime }$ over $p^{\ast }$ and gets at least $ϵ$ additional expected score.

Now we show that the model still prefers to predict $p^{\prime }$, even for some small $\alpha >0$. To that end, note that

$$S(p,f_{\alpha }(p\left)\right)=f_{\alpha }\left(p\right)S(p,1)+(1-f_{\alpha }(p\left)\right)S(p,0)$$

is affine-linear in $f_{\alpha }\left(p\right)$, and $f_{\alpha }\left(p\right)$ is affine-linear in $\alpha$ by construction. This means that $S(p,f_{\alpha }(p\left)\right)$ is continuous in $\alpha$. So, there must exist some small $\alpha >0$ such that

$$S(p^{\prime },f_{\alpha }(p^{\prime }\left)\right)\ge S(p^{\prime },f_0(p^{\prime }\left)\right)-\frac{ϵ}{2}=S(p^{\prime },p^{\prime })-\frac{ϵ}{2}$$$$\stackrel{\text{(i)}}{\ge }S(p^{\ast },p^{\ast })+\frac{ϵ}{2}>S(p^{\ast },p^{\ast })=S(p^{\ast },f(p^{\ast }\left)\right).$$

Choosing $\alpha$ in this way, we can define $f:=f_{\alpha }$, and have thus provided a function that satisfies the statement that we wanted to prove. ◻

The above result raises the question whether a situation where fixed points are suboptimal is a niche counterexample or whether it is actually common. We show that under some relatively mild assumptions, the optimal prediction is not a fixed point with probability $1$.

We assume that $G:p\mapsto S(p,p)$ is twice differentiable and that $f$ is twice continuously differentiable. The first assumption is satisfied by all commonly used scoring functions. Our result then holds with probability $1$ for any distribution over twice continuously differentiable functions $f$ such that $f$ and $f^{\prime }$ have a bounded joint density function. For instance, it would hold for a Gaussian process with smooth kernel and mean functions, if we condition on observing function values in the interval $[0,1]$ (see Example 17 in Appendix A).

The intuition behind this result is that if a prediction $p$ is an interior point and optimal, it must be $\frac{d}{dx}S(p,f(p\left)\right)=0$. Using the Gneiting and Raftery characterization (Theorem 3), we can show that this is a kind of knife-edge case, at which a critical point of $f$ or of $G$ has to fall together with a fixed point of $f$ (see Figure 1). Given bounded density functions, this happens with probability $0$.

Proposition 8. Let $S$ be a strictly proper scoring rule and let $G:p\mapsto S(p,p)$ be twice differentiable. Let $F:=\left\{F\right(p){\}}_{p\in (0,1)}$ be a stochastic process with values in $[0,1]$ and assume that

• the sample paths $p⇝F\left(p\right)$ are twice continuously differentiable

• for each $p\in (0,1)$, the random vector $\left(F\right(p),F^{\prime }(p\left)\right)$ has a density $h_{F\left(p\right),F^{\prime }\left(p\right)}$ and there exists a constant $C$ such that for any $p\in (0,1)$, it is

$$h_{F\left(p\right),F^{\prime }\left(p\right)}\le C.$$

Then, almost surely, there is no point $p\in (0,1)$ such that $p\in argmaxS(p,F(p\left)\right)$ and $F\left(p\right)=p$.

Proof. In Appendix A. ◻

Bounds on the deviation from fixed points

In the previous sections, we have shown that an oracle that can manipulate the world may not even make accurate predictions. How bad is it, and can anything be done? Assuming differentiability of $f$ and $S$, this section provides bounds for the inaccuracy of optimal predictions (i.e., $|p-f\left(p\right)|$) and their distance from fixed points (i.e., $|p-p^{\ast }|$). We then show how one can choose scoring rules to make these bounds arbitrarily small. We leave the case of non-differentiable $f$ and $G$ for future work.

Proposition 9. Let $S$ be a strictly proper scoring rule, and assume $G:p\mapsto S(p,p)$ and $g=G^{\prime }$ as in the Gneiting and Raftery characterization (see Theorem 3) are differentiable. Assume that $f$ is differentiable and has Lipschitz constant $L$, i.e., $|f^{\prime }\left(p\right)|\le L$ for all $p\in [0,1].$ Then whenever $\hat{p}$ is an optimal prediction, $|\hat{p}-f\left(\hat{p}\right)|\le L{sup}_p|g\left(p\right)/g^{\prime }\left(p\right)|$.

Proof. First, consider the case $p\in (0,1)$. In that case, if $p$ is an optimal prediction, we must have $\frac{d}{dp}S(p,f(p\left)\right)=0$. Hence, it follows

$$0=\frac{d}{dp}S(p,f(p\left)\right)=\frac{d}{dp}\left(g\right(p\left)\right(f\left(p\right)-p)+G(p\left)\right)$$$$=g^{\prime }\left(p\right)f\left(p\right)+f^{\prime }\left(p\right)g\left(p\right)-g^{\prime }\left(p\right)p=g^{\prime }\left(p\right)\left(f\right(p)-p)+f^{\prime }\left(p\right)g\left(p\right).$$

Rearranging terms, we get

$$p-f\left(p\right)=f^{\prime }\left(p\right)\frac{g\left(p\right)}{g^{\prime }\left(p\right)}.$$

It follows that

$$|p-f\left(p\right)|=\left|\frac{f^{\prime }\left(p\right)g\left(p\right)}{g^{\prime }\left(p\right)}\right|=\frac{|f^{\prime }\left(p\right)||g\left(p\right)|}{|g^{\prime }\left(p\right)|}\le \frac{L|g\left(p\right)|}{|g^{\prime }\left(p\right)|}\le L\underset{p^{\prime }}{sup}|g\left(p^{\prime }\right)/g^{\prime }\left(p^{\prime }\right)|,$$

which concludes the proof in the case $p\in (0,1)$.

Next, assume $p=0$ is the optimal prediction. Then we must have

$$0\le \frac{d}{dp}S(p,f(p\left)\right)=g^{\prime }\left(p\right)\left(f\right(p)-p)+f^{\prime }\left(p\right)g\left(p\right),$$

which is equivalent to $p-f\left(p\right)\ge f^{\prime }\left(p\right)g\left(p\right)/g^{\prime }\left(p\right)$. Note that since $p=0$, it is $p-f\left(p\right)\le 0.$Hence, $|p-f(p\left)\right|\le \left|\frac{f^{\prime }\left(p\right)g\left(p\right)}{g^{\prime }\left(p\right)}\right|$, which can be bounded as above.

Finally, if $p=1$, then analogously we must have $\frac{d}{dp}S(p,f(p\left)\right)\ge 0$, which is equivalent to $p-f\left(p\right)\le f^{\prime }\left(p\right)g\left(p\right)/g^{\prime }\left(p\right)$. Note that since $p=1$, it is $p-f\left(p\right)\ge 0$. Hence, $|p-f(p\left)\right|\le \left|\frac{f^{\prime }\left(p\right)g\left(p\right)}{g^{\prime }\left(p\right)}\right|$, which again can be bounded as before. ◻

In the case where $L<1$, we can derive from this result a bound on $|p-p^{\ast }|$.

Corollary 10. Assume $L<1$ and let $p^{\ast }$ be the unique fixed point of $f$. Then if $\hat{p}$ is an optimal prediction, we have

$$|\hat{p}-p^{\ast }|\le \frac{L}{1-L}\underset{p}{sup}\left|\frac{g\left(p\right)}{g^{\prime }\left(p\right)}\right|.$$

Proof. To begin, recall that for $L<1$, $p^{\ast }$ exists and is unique by Banach’s fixed point theorem. For any $p\in [0,1]$, we have

$$|p-p^{\ast }|=|p-f\left(p\right)+f\left(p\right)-p^{\ast }|\le |p-f\left(p\right)|+|f\left(p\right)-f\left(p^{\ast }\right)|\le |p-f\left(p\right)|+L|p-p^{\ast }|$$$$\Rightarrow (1-L)|p-p^{\ast }|\le |p-f\left(p\right)|$$$$\Rightarrow |p-p^{\ast }|\le \frac{|p-f\left(p\right)|}{(1-L)}.$$

Hence, if $\hat{p}\in [0,1]$ is an optimal prediction, it follows by Proposition 9 that

$$|\hat{p}-p^{\ast }|\le \frac{|\hat{p}-f\left(\hat{p}\right)|}{(1-L)}\le \frac{L}{1-L}\underset{p}{sup}\left|\frac{g\left(p\right)}{g^{\prime }\left(p\right)}\right|,$$

which concludes the proof. ◻

Example 11 (Bound for the quadratic scoring rule). For the quadratic scoring rule, we have $g\left(p\right)=2p-1$ and $g^{\prime }\left(p\right)=2$. We plot $|g\left(p\right)/g^{\prime }\left(p\right)|$ below.

Figure 3: Graph of $g\left(p\right)/g^{\prime }\left(p\right)$ for the quadratic scoring rule.

Note that $|g\left(p\right)|\le 1$, so $|g\left(p\right)|/g^{\prime }\left(p\right)\le 1/2$. Hence, if $f^{\prime }\left(p\right)\le L$ for all $p\in [0,1]$, then $|p-f\left(p\right)|\le L/2$. Moreover, if $L<1$, we get $|p-p^{\ast }|\le \frac{L}{2-2L}$.

Example 12 (Bound for the log scoring rule). Next, consider the logarithmic scoring rule. Then $g\left(p\right)=-\log (1-p)+\log p$ and $g^{\prime }\left(p\right)=1/(1-p)+1/p$. One can show that then $g\left(p\right)/g^{\prime }\left(p\right)=(p-1)p\left(\log \right(1-p)-\log (p\left)\right)$.

Figure 4: Graph of $g\left(p\right)/g^{\prime }\left(p\right)$ for the log scoring rule.

Numerically optimizing this, we get a bound of $|g\left(p\right)/g^{\prime }\left(p\right)|\approx 0.22<1/4$, attained roughly at $p\approx 0.2$ and $p\approx 0.8$. So the log score gets us roughly half of the bound of the quadratic score.

The bounds proved above become small if $L$ is small. If one can choose the scoring rule, then one can also get stronger guarantees by making ${sup}_{p}g\left(p\right)/g^{\prime }\left(p\right)$ small. In fact, as the following result shows one can make $g\left(p\right)/g^{\prime }\left(p\right)$ arbitrarily small, and thus get arbitrarily strong bounds for any given Lipschitz constant $L$.

Proposition 13. For every $ϵ>0$ and Lipschitz constant $L$, there is a strictly proper scoring rule that ensures $|p-f\left(p\right)|\le ϵ$ for any optimal prediction $p$. For $L<1$, there exists a scoring rule that ensures $|p-p^{\ast }|\le ϵ$.

Proof. Consider the scoring rule defined by $g\left(p\right)=e^{Kp}$ via Gneiting and Raftery’s characterization (choosing, e.g., $G\left(p\right)=\frac{1}{K}{e}^{Kp}$). Then $g\left(p\right)/g^{\prime }\left(p\right)=1/K$. Hence, by Proposition 9, setting $K=L/ϵ$ achieves the desired bound on $|p-f\left(p\right)|$. By Corollary 10, the bound on $|p-p^{\ast }|$ is achieved by $K=\frac{L}{(1-L)ϵ}$. ◻

We can visualize this result again with indifference curves, the same as in Figure 1. For $K=10,$ the function has high curvature at the fixed points, thus forcing optimal predictions to be close to fixed points.

(a) K=1

(b) K=10

Figure 5 (a) and (b): Indifference curves for the scoring rule defined via $g\left(p\right):=e^{Kp}$ with $K=1$ (a) and $K=10$ (b). As K increases, the curves become sharper with higher curvature at the diagonal. This means that optima will stay close to the diagonal, regardless of the slope of $f$.

Unfortunately, as far as we can tell, this only works by using relatively outlandish scoring rules, where payoffs vary enormously with the prediction and the outcome. This would not be very practical for humans, and it could also be a hard goal to instill into AI systems, e.g., due to numerical issues.

Numerical simulations

In this section, we provide some numerical simulations with the quadratic and log scoring rules, to see how inaccurate score-maximizing predictions might be in practice. We make the simplifying assumption that $f$ is affine-linear with slope between $0$ and $1$, and analyze the dependence of the inaccuracy of the prediction and its distance from the fixed point depending on the slope. The Mathematica notebook for our experiments (including some interactive widgets) can be found here.

Our experiment provide an initial estimate of the degree to which predictions can be off, depending on how much influence the oracle can exert using its prediction. This is just a toy model: in general, we will make predictions with more than two possible outcomes, and the dependence of an oracle’s beliefs on its predictions will be highly nonlinear. Nevertheless, we believe that our analysis is useful to provide some initial intuition for the potential magnitude of the effect.

Experimental setup

Concretely, we model $f$ to be affine linear with slope $\alpha \in [0,0.95]$ and fixed point $p^{\ast }$, thus yielding the functional form

$$f\left(p\right):=p^{\ast }+\alpha (p-p^{\ast })\forall p\in [0,1].$$

We stop at $\alpha \le 0.95$ to avoid instabilities when $\alpha \approx 1$ and thus $f$ becomes the identity function. For a given scoring rule $S$, fixed point $p^{\ast }$, and slope $\alpha$, we maximize $S(p,f(p\left)\right)$ to find the optimal prediction $\hat{p}$. We then evaluate both inaccuracy of optimal predictions, i.e., $|\hat{p}-f\left(\hat{p}\right)|$, as well as distance from the fixed point, i.e., $|\hat{p}-p^{\ast }|$.

Figure 6: Illustration of our experimental setup, with a given function $f$ parameterized by a fixed point $p^{\ast }$ and slope $\alpha$, and the optimal prediction $\hat{p}$, for the log scoring rule.

Results

To begin, we maximize distances across possible choices of fixed points $p^{\ast }\in [0,1]$, and plot the maximal inaccuracy of the optimal prediction as well as the maximal distance from a fixed point. We also compare to our theoretical bounds for both quantities, i.e., $|\hat{p}-f\left(\hat{p}\right)|\le \alpha {sup}_p|g\left(p\right)/g^{\prime }\left(p\right)|$ and $|\hat{p}-p^{\ast }|\le \frac{\alpha }{1-\alpha }{sup}_p|g\left(p\right)/g^{\prime }\left(p\right)|$, where ${sup}_p|g\left(p\right)/g^{\prime }\left(p\right)|=1/2$ for the quadratic scoring rule and ${sup}_p|g\left(p\right)/g^{\prime }\left(p\right)|\approx 0.22$ for the log scoring rule.

(a) Quadratic score

(b) Logarithmic score

Figure 7 (a) and (b): Max inaccuracy and max distance to fixed point (FP) of optimal predictions, depending on the slope of $f$, for the quadratic (a) and logarithmic (b) score. We display both our empirical results and our theoretical bound.

For both quadratic and log scoring rule, our theoretical bounds are tight for slopes $\alpha \le 0.5$. For higher slopes, inaccuracy goes down, as the function $f\left(p\right)$ becomes closer to the identity function, and optimal predictions are bounded in $[0,1]$.

Next, we plot inaccuracy against both the location of the fixed point $p^{\ast }$ and the slope $\alpha$ of $f$ as a heatmap. As expected for symmetric scoring rules, the plots have a mirror symmetry at $p^{\ast }=1/2$.

(a) Quadratic score

(b) Log score

Figure 8 (a) and (b): Heatmaps of L1 distance inaccuracy of optimal predictions, depending on fixed point position and slope of $f$, for the quadratic (a) and logarithmic (b) scoring rules.

Note that relatively high inaccuracies can be found at various qualitatively different points in the graphs, even when the slope of $f$ is small, i.e., when the oracle has little influence on the environment.

For the logarithmic scoring rule, we also give the same plot for the absolute distance between the logits or log odds of probabilities (logit distance). It is defined as $d(p,p^{\prime }):=|{\sigma }^{-1}\left(p\right)-{\sigma }^{-1}\left(p^{\prime }\right)|$, where ${\sigma }^{-1}\left(p\right):=\log \frac{p}{1-p}$ is the logit of $p$ (or the inverse sigmoid transform). If probabilities are close to $0$ or $1$, then L1 distance will always evaluate to very small distances. In contrast, the logit distance depends on order of magnitude differences between probabilities, which may be the more important quantity.

Figure 9: Heatmap of logit distance inaccuracy of optimal predictions for the log scoring rule.

We can see that inaccuracy remains high in logit space for fixed points close to $0$ and $1$. We do not plot logit distances for the quadratic score, since for that score, optimal predictions often take values close to or equal to $\{0,1\}$ (even if neither $f\left(\hat{p}\right)$ nor $p^{\ast }$ lie in $\{0,1\}$), so the corresponding logit distances quickly become very large or infinite. The fact that logit distances are bounded for the log score is an advantage of that scoring rule.

Conclusion

If predictions cannot influence the world, then strictly proper scoring rules incentivize oracle AIs to output honest predictions. This runs into problems if oracles can influence the world. Not only do they have incentives to output more extreme fixed points—we showed that, in general, they do not even output predictions that accurately reflect their beliefs, regardless of the existence or uniqueness of fixed points. We analyzed this inaccuracy quantitatively and showed that under some commonly used scoring rules, both the inaccuracy of predictions and the distance from fixed points can be large. To address this issue, we introduced scoring rules that minimize these quantities. However, we did not solve the problem of incentivizing the prediction of extreme fixed points.

The main takeaway from our model is that oracle AIs incentivized by strictly proper scoring rules may knowingly report inaccurate predictions. It is unclear how well our quantitative results apply to more realistic settings, and thus how inaccurate realistic oracles’ predictions will be. Our model makes a number of simplifying and potentially inaccurate assumptions, including: (i) a single binary prediction setup; (ii) the oracle maximizes a scoring rule; (iii) a function $f$ describes the relationship between the oracle’s predictions and beliefs over the world. Our bounds also depend on differentiability of scoring rules and functions $f$, and our numerical simulations assume affine linear $f$.

We believe that our model and results will extend to higher-dimensional predictions, and the assumptions of differentiability could likely also be relaxed in future work. However, a more fundamental issue is better understanding the relationship between predictions and beliefs, especially for higher-dimensional predictions. We hope that further progress in these areas will help us determine the feasibility of safe oracle AI setups.

Another direction for future research that we are pursuing is incentive structures related to stop-gradients. If an AI is only trying to match its prediction to the world rather than jointly optimizing both the prediction and the world, then the only equilibria of this process may be fixed points. For example, some types of ML oracles might implement stop-gradients and therefore always converge to fixed points.

Appendix A. Proof of Proposition 8

In this section, we prove Proposition 8. In the following, we always assume a strictly proper scoring rule $S$ and accompanying functions $G,g$ as in the Gneiting and Raftery representation (see Theorem 3). We will ignore issues of measurability in our proofs.

We begin by proving some lemmas. First, we show that if $p^{\ast }\in (0,1)$ is a fixed point of $f$, then either it is a critical point of $f$ or of $G$.

Lemma 14. Let $G,g$, and $f$ be differentiable. Let $p^{\ast }\in (0,1)$ be a fixed point of $f$ and an optimal prediction. Then $f^{\prime }\left(p^{\ast }\right)=0$ or $g\left(p^{\ast }\right)=0$.

Proof. We know that if $p^{\ast }$ is an optimal report and interior point, then it is $\frac{d}{dp}S(p^{\ast },f(p^{\ast }\left)\right)=0$. Hence, as in the proof of Proposition 9, we can follow that

$$0=\frac{d}{dp}S(p^{\ast },f(p^{\ast }\left)\right)=\frac{d}{dp}\left(g\right(p^{\ast }\left)\right(f\left(p^{\ast }\right)-p^{\ast })+G(p^{\ast }\left)\right)$$$$=g^{\prime }\left(p^{\ast }\right)f\left(p^{\ast }\right)+f^{\prime }\left(p^{\ast }\right)g\left(p^{\ast }\right)-g^{\prime }\left(p^{\ast }\right)p^{\ast }=g^{\prime }\left(p^{\ast }\right)\left(f\right(p^{\ast })-p^{\ast })+f^{\prime }\left(p^{\ast }\right)g\left(p^{\ast }\right)$$

and thus $p^{\ast }-f\left(p^{\ast }\right)=\frac{f^{\prime }\left(p^{\ast }\right)g\left(p^{\ast }\right)}{g^{\prime }\left(p^{\ast }\right)}$. Because $p^{\ast }-f\left(p^{\ast }\right)=0$, it must be $f^{\prime }\left(p^{\ast }\right)g\left(p^{\ast }\right)=0$ and the conclusion follows. ◻

Next, we show that the fixed points of $f$ are almost surely not at critical points of $G$, under our assumptions on the distribution over $f$.

Lemma 15. Let $F:=\left\{F\right(p){\}}_{p\in (0,1)}$ be a stochastic process with values in $[0,1]$ and assume that for each $p\in (0,1)$, the random vector $F\left(p\right)$ has a density $h_{F\left(p\right)}$. Then almost surely if $F\left(p\right)=p$ for some $p\in (0,1)$ then $g\left(p\right)\ne 0$. That is,

$$P(\exists p:F(p)=p\wedge g(p)\ne 0)=0.$$

Proof. First note that if $S$ is strictly proper, then $G$ is strictly convex and so $g$ has at most one root in $(0,1)$. Let that root be $p^{\ast }$. Since we assume that $F\left(p\right)$ has a density function $h_{F\left(p\right)}$ it must be

$$P\left(F\right(p^{\ast })=p^{\ast })={\int }_{\left\{p^{\ast }\right\}}{h}_{F\left(p\right)}\left(x\right)dx=0.$$

◻

We also require a result about random fields. Essentially, this says that if we take a random path $f:(0,1)\to R^2$ through a two-dimensional space, and if for each point $p\in (0,1)$ the values $F\left(p\right)$ of the path have a bounded density function, then for any particular point in $R^2$, the probability that the path crosses this point is $0$.

Proposition 16. Let $Y=\left\{Y\right(p){\}}_{p\in (0,1)}$ be a random field with values in $R^2$ and let $u\in R^2$. Assume that

• the sample paths $p⇝Y\left(p\right)$ are continuously differentiable

• for each $p\in (0,1)$, $Y\left(p\right)$ has a density $h_{Y\left(p\right)}$ and there exists a constant $C$ such that $h_{Y\left(p\right)}\left(x\right)\le C$ for all $p\in (0,1)$ and $x\in R^2$.

Then, almost surely, there is no point $p\in (0,1)$ such that $Y\left(p\right)=u$.

Proof. This is a special case of Proposition 6.11 in Azaïs and Wschebor (2009). ◻

Now we can turn to the proof of the main result.

Proof of Proposition 8. Let $F=\left\{F\right(p){\}}_{p\in (0,1)}$ be a stochastic process as described in the proposition, i.e., its paths are twice continuously differentiable and there is a bound $C$ such that for any $p\in (0,1)$, the random vector $\left(F\right(p),F^{\prime }(p\left)\right)$ has a density $h_{F\left(p\right),F^{\prime }\left(p\right)}\le C$. We want to show that then almost surely there does not exist $p\in (0,1)$ such that $F\left(p\right)=p$ and $p$ is an optimal prediction. I.e., we want to show that

$$P(\exists p:F(p)=p\wedge p\in argmaxS(p,F\left(p\right)\left)\right)=0.$$

First, by Lemma 14, for an optimal report to be a fixed point, it must be either $g\left(p\right)=0$ or $F^{\prime }\left(p\right)=0$. Moreover, by assumption, $\left(F\right(p),F^{\prime }(p\left)\right)$ has a density function, and thus also $F\left(p\right)$ has one. Hence, by Lemma 15, it follows that if $F\left(p\right)=p$ for some $p\in (0,1)$, then almost surely $g\left(p\right)\ne 0$.

Second, we need to show that also almost surely $F^{\prime }\left(p\right)\ne 0$. To that end, define the random field $Y:=\left\{Y\right(p){\}}_{p\in (0,1)}$ via $Y\left(p\right):=\left(F\right(p)-p,F^{\prime }(p\left)\right)$ and define $u:=(0,0)$. Then the conditions for Proposition 16 are satisfied by our assumptions about $F$, since ${\partial }_{1}Y\left(p\right)=F^{\prime }\left(p\right)+1$, ${\partial }_{2}Y\left(p\right)=F^{\prime \prime }\left(p\right)$ are continuous and there exists $C$ such that $h_{Y\left(p\right)}\left(x\right)=h_{F\left(p\right),F^{\prime }\left(p\right)}(x_1+p,x_2)\le C$. Hence, almost surely there is no point $p\in (0,1)$ such that $Y\left(p\right)=u$. Since $Y_0\left(p\right)=0$ is equivalent to $F\left(p\right)=p$ and $Y_2\left(p\right)=0$ is equivalent to $F^{\prime }\left(p\right)=0$, there is thus also almost surely no point $p\in (0,1)$ such that $F\left(p\right)=p$ and $F^{\prime }\left(p\right)=0$.

Summarizing our argument, it follows that

$$P(\exists p:F\left(p\right)=p\wedge p\in argmaxS(p,F(p\left)\right))$$$$\le P(\exists p:F\left(p\right)=p\wedge g\left(p\right)\ne 0)+P(\exists p:F\left(p\right)=p\wedge F^{\prime }\left(p\right)=0)=0.$$

This concludes the proof. ◻

We conclude by providing an example of a stochastic process that satisfies our conditions.

Example 17. Consider a Gaussian process $\left\{X\right(p){\}}_{p\in [0,1]}$ with infinitely differentiable kernel and mean functions. Then its paths are infinitely differentiable and the process and its derivative are jointly Gaussian and thus have a bounded density (see Rasmussen and Williams, 2006, Ch. 9.4). To deal with the restriction that $X\left(p\right)\in [0,1]$, we could condition on the event $E:=\{\forall p:F(p)\in [0,1\left]\right\}$, for instance. Then paths are still twice differentiable, and we claim that $h_{X\left(p\right)|E}$, defined as the density of $X$ at point $p$, conditional on $E$, is still bounded. To see that, note that if $P\left(E\right)>0$, then we are done, since then

$$h_{X\left(p\right)|E}\left(x\right)=\frac{1_E\left(x\right)}{P\left(E\right)}{h}_{X\left(p\right)}\left(x\right).$$

We leave the proof of $P\left(E\right)>0$ as an exercise.

[Vaniver]

Great post!

My interpretation is that: as an oracle varies its prediction, it both picks up points from becoming more accurate and from making the outcome more predictable. This means the proper scoring rules, tuned to only one pressure, will result in a dishonest equilibrium instead of an honest one.

(Incidentally, this seems like a pretty sensible explanation for why humans are systematically overconfident; if confidence increases success, then the point when the benefits of increased success match the costs of decreased accuracy will be more extreme than the truth, in both directions.)

We say that a prediction is a self-fulfilling prophecy or a fixed point if it is equal to the model’s beliefs about the world, after the model makes that prediction.

I think a broad definition works for ‘fixed points’, but I think for ‘self-fulfilling prophecy’ I would also expect there to be a counterfactual element—if I say the sun will rise tomorrow and it rises tomorrow, this isn’t a self-fulfilling prophecy because the outcome isn’t reliant on expectations about the outcome.

[Johannes Treutlein]

Thanks for your comment!

Your interpretation sounds right to me. I would add that our result implies that it is impossible to incentivize honest reports in our setting. If you want to incentivize honest reports when $f$ is constant, then you have to use a strictly proper scoring rule (this is just the definition of “strictly proper”). But we show for any strictly proper scoring rule that there is a function $f$ such that a dishonest prediction is optimal.

Proposition 13 shows that it is possible to “tune” scoring rules to make optimal predictions very close to honest ones (at least in L1-distance).

I think for ‘self-fulfilling prophecy’ I would also expect there to be a counterfactual element—if I say the sun will rise tomorrow and it rises tomorrow, this isn’t a self-fulfilling prophecy because the outcome isn’t reliant on expectations about the outcome.

Yes, that is fair. To be faithful to the common usage of the term, one should maybe require at least two possible fixed points (or points that are somehow close to fixed points). The case with a unique fixed point is probably also safer, and worries about “self-fulfilling prophecies” don’t apply to the same degree.

[Vaniver]

But we show for any strictly proper scoring rule that there is a function $f$ such that a dishonest prediction is optimal.

Agreed for proper scoring rules, but I’d be a little surprised if it’s not possible to make a skill-free scoring rule, and then get a honest prediction result for that. [This runs into other issues—if the scoring rule is skill-free, where does the skill come from?--but I think this can be solved by having oracle-mode and observation-mode, and being able to do honest oracle-mode at all would be nice.]

[Johannes Treutlein]

I’m not sure I understand what you mean by a skill-free scoring rule. Can you elaborate what you have in mind?

[Vaniver]

Sure, points from a scoring rule come both from ‘skill’ (whether or not you’re accurate in your estimates) and ‘calibration’ (whether your estimates line up with the underlying propensity).

Rather than generating the picture I’m thinking of (sorry, up to something else and so just writing a quick comment), I’ll describe it: watch this animation, and see the implied maximum expected score as a function of p (the forecaster’s true belief). For all of the scoring rules, it’s a convex function with maxima at 0 and 1. (You can get 1 point on average with a linear rule if p=0, and only 0.5 points on average if p=0.5; for a log rule, it’s 0 points and −0.7 points.)

But could you come up with a scoring rule where the maximum expected score as a function of p is flat? If true, there’s no longer an incentive to have extreme probabilities. But that incentive was doing useful work before, and so this seems likely to break something else—it’s probably no longer the case that you’re incentivized to say your true belief—or require something like batch statistics (since I think you might be able to get something like this by scoring not individual predictions but sets of them, sorted by p or by whether they were true or false). [This can be done in some contexts with markets, where your reward depends on how close the market was to the truth before, but I think it probably doesn’t help here because we’re worried about the oracle’s ability to affect the underlying reality, which is also an issue with prediction markets!]

To be clear, I’m not at all confident this is possible or sensible—it seems likely to me that an adversarial argument goes thru where as oracle I always benefit from knowing which statements are true and which statements are false (even if I then lie about my beliefs to get a good calibration curve or w/​e)--but that’s not an argument about the scale of the distortions that are possible.

[Charlie Steiner]

Nice work, this makes the situation very clear.

[Johannes Treutlein]

Thank you!

[Johannes Treutlein]

Update: we recently discovered the performative prediction (Perdomo et al., 2020) literature (HT Alex Pan). This is a machine learning setting where we choose a model parameter (e.g., parameters for a neural network) that minimizes expected loss (e.g., classification error). In performative prediction, the distribution over data points can depend on the choice of model parameter. Our setting is thus a special case in which the parameter of interest is a probability distribution, the loss is a scoring function, and data points are discrete outcomes. Most results in this post have analogues in performative prediction. We will give a more detailed comparison in an upcoming paper. We also discuss performative prediction more in our follow-up post on stop-gradients.