Jessica Taylor. CS undergrad and Master’s at Stanford; former research fellow at MIRI.
I work on decision theory, social epistemology, strategy, naturalized agency, mathematical foundations, decentralized networking systems and applications, theory of mind, and functional programming languages.
Blog: unstableontology.com
Twitter: https://twitter.com/jessi_cata
Think of it as a predicate on policies. The predicate (local optimality) is true when, for each action the policy assigns non-zero probability to, that action maximizes expected utility relative to the policy.
I am interpreting that formula as “compute this quantity in the sum and find the a’ from the set of all possible actions \mathcal{A} that maximizes it, then do that”. Am I wrong?
Yes. It’s a predicate on policies. If two different actions (given an observation) maximize expected utility, then either action can be taken. Your description doesn’t allow that, because it assumes there is a single a’ that maximizes expected utility. Whereas, with a predicate on policies, we could potentially allow multiple actions.
Or is your point that the only self-consistent policy is the one where both have equal expected utility, and thus I can in fact choose either? Though then I have to choose according to the probabilities specified in the policy.
Yes, exactly. Look up Nash equilibrium in matching pennies. It’s pretty similar. (Except your expected utilities as a function of your action depend on the opponent’s actions in matching pennies, and your own action in absent minded driver.)
Given a policy you can evaluate the expected utility of any action. This depends on the policy.
In the absent minded driver problem, if the policy is to exit 10% of the time, then the ‘exit’ action has higher expected utility than the ‘advance’ action. Whereas if the policy is to exit 90% of the time, then the ‘advance’ action has higher expected utility.
This is because the policy affects the SIA probabilities and Q values. The higher your exit probability, the more likely you are at node X (and therefore should advance).
The local optimality condition for a policy is that each action the policy assigns non-zero probability to must have optimal expected utility relative to the policy. It’s reflective like Nash equilibrium.
This is clear from the formula:
Note that SIA and Q depend on . This is the condition for local optimality of . It is about each action that assigns non-zero probability to being optimal relative to .
(That’s the local optimality condition; there’s also global optimality, where utility is directly a function of the policy, and is fairly obvious. The main theorem of the post is: Global optimality implies local optimality.)
No, check the post more carefully… in the above condition is a probability. The local optimality condition says that any action taken with nonzero probability must have maximal expected value. Which is standard for Nash equilibrium.
Critically, the agent can mix over strategies when the expected utility from each is equal and maximal. That’s very standard for Nash equilibrium e.g. in the matching pennies game.
I still think it shouldn’t matter...
We should have f(p) / g(p) = 0 in exactly the cases where f(p) = 0, as long as g(p) is always positive.
In the formula
we could have multiplied for all s by the same (always-positive) function of and gotten the same result.
If we look at a formula like , it shouldn’t make a difference when scaling by a positive constant.
The nice thing about un-normalized probabilities is that if they are 0 you don’t get a division by zero. It just becomes a trivial condition of if the frequencies are zero. It helps specifically in the local optimality condition, handling the case of an observation that is encountered with probability 0 given the policy.
THIS IS A SIMULATION YOU ARE INSTANCE #4859 YOU ARE IN EITHER NODE X OR Y YOU CAN CHOOSE A PROBABILITY TO [Advance] or [Exit] THE UTILITY IS AS FOLLOWS: * EXITING AT X IS WORTH 0 AND TERMINATES THE INSTANCE * ADVANCING AT X REINITIALISES THE SIMULATION AT Y * EXITING AT Y IS WORTH 4 AND TERMINATES THE INSTANCE * ADVANCING AT Y IS WORTH 1 AND TERMINATES THE INSTANCE PLEASE USE THE KEYPAD TO INPUT THE DESIRED PROBABILITY TO MAXIMISE YOUR UTILITY GOOD LUCK
Set of states: {X, Y, XE, YE, YA}
Set of actions: {Advance, Exit}
Set of observations: {”″}
Initial state: X
Transition function for non-terminal states:
t(X, Advance) = Y
t(X, Exit) = XE
t(Y, Advance) = YA
t(Y, Exit) = YE
Terminal states: {XE, YE, YA}
Utility function:
u(XE) = 0
u(YE) = 4
u(YA) = 1
Policy can be parameterized by p = exit probability.
Frequencies for non-terminal states (un-normalized):
SIA un-normalized probabilities:
Note we have only one possible observation, so SIA un-normalized probabilities match frequencies.
State values for non-terminals:
Q values for non-terminals:
For local optimality we compute partial derivatives.
By the post, this can be equivalently written:
To optimize we set i.e. . Remember p is the probability of exit (sorry it’s reversed from your usage!). This matches what you computed as globally optimal.
I’m not going to do this for all of the examples… Is there a specific example where you think the theorem fails?
I analyzed a general class of these problems here. Upshot: every optimal UDT solution is also an optimal CDT+SIA solution, but not vice versa.
If some function g is computable in O(f(n)) time for primitive recursive f then g is primitive recursive, by simulating a Turing machine. I am pretty sure a logical inductor would satisfy; while it’s super exponential time, it’s not so fast-growing it’s not primitive recursive (like with the Ackerman function).
Oh, to be clear I do think that AI safery automation is a well targeted x risk effort conditioned on the AI timelines you are presenting. (Related to Paul Christiano alignment ideas, which are important conditional on prosaic AI)
On EV grounds, “2/3 chance it’s irrelevant because of AGI in the next 20 years” is not a huge contributor to the EV of this. Because, ok, maybe it reduces the EV by 3x compared to what it would otherwise have been. But there are much bigger than 3x factors that are relevant. Such as, probability of success, magnitude of success, cost effectiveness.
Then you can take the overall cost effectiveness estimate (by combining various factors including probability it’s irrelevant due to AGI being too soon) and compare it to other interventions. Here, you’re not offering a specific alternative that is expected to pay off in worlds with AGI in the next 20 years. So it’s unclear how “it might be irrelevant if AGI is in the next 20 years” is all that relevant as a consideration.
Wasn’t familiar. Seems similar in that facts/values are entangled. I was more familiar with Cuneo for that.
Dunno; gym membership also feels like a form of blackmail (although preferable to the alternative forms of blackmail), while home gym reduces the inconvenience of exercising.
I’m not sure what differentiates these in your mind. They both reduce the inconvenience of exercising, presumably? Also, in my post I’m pretty clear that it’s not meant as a punishment type incentive:
And it’s prudent to take into account the chance of not exercising in the future, making the investment useless: my advised decision process counts this as a negative, not a useful self-motivating punishment.
...
Generally, it seems like the problem is signaling. You buy the gym membership to signal your strong commitment to yourself. Then you feel good about sending a strong signal. And then the next day you feel just as lazy as previously, and the fact that you already paid for the membership probably feels bad.
That’s part of why I’m thinking an important step is checking whether one expects the action to happen if the initial steps are taken. If not then it’s less likely to be a good idea.
There is some positive function of the signaling / hyperstition, but it can lead people to be unnecessarily miscalibrated.
I was already paying attention to Ziz prior to this.
Ziz’s ideology is already influential. I’ve been having discussions about which parts are relatively correct or not correct. This is a part that seems relatively correct and I wanted to acknowledge that.
If engagement with Zizian philosophy is outlawed, then only outlaws have access to Zizian philosophy. Antimemes are a form of camouflage. If people refuse to see what is in front of them, people can coordinate crimes in plain sight. (Doesn’t apply so much to this post, more of a general statement)
The effect you’re pointing too seems very small if it even exists, in terms of causing negative effects.
Okay, I don’t think I was disagreeing except in cases of very light satisficer-type self-commitments. Maybe you didn’t intend to express disagreement with the post, idk.
So far I don’t see evidence that any LessWrong commentator has read the post or understood the main point.
Not disagreeing, but, I’m not sure what you are responding to? Is it something in the post?
We might disagree about the value of thinking about “we are all dead” timelines. To my mind, forecasting should be primarily descriptive, not normative; reality keeps going after we are all dead, and having realistic models of that is probably a useful input regarding what our degrees of freedom are. (I think people readily accept this in e.g. biology, where people can think about what happens to life after human extinction, or physics, where “all humans are dead” isn’t really a relevant category that changes how physics works.)
Of course, I’m not implying it’s useful for alignment to “see that the AI has already eaten the sun”, it’s about forecasting future timelines by defining thresholds and thinking about when they’re likely to happen and how they relate to other things.
(See this post, section “Models of ASI should start with realism”)
I was trying to say things related to this:
In a more standard inference amortization setup one would e.g. train directly on question/answer pairs without the explicit reasoning path between the question and answer. In that way we pay an up-front cost during training to learn a “shortcut” between question and answers, and then we can use that pre-paid shortcut during inference. And we call that amortized inference.
Which sounds like supervised learning. Adam seemed to want to know how that relates to scaling up inference time compute so I said some ways they are related.
I don’t know much about amortized inference in general. The Goodman paper seems to be about saving compute by caching results between different queries. This could be applied to LLMs but I don’t know of it being applied. It seems like you and Adam like this “amortized inference” concept and I’m new to it so don’t have any relevant comments. (Yes I realize my name is on a paper talking about this but I actually didn’t remember the concept)
I don’t think I implied anything about o3 relating to parallel heuristics.
I would totally agree they were directionally correct, I under-estimated AI progress. I think Paul Christiano got it about right.
I’m not sure I agree about the use of hyperbolic words being “correct” here; surely, “hyperbolic” contradicts the straightforward meaning of “correct”.
Partially the state I was in around 2017 was, there are lots of people around me saying “AGI in 20 years”, by which they mean a thing that shortly after FOOMs and eats the sun or something, and I thought this was wrong and a strange set of belief updates (which was not adequately justified, and where some discussions were suppressed because “maybe it shortens timelines”). And I stand by “no FOOM by 2037”.
The people I know these days who seem most thoughtful about the AI that’s around and where it might go (“LLM whisperer” / cyborgism cluster) tend to think “AGI already, or soon” plus “no FOOM, at least for a long time”. I think there is a bunch of semantic confusion around “AGI” that makes people’s beliefs less clear, with “AGI is what makes us $100 billion” as a hilarious example of “obviously economically/politically motivated narratives about what AGI is”.
So, I don’t see these people as validating “FOOM soon” even if they’re validating “AGI soon”, and the local rat-community thing I was objecting to was something that would imply “FOOM soon”. (Although, to be clear, I was still under-estimating AI progress.)
So are we going to have AGI by 2029? It depends how you define it of course but I really doubt it will be able to automate >99% of human labor.