MichaelStJules

Karma: 573

MichaelStJules Sep 27, 2023, 12:15 AM
1 point
0
in reply to: Anthony DiGiovanni’s comment on: When would AGIs engage in conflict?
Hmm, if A is simulating B with B’s source code, couldn’t the simulated B find out it’s being simulated and lie about its decisions or hide what its actual preferences? Or would its actual preferences be derivable from its weights or code directly without simulation?

MichaelStJules Sep 24, 2023, 4:54 PM
2 points
0
in reply to: Portia’s comment on: Paper: LLMs trained on “A is B” fail to learn “B is A”
I had a similar thought about “A is B” vs “B is A”, but “A is the B” should reverse to “The B is A” and vice versa when the context is held constant and nothing changes the fact, because “is” implies that it’s the present condition and “the” implies uniqueness. However, it might be trained on old and no longer correct writing or that includes quotes about past states of affairs. Some context might still be missing, too, e.g. for “A is the president”, president of what? It would still be a correct inference to say “The president is A” in the same context, at least, and some others, but not all.

Also, the present condition can change quickly, e.g. “The time is 5:21:31 pm EST” and “5:21:31 pm EST is the time” quickly become false, but I think these are rare exceptions in our use of language.

MichaelStJules Sep 22, 2023, 4:13 AM
1 point
0
in reply to: MichaelStJules’s comment on: Impossibility results for unbounded utilities
p.37-38 in Goodsell, 2023 gives a better proposal, which is to clip/truncate the utilities into the range $[- t, t]$ and compare the expected clipped utilities in the limit as $t \to \infty$ . This will still suffer from St Petersburg lottery problems, though.

MichaelStJules Sep 21, 2023, 7:35 AM
1 point
0
on: There are no coherence theorems
Looking at Gustafsson, 2022′s money pumps for completeness, the precaution principles he uses just seem pretty unintuitive to me. The idea seems to be that if you’ll later face a decision situation where you can make a choice that makes you worse off but you can’t make yourself better off by getting there, you should avoid the decision situation, even if it’s entirely under your control to make a choice in that situation that won’t leave you worse off. But, you can just make that choice that won’t leave you worse off later instead of avoiding the situation altogether.
Here’s the forcing money pump:
It seems obvious to me that you can just stick with A all the way through, or switch to B, and neither would violate any of your preferences or be worse than any other option. Gustafsson is saying that would be irrational, it seems because there’s some risk you’ll make the wrong choices. Another kind of response like your policy I can imagine is that unless you have preferences otherwise (i.e. would strictly prefer another accessible option to what you have now), you just stick with the status quo, as the default. This means sticking with A all the eay though, because you’re never offered a strictly better option than it.
Another problem with the precaution principles is that they seem much less plausible when you seriously entertain incompleteness, rather than kind of treat incompleteness like equivalence. He effectively argues that at node 3, you should pick B, because otherwise at node 4, you could end up picking B-, which is worse than B, and there’s no upside. But that basically means claiming that one of the following must hold:
1. you’ll definitely pick B- at 4, or
2. B is better than any strict probabilistic mixture of A and B-.
But both are false in general. 1 is false in general because A is permissible at 4. 2 is false in general because A and B are incomparable and incomparability can be infectious (e.g. MacAskill, 2013), so B can be incomparable with a strict probabilistic mixture of A and B-. It also just seems unintuitive, because the claim is made generally, and so would have to hold no matter how low the probability assigned to B- is, as long it’s positive.
Imagine A is an apple, B is a banana and B- is a slightly worse banana, and I have no preferences between apples and bananas. It would be odd to say that a banana is better than an apple or a tiny probability of a worse banana. This would be like using the tiny risk of a worse banana with the apple to break a tie between the apple and the banana, but there’s no tie to break, because apples and bananas are incomparable.
If A and B were equivalent, then B would indeed very plausibly be better than a strict probabilistic mixture of A and B-. This would follow from Independence, or if A, B and B- are deterministic outcomes, statewise dominance. So, I suspect the intuitions supporting the precaution principles are accidentally treating incomparability like equivalence.
I think a more useful way to think of incomparability is as indeterminancy about which is better. You could consider what happens if you treat A as (possibly infinitely) better than B in one whole treatment of the tree, and consider what happens if you treat B as better than A in a separate treatment, and consider what happens if you treat them as equivalent all the way through (and extend your preference relation to be transitive and continue to satisfy stochastic dominance and independence in each case). If B were better, you’d end up at B, no money pump. If A were better, you’d end up at A, no money pump. If they were equivalent, you’d end up at either (or maybe specifically B, because of precaution), no money pump.

MichaelStJules Sep 21, 2023, 1:02 AM
1 point
0
in reply to: EJT’s comment on: There are no coherence theorems
I think a multi-step decision procedure would be better. Do what your preferences themselves tell you to do and rule out any options you can with them. If there are multiple remaining incomparable options, then apply your original policy to avoid money pumps.

MichaelStJules Sep 20, 2023, 4:28 AM
1 point
0
in reply to: EJT’s comment on: There are no coherence theorems
This also looks like a generalization of stochastic dominance.

MichaelStJules Sep 20, 2023, 2:33 AM
4 points
3
on: There are no coherence theorems
Coming back to this, the policy
if I previously turned down some option X, I will not choose any option that I strictly disprefer to X
seems irrational to me if applied in general. Suppose I offer you $X$ and $Y$ , where both $X$ and $Y$ are random, and $Y$ is ex ante preferable to $X$ , e.g. stochastically dominates $X$ , but has some chance of being worse than $X$ . You pick $Y$ . Then you evaluate $Y$ to get $y$ . However, suppose you get unlucky, and $y$ is worse than $X$ . Suppose further that there’s a souring of $X$ , $X^{-}$ , that’s still preferable to $y$ . Then, I offer you to trade $y$ for $X^{-}$ . It seems irrational to not take $X^{-}$ .
Maybe what you need to do is first evaluate according to your multi-utility function (or stochastic dominance, which I think is a special case) to rule out some options, i.e. to rule out not trading $y$ for $X^{-}$ when the latter is better than the former, and then apply your policy to rule out more options.

MichaelStJules Sep 18, 2023, 7:19 PM
2 points
1
in reply to: MichaelStJules’s comment on: Microdooms averted by working on AI Safety
Also, the estimate of the current number of researchers probably underestimates the number of people (or person-hours) who will work on AI safety. You should probably expect further growth to the number of people working on AI safety, because the topic is getting mainstream coverage and support, Hinton and Bengio have become advocates, and it’s being pushed more in EA (funding, community building, career advice).

However, the FTX collapse is reason to believe there will be less funding going forward.

MichaelStJules Sep 17, 2023, 11:56 PM
13 points
6
on: Microdooms averted by working on AI Safety
Some other possibilities that may be worth considering and can further reduce impact, at least for an individual looking to work on AI safety themself:
1. Some work is net negative and increases the risk of doom or wastes the time and attention of people who could be doing more productive things.
2. Practical limits on the number of people working at a time, e.g. funding, management/supervision capacity. This could mean some people could have much lower probability of making a difference, if them taking a position pushes someone else who would have out from the field, or into (possibly much) less useful work.

MichaelStJules Aug 31, 2023, 9:31 AM
1 point
0
in reply to: Anthony DiGiovanni’s comment on: When would AGIs engage in conflict?
An AGI could give read and copy access to the code being run and the weights directly on the devices from which the AGI is communicating. That could still be a modified copy of the original and more powerful (or with many unmodified copies) AGI, though. So, the other side may need to track all of the copies, maybe even offline ones that would go online on some trigger or at some date.

Also, giving read and copy access could be dangerous to the AGI if it doesn’t have copies elsewhere.

MichaelStJules Aug 27, 2023, 6:58 PM
2 points
0
on: Eliezer Yudkowsky Is Frequently, Confidently, Egregiously Wrong
Some other discussion of his views on (animal) consciousness here (and in the comments).

MichaelStJules Aug 27, 2023, 6:53 PM
1 point
−8
on: Eliezer Yudkowsky Is Frequently, Confidently, Egregiously Wrong
My understanding from Eliezer’s writing is that he’s an illusionist (and/or a higher-order theorist) about consciousness. However, illusionism (and higher-order theories) are compatible with mammals and birds, at least, being conscious. It depends on the specifics.
I’m also an illusionist about consciousness and very sympathetic to the idea that some kinds of higher-order processes are required, but I do think mammals and birds, at least, are very probably conscious, and subject to consciousness illusions. My understanding is that Humphrey (Humphrey, 2022, Humphrey, 2023a, Humphrey, 2023b, Humphrey, 2017, Romeo, 2023, Humphrey, 2006, Humphrey, 2011) and Muehlhauser (2017) (a report for Open Phil, but representing his own views) would say the same. Furthermore, I think the standard interpretation of illusionism doesn’t require consciousness illusions or higher-order processes in conscious subjects at all, and instead a system is conscious if connecting a sufficiently sophisticated introspective system to it the right way would lead to consciousness illusions, and this interpretation would plausibly attribute consciousness more widely, possibly quite widely (Blackmore, 2016 (available submitted draft), Frankish, 2020, Frankish, 2021, Frankish, 2023, Graziano, 2021, Dung, 2022).
If I recall correctly, Eliezer seemed to give substantial weight to relatively sophisticated self- and other-modelling, like cognitive empathy and passing the mirror test. Few animals seem to pass the mirror test, so that would be reason for skepticism.
However, maybe they’re just not smart enough to infer that the reflection is theirs, or they don’t rely enough on sight. Or, they may recognize themselves in other ways or to at least limited degrees. Dogs can remember what actions they’ve spontaneously taken (Fugazza et al., 2020) and recognize their own bodies as obstacles (Lenkei, 2021), and grey wolves show signs of self-recognition via a scent mirror test (Cazzolla Gatti et al., 2021, layman summary in Mates, 2021). Pigeons can discriminate themselves from conspecifics with mirrors, even if they don’t recognize the reflections as themselves (Wittek et al., 2021, Toda and Watanabe, 2008). Mice are subject to the rubber tail illusion and so probably have a sense of body ownership (Wada et al., 2016).
Furthermore, Carey and Fry (1995) show that pigs generalize the discrimination between non-anxiety states and drug-induced anxiety to non-anxiety and anxiety in general, in this case by pressing one lever repeatedly with anxiety, and alternating between two levers without anxiety (the levers gave food rewards, but only if they pressed them according to the condition). Similar experiments were performed on rodents, as discussed in Sánchez-Suárez, 2016, in section 4.d., starting on p.81. Rats generalized from hangover to morphine withdrawal and jetlag, and from high doses of cocaine to movement restriction, from an anxiety-inducing drug to aggressive defeat and predator cues. Of course, anxiety has physical symptoms, so maybe this is what they’re discriminating, not the negative affect.
There are also of course many non-illusionist theories of consciousness that attribute consciousness more widely that are defended (although I’m personally not sympathetic, unless they’re illusionist-compatible), and theory-neutral or theory-light approaches. On theory-neutral and theory-light approaches, see Low, 2012, Sneddon et al., 2014, Le Neindre et al., 2016, Rethink Priorities, 2019, Birch, 2020, Birch et al., 2022, Mason and Lavery, 2022, generally giving more weight to the more recent work.

MichaelStJules Aug 2, 2023, 6:55 AM
1 point
0
on: Is Deontological AI Safe? [Feedback Draft]
I think it’s worth pointing out that from the POV of such ethical views, non-extinction could be an existential risk relative to extinction, or otherwise not that important (see also the asymmetric views in Thomas, 2022). If we assign some credence to those views, then we might instead focus more of our resources on avoiding harms without also (significantly) increasing extinction risks, perhaps especially reducing s-risks or the torture of sentient beings.
Furthermore, the more we reduce the risks of such harms, the less prone deontological (and other morally asymmetric) AI could be to aim for extinction.

MichaelStJules Aug 2, 2023, 5:55 AM
2 points
0
on: There are no coherence theorems
The arguments typically require agents to make decisions independently of the parts of the decision tree in the past (or that are otherwise no longer accessible, in case they were ruled out). But an agent need not do that. An agent can always avoid getting money pumped by just following the policy of never picking an option that completes a money pump (or the policy of never making any trades, say). They can even do this with preference cycles.
Does this mean money pump arguments don’t tell us anything? Such a policy may have other costs that an agent would want to avoid, if following their preferences locally would otherwise lead to getting money pumped (e.g. as Gustafsson (2022) argues in section 7 Against Resolute Choice), but how important could depend on those costs, including how frequently they expect to incur them, as well as the costs of changing their preferences to satisfy rationality axioms. It seems bad to pick options you’ll foreseeable regret. However, changing your preferences to fit some proposed rationality requirements also seems foreseeably regrettable in another way: you have to give up things you care about or some ways you care about them. And that can be worse than your other options for avoiding money pumps, or even, sometimes, getting money pumped.
Furthermore, agents plausibly sometimes need to make commitments that would bind them in the future, even if they’d like to change their minds later, in order to win in Parfit’s hitchhiker, say.
Similarly, if instead of money pumps, an agent should just avoid any lottery that’s worse than (or strictly statewise dominated by, or strictly stochastically dominated by, under some suitable generalization^[1]) another they could have guaranteed, it’s not clear that’s a requirement of rationality, either. If I prefer A<B<C<A, then it doesn’t seem more regrettable if I pick one option than if I pick another (knowing nothing else), even though no matter what option I pick, it seems regrettable that I didn’t pick another. Choosing foreseeably regrettable options seems bad, but if every option is (foreseeably) regrettable in some way, and there’s no least of the evils, then is it actually irrational?
Furthermore, if a superintelligence is really good at forecasting, then maybe we should expect it to have substantial knowledge of the decision tree in advance, and to typically be able to steer clear of situations where it might face a money pump or other dilemmas, and if it ever does get money pumped, the costs of all money pumps would be relatively small compared to its gains.
1. ^
  $X$ (strictly) stochastically dominates $Y$ iff there’s a “probability rearrangement” of $X$ , $X^{'}$ , such that $X^{'}$ (strictly) statewise dominates $Y$ .

MichaelStJules Aug 2, 2023, 4:36 AM
1 point
0
in reply to: Max H’s comment on: There are no coherence theorems
See also EJT’s comment here (and the rest of the thread). You’d just pick any one of the utility functions. You can also probably drop continuity for something weaker, as I point out in my reply there.

MichaelStJules Aug 2, 2023, 1:44 AM
2 points
0
in reply to: EJT’s comment on: There are no coherence theorems
This is cool. I don’t think violations of continuity are also in general exploitable, but I’d guess you should also be able to replace continuity with something weaker from Russell and Isaacs, 2020, just enough to rule out St. Petersburg-like lotteries, specifically any one of Countable Independence (which can also replace independence), the Extended Outcome Principle (which can also replace independence) or Limitedness, and then replace the real-valued utility functions with utility functions representable by “lexicographically ordered ordinal sequences of bounded real utilities”.

MichaelStJules Aug 1, 2023, 6:58 PM
4 points
0
on: There are no coherence theorems
EDIT: Looks like a similar point made here.
I wonder if we can “extend” utility maximization representation theorems to drop Completeness. There’s already an extension to drop Continuity by using an ordinal-indexed vector (sequence) of real numbers, with entries sorted lexicographically (“lexicographically ordered ordinal sequences of bounded real utilities”, Russell and Isaacs, 2020). If we drop Completeness, maybe we can still represent the order with a vector of independent but incomparable dimensions across which it must respect ex ante Pareto efficiency (and each of those dimensions could also be split into an ordinal-indexed vector of real numbers with entries sorted lexicographically, if we’re also dropping Continuity)?
These also give us examples of somewhat natural/non-crazy orders that are consistent with dropping Completeness. I’ve seen people (including some economists) claim interpersonal utility comparisons are impossible and that we should only seek Pareto efficiency across people and not worry about tradeoffs between people. (Said Achmiz already pointed this and other examples out.)
Intuitively, the dimensions don’t actually need to be totally independent. For example, the order could be symmetric/anonymous/impartial between some dimensions, i.e. swapping values between these dimensions gives indifference. You could also have some strict preferences over some large tradeoffs between dimensions, but not small tradeoffs. Or even, maybe you want more apples and more oranges without tradeoffs between them, but also prefer more bananas to more apples and more bananas to more oranges. Or, a parent, having to give a gift to one of their children, may strictly prefer randomly choosing over picking one child to give it to, and find each nonrandom option incomparable to one another (although this may have problems when they find out which one they will give to, and then give them the option to rerandomize again; they might never actually choose).
Maybe you could still represent all of this with a large number of, possibly infinitely many, real-valued utility functions (or utility functions representable by “lexicographically ordered ordinal sequences of bounded real utilities”) instead. So, the correct representation could still be something like a (possibly infinite) set of utility functions (each possibly a “lexicographically ordered ordinal sequences of bounded real” utility functions), across which you must respect ex ante Pareto efficiency. This would be similar to the maximality rule over your representor/credal set/credal committee for imprecise credences (Mogensen, 2019).
Then, just combine this with your policy “if I previously turned down some option X, I will not choose any option that I strictly disprefer to X”, where strictly disprefer is understood to mean ex ante Pareto dominated.
But now this seems like a coherence theorem, just with a broader interpretation of “expected utility”.
To be clear, I don’t know if this “theorem” is true at all.
Possibly also related: McCarthy et al., 2020 have a utilitarian representation theorem that’s consistent with “the rejection of all of the expected utility axioms, completeness, continuity, and independence, at both the individual and social levels”. However, it’s not a real-valued representation. It reduces lotteries over a group of people to a lottery over outcomes for one person, as the probabilistic mixture of each separate person’s lottery into one lottery.

MichaelStJules Jul 30, 2023, 8:24 PM
1 point
0
on: Consciousness as a conflationary alliance term
There’s a recent survey of the general public’s answers to “In your own words, what is consciousness?”
Consciousness: In your own words by Michael Graziano and Isaac Ray Christian, 2022
Abstract:
Surprisingly little is known about how the general public understands consciousness, yet information on common intuitions is crucial to discussions and theories of consciousness. We asked 202 members of the general public, “In your own words, what is consciousness?” and analyzed the frequencies with which different perspectives on consciousness were represented. Almost all people (89%) described consciousness as fundamentally receptive – possessing, knowing, perceiving, being aware, or experiencing. In contrast, the perspective that consciousness is agentic (actively making decisions, driving output, or controlling behavior) appeared in only 33% of responses. Consciousness as a social phenomenon was represented by 24% of people. Consciousness as being awake or alert was mentioned by 19%. Consciousness as mystical, transcending the physical world, was mentioned by only 10%. Consciousness in relation to memory was mentioned by 6%. Consciousness as an inner voice or inner being – the homunculus view – was represented by 5%. Finally, only three people (1.5%) mentioned a specific, scholarly theory about consciousness, suggesting that we successfully sampled the opinions of the general public rather than capturing an academic construct. We found little difference between men and women, young and old, or US and non-US participants, except for one possible generation shift. Young, non-US participants were more likely to associate consciousness with moral decision-making. These findings show a snapshot of the public understanding of consciousness – a network of associated concepts, represented at varying strengths, such that some are more likely to emerge when people are asked an open-ended question about it.

MichaelStJules Jul 6, 2023, 11:47 PM
5 points
0
in reply to: Ege Erdil’s comment on: When do “brains beat brawn” in Chess? An experiment
I think it’s more illustrative than anything, and a response to Robert Miles using chess against Magnus Carlsen as an analogy for humans vs AGI. The point is that a large enough material advantage can help someone win against a far smarter opponent. Somewhat more generally, I think arguments for AI risk often put intelligence on a pedestal, without addressing its limitations, including the physical resource disadvantages AGIs will plausibly face.
I agree that the specifics of chess probably aren’t that helpful for informing AI risk estimates, and that a better tuned engine could have done better against the author.
Maybe better experiments to run would be playing real-time strategy games against a far smarter but materially disadvatanged AI, but this would also limit the space of actions an AI could take relative to the real world.

MichaelStJules Jul 4, 2023, 8:41 PM
3 points
0
in reply to: Cody Miner’s comment on: When do “brains beat brawn” in Chess? An experiment
For my 2nd paragraph, I meant that the experiment would underestimate the required resource gap. Being down exactly by a queen at the start of a game is not as bad as being down exactly by a queen later into the game when there are fewer pieces overall left, because that’s a larger relative gap in resources.