quila

Karma: 689

suffering-focused-altruist/longtermist, focused on ASI alignment

PMs open (especially for fellow non-extroverts)

my pgp public key, mainly to prevent future LLM impersonation, you can always ask me to sign a dated message. [the private key is currently stored in plaintext within an encrypted drive, so is vulnerable to being read by local programs]:

I have signed no contracts or agreements whose existence I cannot mention.

quila 30 Dec 2024 16:53 UTC
3 points
0
in reply to: Daniel Kokotajlo’s comment on: By default, capital will matter more than ever after AGI
Okay thanks for clarifying! I may have misunderstood your comment. I’m still confused by the existence of the original post with this many upvotes.

quila 30 Dec 2024 16:23 UTC
0 points
0
in reply to: Daniel Kokotajlo’s comment on: By default, capital will matter more than ever after AGI
Why would strong superintelligence coexist with an economy? Wouldn’t an aligned (or unaligned) superintelligence antiquate it all?

quila 30 Dec 2024 13:19 UTC
0 points
0
in reply to: Daniel Kokotajlo’s comment on: By default, capital will matter more than ever after AGI
You’ll probably be able to buy planets post-AGI for the price of houses today
I am confused by the existence of this discourse. Do its participants not believe strong superintelligence is possible?

quila 29 Dec 2024 20:59 UTC
1 point
0
in reply to: notfnofn’s comment on: notfnofn’s Shortform
if both participants are superintelligent and can simulate each other before submitting answers^[1], and if the value on outcomes is something like: loss 0, draw 0.5, win 1, (game never begins 0.5), then i think one of these happens:
- the game ends in a draw as you say
- you collaborate to either win or lose 50% of the time (same EV)
- it fails to begin because you’re both programs that try to simulate the other and this is infinitely recursive / itself non-terminating.
1. ^
  even if the relevant code which describes the ASI’s competitor’s policy is >2N, it’s not stated that the ASI is not able execute code of that length prior to its submission.
  there’s technically an asymmetry where if the competitor’s policy’s code is >2N, then the ASI can’t include it in their submission, but i don’t see how this would effect the outcome

quila 28 Dec 2024 21:45 UTC
6 points
0
on: quila’s Shortform
i’ve wished to have a research buddy who is very knowledgeable about math or theoretical computer science to answer questions or program experiments (given good specification). but:
- idk how to find such a person
- such a person may wish to focus on their own agenda instead
- unless they’re a friend already, idk if i have great evidence that i’d be impactful to support.
so: i could instead do the inverse with someone. i am good at having creative ideas, and i could try to have new ideas about your thing, conditional on me (1) being able to {quickly understand it} and reason about it and (2) not thinking it is doomed.
if you want me to potentially try doing this for your focuses, message me here. (constraints: you must be focused on the ‘hard problems of alignment’, and accept me communicating only with text)
i think that in either starting arrangement, if it worked out well, then our models would eventually overlap in the research-direction-relevant parts and we’d form a kind of superorganism that uses both of our abilities. but it going that well may be rare (i don’t actually know!). the cost/benefit looks good to me.

quila 24 Dec 2024 3:13 UTC
17 points
22
in reply to: Geoffrey Wood’s comment on: What Goes Without Saying
I don’t understand your objection.
I believe that persuasion should happen on merits of arguments, and that trying to activate the social biases of the reader is defecting^[1] from that norm (even if it’s normal writing practice elsewhere).
Looking at the points of view espoused, they seem to be quite positive for their adherents.
There’s no way to ensure this would be only done with positive views, because many authors think their beliefs would be positive to spread.
1. ^
  (by some amount; not a binary)

quila 23 Dec 2024 17:19 UTC
2 points
0
in reply to: Johannes C. Mayer’s comment on: Vegans need to eat just enough Meat—emperically evaluate the minimum ammount of meat that maximizes utility
note the psychological cost to me would be similar to that of eating a part of a human corpse, so while i agree doing personal experiments is generally worth it and doesn’t require existing studies, it is not so obvious for me in this case. the cost may well be a part of, or at least a change to, my metaphorical soul.
will check the linked thread

quila 23 Dec 2024 7:45 UTC
3 points
0
in reply to: Gordon Seidoh Worley’s comment on: Vegans need to eat just enough Meat—emperically evaluate the minimum ammount of meat that maximizes utility
the challenge in a vegan diet is getting enough lysine
huh, i think this may be the first time i’ve heard this, lysine is not mentioned^[1] in examine.com’s vegan nutrition guide (ways to access: 1, 2).
1. ^
  actually, it’s mentioned once in passing in this excerpt:

quila 23 Dec 2024 7:34 UTC
3 points
2
on: Vegans need to eat just enough Meat—emperically evaluate the minimum ammount of meat that maximizes utility
This generates a lot of data on how much of a positive impact [on cognition] eating meat has.
Is there really no data on this already?
Are we not at the point where any effects can be reduced to nutritional content which can also be intaken in vegan ways? After all, things are fundamentally made of things much smaller than the level of analysis of “meat” or “not meat”.
This post is unfortunately not useful to me as the suggestion seems based on the anecdote of someone who was iron deficient, so approximately no evidence in either direction for me, as I already knew of that class of people, and it’s not infeasible to intake iron; for me, a useful version of the post would focus on the above two questions.

quila 22 Dec 2024 18:38 UTC
24 points
4
on: Orienting to 3 year AGI timelines
The government realizes that AI will be decisive for national power and locks down the AGI companies in late 2026. This takes the form of extreme government oversight bordering on nationalization. Progress stays at a similar pace because of the race with other nuclear weapons states
if multiple nuclear states started taking ASI relatively very seriously (i.e. apart from being ignorant of alignment, and being myopically focused on state power instead of utopia/moral good), and started racing, any state behind in that race could threaten to nuke any state which continues to try to bring about ASI. in other words, the current Mutually Assured Destruction can be unilaterally extended to trigger in response to things other than some state firing nukes.
this is (1) a possible out as it would halt large-scale ASI development, and (2) could happen unilaterally by cause of states myopically seeking dominance.
however, if the nuclear states in question just think ASI will be a very good labor automator, then maybe none would be willing to do nuclear war over it, even if it would technically be in the interest of the myopic ‘state power’ goal^[1]. i don’t know. (so maybe (2) needs a minimum of seriousness higher than ‘it will automate lots of labor’ but lower than ‘it is a probable extinction risk’)
1. ^
  “(??? why?)” by which i mean it seems absurd/perplexing (however likely) that people would be so myopic. ‘state i was born in having dominance’ is such an alien goal also.)

quila 21 Dec 2024 18:46 UTC
9 points
0
on: quila’s Shortform
i observe that processes seem to have a tendency towards what i’ll call “surreal equilibria”. [status: trying to put words to a latent concept. may not be legible, feel free to skip. partly ‘writing as if i know the reader will understand’ so i can write about this at all. maybe it will interest some.]
progressively smaller-scale examples:
- it’s probably easiest to imagine this with AI neural nets, procedurally following some adapted policy even as the context changes from the one they grew in. if these systems have an influential, hard to dismantle role, then they themselves become the rules governing the progression of the system for whatever arises next, themselves ceasing to be the actors or components they originally were; yet as they are “intelligent” they still emit the words as if the old world is true; they become simulacra, the automatons keep moving as they were, this is surreal. going out with a whimper.
  - i don’t mean this to be about AI in particular; the A in AI is not fundamental.
- early → late-stage capitalism. early → late-stage democracy.
  - structures which became ingrained as rules of the world. note the difference between “these systems have Naturally Changed from an early to late form” and “these systems became persistent constraints, and new adapted optimizers sprouted within them”.
it looks like i’m trying to describe an iterative pattern of established patterns becoming constraints bearing permanent resemblance to what they were, and new things sprouting up within the new context / constrained world, eventually themselves becoming constraints.^[1]
i also had in mind smaller scale examples.
- a community forms around some goal and decides to moderate and curate itself in some consistent way, hoping this will lead to good outcomes; eventually the community is no longer the thing it set out to be; the original principles became the constraints. (? - not sure how much this really fits)
- a group of internet friends agrees to regularly play a forum game but eventually they’re each just ‘going along with it’, no longer passionate about the game itself. “continuing to meet to do the thing” was a policy and stable meta-pattern that continued beyond its original context. albeit in this case it was an easily disrupted pattern. but for a time it led to a kind of deadness in behavior, me and those friends became surreal?
  - this is possibly a stretch from what i was originally describing. i’m just sampling words from my mind, here, and hoping they correlate to the latent which i wanted to put words to.
this feels related to goodhart, but where goodhart is framed more individually, and this is more like “a learned policy and its original purpose coming apart as a tendency of reality”.
1. ^
  tangential: in this frame physics can be called the ‘first constraint’

quila 21 Dec 2024 6:52 UTC
28 points
25
on: What Goes Without Saying
What Goes Without Saying
There are people I can talk to, where all of the following statements are obvious. They go without saying. We can just “be reasonable” together, with the context taken for granted.
And then there are people who…don’t seem to be on the same page at all.
This is saying, through framing, “If you do not agree with the following, you are unreasonable; you would be among those who do not understand What Goes Without Saying, those ‘who…don’t seem to be on the same page at all.’.” I noticed this caused an internal pressure towards agreeing at first, before even knowing what the post wanted me to agree with.

quila 12 Dec 2024 2:22 UTC
1 point
0
in reply to: KvmanThinking’s comment on: fake alignment solutions????
There are all sorts of “strategies” (turn it off, raise it like a kid, disincentivize changing the environment, use a weaker AI to align it) that people come up with when they’re new to the field of AI safety, but that are ineffective. And their ineffectiveness is only obvious and explainable by people who specifically know how AI behaves.
yep but the first three all fail for the shared reason of “programs will do what they say to do, including in response to your efforts”. (the fourth one, ‘use a weaker AI to align it’, is at least obviously not itself a solution. the weakest form of it, using an LLM to assist an alignment researcher, is possible, and some less weak forms likely are too.)
when i think of other ‘newly heard of alignment’ proposals, like boxing, most of them seem to fail because the proposer doesn’t actually have a model of how this is supposed to work or help in the first place. (the strong version of ‘use ai to align it’ probably fits better here)
(there are some issues which a programmatic model doesn’t automatically make obvious to a human: they must follow from it, but one could fail to see them without making that basic mistake. probable environment hacking and decision theory issues come to mind. i agree that on general priors this is some evidence that there are deeper subjects that would not be noticed even conditional on those researchers approving a solution.)
i guess my next response then would be that some subjects are bounded, and we might notice (if not ‘be able to prove’) such bounds telling us ‘theres not more things beyond what you have already written down’, which would be negative evidence (strength depending on how strongly we’ve identified a bound). (this is more of an intuition, i don’t know how to elaborate this)
(also on what johnswentworth wrote: a similar point i was considering making is that the question is set up in a way that forces you into playing a game of “show how you’d outperform ~~magnus carlsen~~ {those researchers} in ~~chess~~ alignment theory”—for any consideration you can think of, one can respond that those researchers will probably also think of it, which might preclude them from actually approving, which makes the conditional ‘they approve but its wrong’^[1] harder to be true and basically dependent on them instead of object-level properties of alignment.)
i am interested in reading more arguments about the object-level question if you or anyone else has them.
If the solution to alignment were simple, we would have found it by now [...] That there is one simple thing from which comes all of our values, or a simple way to derive such a thing, just seems unlikely.
the pointer to values does not need to be complex (even if the values themselves are)
If the solution to alignment were simple, we would have found it by now
generally: simple things don’t have to be easy to find. the hard part can be locating them within some huge space of possible things. (math (including is use in laws of physics) come to mind?). (and specifically to alignment: i also strongly expect an alignment solution to … {have some set of simple principles from which it can be easily derived (i.e. whether the program itself ends up long)}, but idk if i can legibly explain why. real complexity usually results from stochastic interactions in a process, but “aligned superintelligent agent” is a simply-defined, abstract thing?)
1. ^
  ig you actually wrote ‘they dont notice flaws’, which is ambiguously between ‘they approve’ and ‘they don’t find affirmative failure cases’. and maybe the latter was your intent all along.
  it’s understandable because we do have to refer to humans to call something unintuitive.

quila 11 Dec 2024 4:34 UTC
1 point
0
on: fake alignment solutions????
That for every true alignment solution, there are dozens of fake ones.
Is this something that I should be seriously concerned about?
if you truly believe in a 1-to-dozens ratio between^[1] real and ‘fake’ (endorsed by eliezer and others but unnoticedly flawed) solutions, then yes. in that case, you would naturally favor something like human intelligence augmentation, at least if you thought it had a chance of succeeding greater than p(chance of a solution being proposed which eliezer and others deem correct) × ¹⁄₂₄
I believe that before we stumble on an alignment solution, we will stumble upon an “alignment solution”—something that looks like an alignment solution, but is flawed in some super subtle, complicated way that means that Earth still gets disassembled into compute or whatever, but the flaw is too subtle and complicated for even the brightest humans to spot
i suggest writing why you believe that. in particular, how do you estimate the prominence of ‘unnoticeably flawed’ alignment solutions given they are unnoticeable (to any human)?^[2] where does “for every true alignment solution, there are dozens of fake ones” come from?
A really complicated and esoteric, yet somehow elegant
why does the proposed alignment solution have to be really complicated? overlooked mistakes become more likely as complexity increases, so this premise favors your conclusion.
1. ^
  (out of the ones which might be proposed, to avoid technicalities about infinite or implausible-to-be-thought-of proposals)
2. ^
  (there are ways you could in principle, for example if there were a pattern of the researchers continually noticing they made increasingly unintuitive errors up until the ‘final’ version (where they no longer notice, but presumably this pattern would be noticed); or extrapolation from general principles about {some class of programming that includes alignment} (?) being easy to make hard-to-notice unintuitive mistakes in)

quila 11 Dec 2024 3:16 UTC
4 points
0
in reply to: Fiora from Rosebloom’s comment on: deluks917′s Shortform
good to hear it’s at least transparent enough for you to describe it directly like this. (edit: though the points in dawnlights post seem scarier)

quila 10 Dec 2024 5:27 UTC
18 points
14
in reply to: dawnlightmelody’s comment on: deluks917′s Shortform
she attempted to pressure me to take MDMA under her supervision. I ended up refusing, and she relented; however, she then bragged that because she had relented, I would trust her more and be more likely to take MDMA the next time I saw her.
this seems utterly evil, especially given MDMA is known as an attachment inducing drug.
edit: more generally, it seems tragic for people who are socially-vulnerable and creative to end up paired with adept manipulators.
a simple explanation is that because creativity is (potentially very) useful, vulnerable creative people will be targets for manipulation. but i think there are also dynamics in communities with higher [illegibility-tolerance? esoterism?] which enable this, which i don’t know how to write about. i hope someone tries to write about it.

quila 8 Dec 2024 5:49 UTC
2 points
0
on: A good way to build many air filters on the cheap
upvoted, i think this article would be better with comparison to the recommendations in thomas kwa’s shortform about air filters

quila 6 Dec 2024 14:01 UTC
1 point
0
in reply to: harfe’s comment on: quila’s Shortform
But maybe you only want to “prove” inner alignment and assume that you already have an outer-alignment-goal-function
correct, i’m imagining these being solved separately

quila 6 Dec 2024 13:35 UTC
2 points
0
on: quila’s Shortform
a possible research direction which i don’t know if anyone has explored: what would a training setup which provably creates a (probably^[1]) aligned system look like?
my current intuition, which is not good evidence here beyond elevating the idea from noise, is that such a training setup might somehow leverage how the training data and {subsequent-agent’s perceptions/evidence stream} are sampled from the same world, albeit with different sampling procedures. for example, the training program could intake both a dataset and an outer-alignment-goal-function, and select for prediction of the dataset (to build up ability) while also doing something else to the AI-in-training; i have no idea what that something else would look like (and it seems like most of this problem).
has this been thought about before? is this feasible? why or why not?
(i can clarify if any part of this is not clear.)
(background motivator: in case there is no finite-length general purpose search algorithm^[2], alignment may have to be of trained systems / learners)
1. ^
  (because in principle, it’s possible to get unlucky with sampling for the dataset. compare: it’s possible for an unlucky sequence of evidence to cause an agent to take actions which are counter to its goal.)
2. ^
  by which i mean a program capable of finding something which meets any given criteria met by at least one thing (or writing ‘undecidable’ in self-referential edge cases)

quila 2 Dec 2024 12:53 UTC
2 points
0
on: quila’s Shortform
a moral intuition i have: to avoid culturally/conformistly-motivated cognition, it’s useful to ask:
if we were starting over, new to the world but with all the technology we have now, would we recreate this practice?
example: we start and out and there’s us, and these innocent fluffy creatures that can’t talk to us, but they can be our friends. we’re just learning about them for the first time. would we, at some point, spontaneously choose to kill them and eat their bodies, despite us having plant-based foods, supplements, vegan-assuming nutrition guides, etc? to me, the answer seems obviously not. the idea would not even cross our minds.
(i encourage picking other topics and seeing how this applies)