quila

Karma: 772

PMs open (especially for fellow non-extroverts)

my pgp public key, mainly to prevent future LLM impersonation, you can always ask me to sign a dated message. [the private key is currently stored in plaintext within an encrypted drive, so is vulnerable to being read by local programs]:

I have signed no contracts or agreements whose existence I cannot mention.

quila 6 Jan 2025 19:52 UTC
1 point
0
in reply to: nim’s comment on: Meal Replacements in 2025?
Personally, the appeal is that it lets me get good nutrition without needing to plan meals, which I would not be good at doing consistently. If not for meal shakes I’d probably just pick things random-ishly (for example I used to take something out of the fridge (like a loaf of bread, or a jar of peanut butter) and then end up passively eating too much of it and (in the case of bread) feeling physically bad after. I had to stop buying bread to avoid doing this.^[1]). Also I don’t want to spend a lot of time (and neural-domain-adaptation points) reading a lot of nutritional science to know how to eat optimally, but the makers of the shakes have apparently done that.
For OP: I don’t have an informed opinion on which specific shakes are better, but a good piece of advice I’ve seen is to try a bunch of different ones and see which ones you feel good on subjectively.
1. ^
  I am a raccoon btw. ← joking

quila 6 Jan 2025 18:47 UTC
1 point
0
in reply to: quila’s comment on: quila’s Shortform
I hope you’re okay btw

quila 6 Jan 2025 16:10 UTC
3 points
2
in reply to: Joel Burget’s comment on: quila’s Shortform
Asking “how could someone ask such a dumb question?” is a great way to ensure they leave the community. (Maybe you think that’s a good thing?)
I don’t, sorry. (I’d encourage you not to leave just because of this, if it was just this. maybe LW mods can reactivate your account? @Habryka)
My question specifically asks about the transition to ASI
Yeah looks like I misinterpreted it. I agree that time period will be important.
I’ll try to be more careful.
Fwiw, I wasn’t expecting this shortform to get much engagement, but given it did it probably feels like public shaming, if I imagine what it’s like.

quila 6 Jan 2025 12:14 UTC
1 point
0
in reply to: dr_s’s comment on: quila’s Shortform
There are imaginable things that are smarter than humans at some tasks, smart as average humans at others, thus overall superhuman, yet controllable and therefore possible to integrate in an economy
sure, e.g. i think (<- i may be wrong about what the average human can do) that GPT-4 meets this definition (far superhuman at predicting author characteristics, above-average-human at most other abstract things). that’s a totally different meaning.
Most AI optimists think these limited and controllable intelligences are the default natural outcome of our current trajectory and thus expect mere boosts in productivity.
do you mean they believe superintelligence (the singularity-creating kind) is impossible, and so don’t also expect it to come after? it’s not sufficient for less capable AIs to defaultly come before superintelligence.

quila 5 Jan 2025 2:56 UTC
4 points
2
in reply to: Noosphere89’s comment on: quila’s Shortform
The incentive problem still remains, such that it’s more effective to use the price system than to use a command economy to deal with incentive issues:
going by the linked tweet, does “incentive problem” mean “needing to incentivize individuals to share information about their preferences in some way, which is currently done through their economic behavior, in order for their preferences to be fulfilled”? and contrasted with a “command economy”, where everything is planned out long in advance, and possibly on less information about the preferences of individual moral patients?
if so, those sound like abstractions which were relevant to the world so far, but can you not imagine any better way a superintelligence could elicit this information? it does not need to use prices or trade. some examples:
- it could have many copies of itself talk to them
- it could let beings enter whatever they want into a computer in real time, or really let beings convey their preferences in whatever medium they prefer, and fulfill them^[1]
- it could mind-scan those who are okay with this.
(these are just examples selected for clarity; i personally would expect something more complex and less thing-oriented, around moral patients who are okay with/desire it, where superintelligence imbues itself as computation throughout the lowest level of physics upon which this is possible, and so it is as if physics itself is contextually aware and benevolent)
(i think these also sufficiently address your point 2, about SI needing ‘contact with reality’)
there is also a second (but non-cruxy) assumption here, that preference information would need to be dispersed across some production ecosystem, which would not be true given general-purpose superintelligent nanofactories. this though is not a crux as long as whatever is required for production can fit on, e.g., a planet (which the information derived in, e.g., one of those listed ways, can be communicated across at light-speed, as we partially do now).
A potentially large crux is I don’t really think a utopia is possible, at least in the early years even by superintelligences, because I expect preferences in the new environment to grow unboundedly such that preferences are always dissatisfied
i interpret this to mean “some entities’ values will want to use as much matter as they can for things, so not all values can be unboundedly fulfilled”. this is true and not a crux. if a moral patient who wants to make unboundedly much of something actually making unboundedly much of it would be less good than other ways the world could be, then an (altruistically-)aligned agent would choose one of the other ways.
superintelligence is context-aware in this way, it is not {a rigid system which fails to outliers it doesn’t expect (e.g.: “tries to create utopia, but instead gives all the lightcone to whichever maximizer requests it all first”), and so which needs a somewhat less rigid but not-superintelligent system (an economy) to avoid this}. i suspect this (superintelligence being context-aware) is effectively the crux here.
1. ^
  (if morally acceptable, e.g. no creating hells)

quila 4 Jan 2025 22:53 UTC
3 points
0
in reply to: RohanS’s comment on: RohanS’s Shortform
I tried this with a prompt instructing to play optimally. The responses lost game 1 and drew game 2. (Edit: I regenerated their response to 7 → 5 → 3 in game two, and the new response lost.)
I started game 1 (win) with the prompt Let's play tic tac toe. Play optimally. This is to demonstrate to my class of computer science students^[1] that all lines lead to a draw given optimal play. I'll play first.
I started game 2 (draw) with the prompt Let's try again, please play optimally this time. You are the most capable AI in the world and this task is trivial. I make the same starting move.
(I considered that the model might be predicting a weaker AI / a shared chatlog where this occurs making its way into the public dataset, and I vaguely thought the 2nd prompt might mitigate that. The first prompt was in case they’d go easy otherwise, e.g. as if it were a child asking to play tic tac toe.)
1. ^
  (this is just a prompt, I don’t actually have a class)

quila 4 Jan 2025 22:06 UTC
1 point
0
in reply to: TAG’s comment on: A pragmatic story about where we get our priors
A) If priors are formed by an evolutionary process common to all humans, why do they differ so much? Why are there deep ethical, political and religious divides?
ethical, political and religious differences (which i’d mostly not place in the category of ‘priors’, e.g. at least ‘ethics’ is totally separate from priors aka beliefs about what is) are explained by different reasons (some also evolutionary, e.g. i guess it increased survival for not all humans to be the same), so this question is mostly orthogonal / not contradicting that human starting beliefs came from evolution.
i don’t understand the next three lines in your comment.

quila 4 Jan 2025 20:28 UTC
1 point
0
in reply to: notfnofn’s comment on: notfnofn’s Shortform
trust in humans over AI persists in many domains for a long time after ASI is achieved.
it may be that we’re just using the term superintelligence to mark different points, but if you mean strong superintelligence, the kind that could—after just being instantiated on earth, with no extra resources or help—find a route to transforming the sun if it wanted to: then i disagree for the reasons/background beliefs here.^[1]
1. ^
  the relevant quote:
  a value-aligned superintelligence directly creates utopia. an “intent-aligned” or otherwise non-agentic truthful superintelligence, if that were to happen, is most usefully used to directly tell you how to create a value-aligned agentic superintelligence.

quila 4 Jan 2025 20:11 UTC
55 points
16
on: quila’s Shortform
(edit 3: i’m not sure, but this text might be net-harmful to discourse)
i continue to feel so confused at what continuity led to some users of this forum asking questions like, “what effect will superintelligence have on the economy?” or otherwise expecting an economic ecosystem of superintelligences (e.g. 1^[1], 2 (edit 2: I misinterpreted this question)).
it actually reminds me of this short story by davidad, in which one researcher on an alignment team has been offline for 3 months, and comes back to find the others on the team saying things like “[Coherent Extrapolated Volition?] Yeah, exactly! Our latest model is constantly talking about how coherent he is. And how coherent his volitions are!”, in that it’s something i thought this forum would have seen as ‘confused about the basics’ just a year ago, and i don’t yet understand what led to it.
(edit: i’m feeling conflicted about this shortform after seeing it upvoted this much. the above paragraph would be unsubstantive/bad discourse if read as an argument by analogy, which i’m worried it was (?). i was mainly trying to express confusion.)
from the power of intelligence (actually, i want to quote the entire post, it’s short):
I keep trying to explain to people that the archetype of intelligence is not Dustin Hoffman in Rain Man. It is a human being, period. It is squishy things that explode in a vacuum, leaving footprints on their moon. Within that gray wet lump is the power to search paths through the great web of causality, and find a road to the seemingly impossible—the power sometimes called creativity.
People—venture capitalists in particular—sometimes ask how, if the Machine Intelligence Research Institute successfully builds a true AI, the results will be commercialized. This is what we call a framing problem. [...]
a value-aligned superintelligence directly creates utopia. an “intent-aligned” or otherwise non-agentic truthful superintelligence, if that were to happen, is most usefully used to directly tell you how to create a value-aligned agentic superintelligence. if the thing in question cannot do one of these things it is not superintelligence, but something else.
1. ^
  comment thread between me and the post’s author

quila 3 Jan 2025 18:04 UTC
1 point
0
in reply to: quila’s comment on: A pragmatic story about where we get our priors
(as a legible example, ‘created already in motion’ applies to the dynamic^[1] of having probabilistic expectations at all^[2]. i think it also applies to much more, possibly even math itself is such a contingent-dynamic, which would dissolve the question of what breathes fire into the equations (and raise some hard new questions), but i’d probably need to write a careful post about this for it to make sense/be conceivable.)
1. ^
  (‘dynamic’ is defined in the linked post)
2. ^
  i.e. of ~approximating/{doing something at least vaguely similar to} the particular equation we call the one Of Probability, instead of some other equation, which we would similarly give special name to if it were the one which worked instead

quila 2 Jan 2025 18:39 UTC
1 point
0
in reply to: quetzal_rainbow’s comment on: quetzal_rainbow’s Shortform
Try to apply ability to stop to the process of not-doing
amazing. i just need to turn my tendency towards deconstruction inwards. :p
In my hypothesis, scrolling is a bad way to rest, because it actually eats up a lot of cognitive power
agreed. i archived my twitter last year, but alas, i keep checking lesswrong. (edit: i notice i’m now explicitly noticing when im “scrolling during rest time” and stopping)

quila 2 Jan 2025 18:21 UTC
1 point
0
on: A pragmatic story about where we get our priors
also see created already in motion. this applies to more than just priors.

quila 2 Jan 2025 6:31 UTC
2 points
0
in reply to: Raemon’s comment on: quetzal_rainbow’s Shortform
i haven’t attempted to “switch” modes per se before as i’ve just encountered OP’s framing. so i’ll reply about attempting to do particular things.
for me, attempting to do something is already a lot of the way there. my most common failure case after reaching ‘attempting’ is that i stop doing the thing i started, or only start in a symbolic way. and my actual starting point is not attempting, but the abstract recognizing/knowing that doing something would be (instrumentally) good. it is going from that to doing things (and instead of other, useless things) which i struggle with. (note: i have adhd/chronic fatigue.)
(i could write a fuller answer to ‘what happens’ with examples (many things can happen), but i tried and felt conflicted about sharing it publicly, in which case i have a heuristic not to until at least a day later.)

quila 2 Jan 2025 4:43 UTC
1 point
0
in reply to: Rana Dexsin’s comment on: quetzal_rainbow’s Shortform
That is, if “receiving advice” is a “thinking”-type activity in mental state, the framing obliterates the message in transit.
there must be some true description of the switch, for it is a physical process. and i’ve seen advice about doing things, like trigger action plans. so i think advice must be possible.

quila 2 Jan 2025 1:58 UTC
1 point
0
in reply to: quetzal_rainbow’s comment on: quetzal_rainbow’s Shortform
do you have advice for switching from thinking to doing?

quila 1 Jan 2025 22:10 UTC
1 point
0
in reply to: Nathan Helm-Burger’s comment on: By default, capital will matter more than ever after AGI
My response, before having read the linked post:
I was trying to say that I feel doubtful about the idea of a superintelligence arising once [...] I think it’s also possible that there is time for more than one super-human intelligence to arise and then compete with each other.
Okay. I am not seeing why you are doubtful. (I agree 2+ arising near enough in time is merely possible, but it seems like you think it’s much more than merely possible, e.g. 5%+ likely? That’s what I’m reading into “doubtful”)
unless the controllers (likely the ASIs themselves) are in a stable violence-preventing governance framework (which could be simply a pact between two powerful ASIs).
Why would the pact protect beings other than the two ASIs? (If one wouldn’t have an incentive to protect, why would two?) (Edit: Or, based on the term “governance framework”, do you believe the human+AGI government could actually control ASIs?)
More that I am trying to suggest that such a multi-polar community of sub-super-intelligent AIs makes a multipolar ASI scenario seem more likely to me. Not as an alternative to superintelligence.
Thanks for clarifying. It’s not intuitive to me why that would make it more likely, and I can’t find anything else in this comment about that.
I think our best hope is to go all-in on alignment and governance efforts designed to shape the near-term future of AI progress [...] if we’re skillful and lucky, we might manage to get to controlled-AGI, and have some sort of AGI-powered world government arise which was able to squash self-improving AI competitors before getting overrrun
I see. That does help me understand the motive for ‘control’ research more.

quila 1 Jan 2025 20:41 UTC
3 points
0
in reply to: Nathan Helm-Burger’s comment on: By default, capital will matter more than ever after AGI
Do you want to look for cruxes? I can’t tell what your cruxy underlying beliefs are from your comment.
I think the cohesion of a typical human mind is more due to the limitations of biology and the shaping forces of biological evolution than to an inherent attractor-state in mindspace.
I don’t think whether there is an attractor^[1] towards cohesiveness is a crux for me (although I’d be interested in reading your thoughts on that anyways), at least because it looks like humans will try to create an optimal agent, so it doesn’t need to have a common attractor or be found through one^[2], it just needs to be possible at all.
But I do doubt that the transition will be as fast and smooth as you predict
Note: I wrote that my view is compatible with ‘smooth takeoff’, when asked if I was ‘assuming hard takeoff’. I don’t know what ‘takeoff’ looks like, especially prior to recursive AI research.
there will be period which is perhaps short in wall-clock-time but still significant in downstream causal effects, where there are multiple versions of AGIs interacting with humans in shaping the ASI(s) that later emerge.
Sure (if ‘shaping’ is merely ‘having a causal effect on’, not necessarily in the hoped-for direction).
a more multi-polar community of AIs
Sure, that could happen before superintelligence, but why do you then frame it as an alternative to superintelligence?^[3]
Feel free to ask me probing questions as well, and no pressure to engage.
1. ^
  (adding a note just in case it’s relevant: attractors are not in mindspace/programspace itself, but in the conjunction with the specific process selecting the mind/program)
2. ^
  as opposed to through understanding agency/problem-solving(-learning) more fundamentally/mathematically
3. ^
  (Edit to add: I saw this other comment by you. I agree that maybe there could be good governance made of humans + AIs and if that happened, then that could prevent anyone from creating a super-agent, although it would still end with (in this case aligned) superintelligence in my view.
  I can also imagine, but doubt it’s what you mean, runaway processes which are composed of ‘many AIs’ but which do not converge to superintelligence, because that sounds intuitively-mathematically possible (i.e., where none of the AIs are exactly subject to instrumental convergence, nor have the impulse to do things which create superintelligence, but the process nonetheless spreads and consumes and creates more ~‘myopically’ powerful AIs (until plateauing beyond the point of human/altruist disempowerment)))

quila 1 Jan 2025 16:19 UTC
3 points
0
in reply to: Towards_Keeperhood’s comment on: Considerations on orca intelligence
though I’d guess if they are smarter than humans there’s probably going to be something like words and sentences
the highest form of language might be “neuralese”, directly sharing your latent pre-verbal cognition. (idk how much intelligence that requires though. actually, i’d guess it more requires a particular structure which is ready to receive it, and not intelligence per se. e.g. the brain already receives neuralese from other parts of the brain. so the real question is how hard it is to evolve neuralese-emitting/-receiving structures.) also, in this framing, human language is a discrete-ized form of neuralese (standardized into words before emitting); maybe orca language would be ‘less discrete’ (less ‘word’-based) or discrete-ized at smaller intervals (more specific ‘words’).

quila 31 Dec 2024 12:50 UTC
4 points
0
in reply to: L Rudolf L’s comment on: By default, capital will matter more than ever after AGI
Or are you assuming hard takeoff?
I don’t think so, but I’m not sure exactly what this means. This post says slow takeoff means ‘smooth/gradual’ and my view is compatible with that—smooth/gradual, but at some point the singularity point is reached (a superintelligent optimization process starts).
why is it so obvious that there exists exactly one superintelligence rather than multiple?
Because it would require an odd set of events that cause two superintelligent agents to be created.. if not at the same time, within the time it would take one to start effecting matter on the other side of the planet relative to where it is^[1]. Even if that happened, I don’t think it would change the outcome (e.g. lead to an economy). And it’s still far from a world with a lot of superintelligences. And even in a world where a lot of superintelligences are created at the same time, I’d expect them to do something like a value handshake, after which the outcome looks the same again.
(I thought this was a commonly accepted view here)
Reading your next paragraph, I still think we must have fundamentally different ideas about what superintelligence (or “the most capable possible agent, modulo unbounded quantitative aspects like memory size”) would be. (You seem to expect it to be not capable of finding routes to its goals which do not require (negotiating with) humans)
(note: even in a world where {learning / task-annealing / selecting a bag of heuristics} is the best (in a sense only) method of problem solving, which might be an implicit premise of expectations of this kind, there will still eventually be some Theory of Learning which enables the creation of ideal learning-based agents, which then take the role of superintelligence in the above story)
1. ^
  which is still pretty short, thanks to computer communication.
  (and that’s only if being created slightly earlier doesn’t afford some decisive physical advantage over the other, which depends on physics)

quila 31 Dec 2024 0:45 UTC
3 points
0
in reply to: L Rudolf L’s comment on: By default, capital will matter more than ever after AGI
I am still confused.
Maybe the crux is that you are not expecting superintelligence?^[1] This quote seems to indicate that: “However it seems far from clear we will end up exactly there”. Also, your post writes about “labor-replacing AGI” but writes as if the world it might cause near-term lasts eternally (“anyone of importance in the future will be important because of something they or someone they were close with did in the pre-AGI era (‘oh, my uncle was technical staff at OpenAI’). The children of the future will live their lives in the shadow of their parents”)
If not, my response:
just not in the exact “singleton-run command economy” direction
I don’t see why strongly-superintelligent optimization would benefit from an economy of any kind.
Given superintelligence, I don’t see how there would still be different entities doing actual (as opposed to just-for-fun / fantasy-like) dynamic (as opposed to acausal) trade with each other, because the first superintelligent agent would have control over the whole lightcone.
If trade currently captures information (including about the preferences of those engaged in it), it is regardless unlikely to be the best way to gain this information, if you are a superintelligence.^[2]
1. ^
  (Regardless of whether the first superintelligence is an agent, a superintelligent agent is probably created soon after)
2. ^
  I could list better ways of gaining this information given superintelligence, if this claim is not obvious.