quetzal_rainbow

Karma: 2,139

quetzal_rainbow Apr 20, 2025, 9:04 AM
2 points
0
in reply to: Lucius Bushnaq’s comment on: quetzal_rainbow’s Shortform
I mostly think about alignment methods like “model-based RL which maximizes reward iff it outputs action which is provably good under our specification of good”.

quetzal_rainbow Apr 19, 2025, 1:15 PM
2 points
0
on: Probability Theory Fundamentals 102: Territory that Probability is in the Map of
I think the problem here is that you do not quite understand the problem.
It’s not that we “imagine that we’ve imagined the whole world, do not notice any contradictions and call it a day”. It’s that we know there exists idealized procedure which doesn’t produce stupid answers, like, it can’t be money-pumped. We also know that if we try to approximate this procedure harder (consider more hypotheses, compute more inferences) we are going to get in expectation better results. It is not, say, property of null hypothesis testing—the more hypotheses you consider, the more likely you to either p-hack or drive p-value into statistical insignificance due to excessive multiple testing correction.
The whole computationally unbounded Bayesian business is more about “here is an idealized procedure X, and if we don’t do anything visibly for us stupid from perspective of X, then we can hope that our losses won’t be unbounded from certain notion of boundedness”. It is not obvious that your procedure can be understood this way.

quetzal_rainbow Apr 18, 2025, 7:17 PM
2 points
0
in reply to: tailcalled’s comment on: johnswentworth’s Shortform
No, my point is that in worlds where intelligence is possible, almost all obstacles are common.

quetzal_rainbow Apr 17, 2025, 1:52 PM
16 points
1
on: quetzal_rainbow’s Shortform
We can probably survive in the following way:
1. RL becomes the main way to get new, especially superhuman, capabilities.
2. Because RL pushes models hard to do reward hacking, it’s difficult to reliably get models to do something difficult to verify. Models can do impressive feats, but nobody is stupid enough to put AI into positions which usually imply responsibility.
3. This situation conveys how difficult alignment is and everybody moves toward verifiable rewards or similar approaches. Capabilities progress becomes dependent on alignment progress.

quetzal_rainbow Apr 17, 2025, 1:29 PM
2 points
0
in reply to: tailcalled’s comment on: johnswentworth’s Shortform
I feel like I’m failing to convey the level of abstraction I intend to.

I’m not saying that durability of object implies intelligence of object. I’m saying that if the world is ordered in a way that allows existence of distinct durable and non-durable objects, that means the possibility of intelligence which can notice that some objects are durable and some are not and exploit this fact.

If the environment is not ordered enough to contain intelligent beings, it’s probably not ordered enough to contain distinct durable objects too.

To be clear, by “environment” I mean “the entire physics”. When I say “environment not ordered enough” I mean “environment with physical laws chaotic enough to not contain ordered patterns”.

quetzal_rainbow Apr 16, 2025, 10:31 AM
2 points
0
in reply to: tailcalled’s comment on: johnswentworth’s Shortform
I mean, we exist and we are at least somewhat intelligent, which implies strong upper bound on heterogenity of environment.

On the other hand, words like “durability” imply possibility of categorization, which itself implies intelligence. If environment is sufficiently heterogenous, you are durable at one second and evaporate at another.

quetzal_rainbow Apr 16, 2025, 9:12 AM
3 points
0
in reply to: samuelshadrach’s comment on: Joseph Miller’s Shortform
I’m trying to write post, but, well, it’s hard.

Many people wrongly assume that the main way to use bioweapons is to create small amount of pathogen to release it in environment with outbreak as an intended outcome. (I assume that where your sentence about DNA synthesis comes from.) The problem is that creating outbreaks in practice is very hard, we, thankfully, don’t know reliable way to do that. In practice, the way that bioweapons work reliably is “bomb-saturate the entire area with anthrax such that first wave of death is going to be from anaphylactic shock rather than infection” and to create necessary amount of pathogen you need industrial infrastructure which doesn’t exist, because nobody in our civilization cultivates anthrax at industrial scale.

quetzal_rainbow Apr 16, 2025, 8:40 AM
2 points
0
in reply to: tailcalled’s comment on: johnswentworth’s Shortform
Filter for homogenity of environment is anthropic selection—if environment is sufficiently heterogeneous, it kills everyone who tries to reach out of its ecological niche, general intelligence doesn’t develop and we are not here to have this conversation.

quetzal_rainbow Mar 29, 2025, 6:06 AM
14 points
0
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
In some sense this is a core idea of UDT: when coordinating with forks of yourself, you defer to your unique last common ancestor. When it’s not literally a fork of yourself, there’s more arbitrariness but you can still often find a way to use history to narrow down on coordination Schelling points (e.g. “what would Jesus do”)
I think this is wholly incorrect line of thinking. UDT operates on your logical ancestor, not literal.
Say, if you know enough science, you know that normal distribution is a maxentropy distribution for fixed mean and variance, and therefore, optimal prior distribution under certain set of assumptions. You can ask yourself question “let’s suppose that I haven’t seen this evidence, what would be my prior probability?” and get an answer and cooperate with your counterfactual versions which have seen other versions of evidence. But you can’t cooperate with your hypothetical version which doesn’t know what normal distribution is, because, if it doesn’t know about normal distribution, it can’t predict how you would behave and account for this in cooperation.
Sufficiently different versions of yourself are just logically uncorrelated with you and there is no game-theoretic reason to account for them.

quetzal_rainbow Mar 24, 2025, 8:59 AM
3 points
1
on: Solving willpower seems easier than solving aging
I’m quite skeptical about “easier” due to usual Algernon argument.
Roughly, conceptually, animal activity can be divided into 3 phases:
1. Active, when animal is looking for food or mate or something.
2. Sleep (which seems to be a requirement for Earth animals design).
3. Passive phase in-between when animal doesn’t do anything particular, but it shouldn’t sleep because of predators (the main problem with sleep as an evolutionary adaptation).
Large herbivores have very short sleep phase and active/passive phase are not very meaningfully different, because food doesn’t hide from them and is not very energy-dense, and they should always look out for predators. Large predators have very long sleep phase, short active phase and don’t have passive phase—because their search for food is very energy-intense and they don’t need to look out for predators.
Humans seem to be comfortably between these two extremes, but I think that unproductive phase in your day is just what it is—unproductive phase, meant only for you to be alert about predators (and fellow tribesmen). You can probably extend your productive phase somewhat by doing known things, like healthy diet, normal sleep, exercise, but it is likely that you have strong biological limit on what you can productively do per day which you can’t cross without degradation in quality of life.
I don’t think caffeine and amphetamines help here. They just forcefully redistribute activity—if you have ADHD, you are going to work instead of doing whatever you feel like and if you are night owl, you can function better at morning.
You can play on difference between “activity which requires you to be biologically active” and “meaningful activity”, like, I don’t think watching TV with family is very energetically demanding. Although, it seems to be a time organization problem?
I think that actual solution to “lack of biological willpower” is something like “large bionanotech system which resets organism in the way like sleep resets it” and honestly, I think if you can develop such systems you are not far from actual aging treatment.

quetzal_rainbow Mar 18, 2025, 7:44 PM
2 points
0
in reply to: Lukas_Gloor’s comment on: AI #107: The Misplaced Hype Machine
“The best possible life” for me pretty much includes “everyone who I care about is totally happy”?
And parents certainly do dangerous risky things to provide better future for their children all the time.

quetzal_rainbow Mar 16, 2025, 4:14 PM
21 points
9
on: quetzal_rainbow’s Shortform
When we should expect “Swiss cheese” approach in safety/security to go wrong:

quetzal_rainbow Mar 2, 2025, 11:07 AM
11 points
14
on: Maintaining Alignment during RSI as a Feedback Control Problem
I think the general problem with your metaphor is that we don’t know “relevant physics” of self-improvement. We can’t plot “physically realistic” trajectory of landing in “good values” land and say “well, we need to keep ourselves in direction of this trajectory”. BTW, MIRI has a dialogue with this metaphor.
And most of your suggestions are like “let’s learn physics of alignment”? I have nothing against that, but it is the hard part, and control theory doesn’t seem to provide a lot of insight here. It’s a framework at best.

quetzal_rainbow Mar 1, 2025, 9:39 AM
2 points
0
in reply to: Said Achmiz’s comment on: On Eating the Sun
Yes, that’s why it’s compromise—nobody will totally like it. But if Earth is going to exist for trillions of years, it will radically change too.

quetzal_rainbow Feb 23, 2025, 5:05 PM
7 points
2
in reply to: RedMan’s comment on: Joseph Miller’s Shortform
My honest opinion is that WMD evaluations of LLMs are not meaningfully related to X-risk in the sense of “kill literally everyone.” I guess current or next-generation models may be able to assist a terrorist in a basement in brewing some amount of anthrax, spraying it in a public place, and killing tens to hundreds of people. To actually be capable to kill everyone from a basement, you would need to bypass all the reasons industrial production is necessary at the current level of technology. A system capable to bypass the need for industrial production in a basement is called “superintelligence,” and if you have a superintelligent model on the loose, you have far bigger problems than schizos in basements brewing bioweapons.

I think “creeping WMD relevance”, outside of cyberweapons, is mostly bad, because it is concentrated on mostly fake problem, which is very bad for public epistemics, even if we forget about lost benefits from competent models.

quetzal_rainbow Feb 23, 2025, 11:16 AM
33 points
4
in reply to: RedMan’s comment on: Joseph Miller’s Shortform
Well, I have bioengineering degree, but my point is that “direct lab experience” doesn’t matter, because WMDs in quality and amount necessary to kill large numbers of enemy manpower are not produced in labs. They are produced in large industrial facilities and setting up large industrial facility for basically anything is on “hard” level of difficulty. There is a difference between large-scale textile industry and large-scale semiconductor industry, but if you are not government or rich corporation, all of them lie in “hard” zone.
Let’s take, for example, Saddam chemical weapons program. First, industrial yields: everything is counted in tons. Second: for actual success, Saddam needed a lot of existing expertise and machinery from West Germany.
Let’s look at Soviet bioweapons program. First, again, tons of yield (someone may ask yourself, if it’s easier to kill using bioweapons than conventional weaponry, why somebody needs to produce tons of them?). Second, USSR built the entire civilian biotech industry around it (many Biopreparat facilities are active today as civilian objects!) to create necessary expertise.
The difference with high explosives is that high explosives are not banned by international law, so there is a lot of existing production, therefore you can just buy them on black market or receive from countries which don’t consider you terrorist. If you really need to produce explosives locally, again, precursors, machinery and necessary expertise are legal and widespread sufficiently that they can be bought.
There is a list of technical challenges in bioweaponry where you are going to predictably fuck up if you have biological degree and you think you know what you are doing but in reality you do not, but I don’t write out lists of technical challenges on the way to dangerous capabilities, because such list can inspire someone. You can get an impression about easier and lower-stakes challenges from here.

quetzal_rainbow Feb 22, 2025, 5:39 PM
16 points
3
in reply to: Joseph Miller’s comment on: Joseph Miller’s Shortform
The trick is that chem/bio weapons can’t, actually, “be produced simply with easily available materials”, if we talk about military-grade stuff, not “kill several civilians to create scary picture in TV”.

quetzal_rainbow Feb 21, 2025, 4:21 PM
3 points
0
in reply to: Mo Putera’s comment on: Mo Putera’s Shortform
It’s very funny that Rorschach linguistic ability is totally unremarkable comparing to modern LLMs.

quetzal_rainbow Feb 20, 2025, 2:54 PM
6 points
0
in reply to: Ruby’s comment on: Why do we have the NATO logo?
The real question is why does NATO have our logo.
This is LGBTESCREAL agenda

quetzal_rainbow Feb 18, 2025, 8:31 PM
5 points
1
on: Abstract Mathematical Concepts vs. Abstractions Over Real-World Systems
I think there is an abstraction between “human” and “agent”: “animal”. Or, maybe, “organic life”. Biological systematization (meaning all ways to systematize: phylogenetic, morphological, functional, ecological) is a useful case study for abstraction “in the wild”.