quetzal_rainbow

Karma: 2,252

quetzal_rainbow May 4, 2025, 7:48 PM
9 points
4
on: Interpretability Will Not Reliably Find Deceptive AI
There is a conceptual path for interpretability to lead to reliability: you can understand model in sufficient details to know how it produces intelligence and then make another model out of interpreted details. Obviously, it’s not something that we can expect to happen anytime soon, but it’s something that army of interpretability geniuses in datacenter could do.

quetzal_rainbow May 4, 2025, 8:58 AM
16 points
7
on: The Ukraine War and the Kill Market
I think current Russia-Ukraine war is a perfect place to implement such system. It’s an attrition war, there is not many goals which are not reduced to “kill and destroy as many as you can”. There is a strategic aspect: Russia pays exorbitant compensations to families of killed soldiers, decades of income for poor regions. So, when Russian soldier dies, two things can happen:
1. Russian government dumps unreasonable amount of money on the market, contributing to inflation;
2. Russian government fails to pay (for 1000 and 1 stupid bureaucratic reasons), which erodes trust of would-be soldiers and reduces Russia’s mobilization potential.
I can easily see how such system could fail terribly in, say, Afghanistan (if you paid for every killed terrorist, there is an easy loophole “kill civilian, say they are terrorist”). It’s fine for current stage of Ukraine war.
Also I don’t see how kill markets contribute to ability of military to coup. Payments are made in purely virtual points, soldiers can’t spend them on something else.

quetzal_rainbow May 3, 2025, 10:10 PM
3 points
0
in reply to: tailcalled’s comment on: Julian Bradshaw’s Shortform
By “artificial employee” I mean “something than can fully replace human employee, including their agentic capabilities”. And, of course, it should be much more useful than generic AI chatbot, it should be useful like owning Walmart (1,200,000 employees) is useful.

quetzal_rainbow May 3, 2025, 8:04 PM
2 points
0
in reply to: tailcalled’s comment on: Julian Bradshaw’s Shortform
“Set up a corporation with a million of artificial employees” is pretty legible, but human amount of agency is catastrophically insufficient for it.

quetzal_rainbow May 2, 2025, 8:59 PM
2 points
0
in reply to: faul_sname’s comment on: quetzal_rainbow’s Shortform
The emphasis here is not on properties of model behavior but on how developers relate to model testing/understanding.

quetzal_rainbow May 2, 2025, 5:36 PM
16 points
−1
on: quetzal_rainbow’s Shortform
Recent update from OpenAI about 4o sycophancy surely looks like Standard Misalignment Scenario #325:
Our early assessment is that each of these changes, which had looked beneficial individually, may have played a part in tipping the scales on sycophancy when combined.
<...>
One of the key problems with this launch was that our offline evaluations—especially those testing behavior—generally looked good. Similarly, the A/B tests seemed to indicate that the small number of users who tried the model liked it.
<...>
some expert testers had indicated that the model behavior “felt” slightly off.
<...>
We also didn’t have specific deployment evaluations tracking sycophancy.
<...>
In the end, we decided to launch the model due to the positive signals from the users who tried out the model.

quetzal_rainbow Apr 29, 2025, 11:31 AM
3 points
0
in reply to: atharva’s comment on: Is Reality Ugly?
Not really, classical universe can be spatially infinite and contain infinite number of your copies.

quetzal_rainbow Apr 23, 2025, 5:18 AM
53 points
21
on: To Understand History, Keep Former Population Distributions In Mind
The other aspect is age. Russia in 1917 had half of population younger than 18, while modern Russia has median age of 39, with all implications about political stability and conservatism.

quetzal_rainbow Apr 22, 2025, 6:13 PM
21 points
12
on: Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quilt
I’m not commenting anything about positive suggestions in this post, but it has some really strange understanding of what it argues with.
an ‘Axiom of Rational Convergence’. This is the powerful idea that under sufficiently ideal epistemic conditions – ample time, information, reasoning ability, freedom from bias or coercion – rational agents will ultimately converge on a single, correct set of beliefs, values, or plans, effectively identifying ‘the truth’.
There is no reason to “agree on values” (in a sense of actively seeking agreement instead of observing it if it is here) and I don’t think many rationalists would argue with that? Values are free parameters in rational behavior, they can be arbitrary. Different humans have different values and this is fine. CEV is about acting where human values are coherent, not about imposing coherence out of nowhere.

quetzal_rainbow Apr 22, 2025, 5:38 PM
6 points
2
on: Most AI value will come from broad automation, not from R&D
I feel really weird about this article, because I at least spiritually agree with the conclusion and my lower bound on AGI extinction scenario is “automated industry grows 40%/y and at first everybody becomes filthy rich and in next decade necrosphere devours everything” and this scenario seems to be weirdly underdiscussed relatively to “AGI develops self-sustaining supertech and kills everyone within one month” by both skeptics and doomers.
Thus said, I disagree with almost everything else.
You underestimate effect of RnD automation because you seem to ignore that a lot of bottlenecks in RnD and adoption of results are actually human bottlenecks. Like, even if we automate everything in RnD which is not free scientific creativity, we are still bottlenecked by facts that humans can keep in working memory only 10 things and that multitasking basically doesn’t exist and that you can perform several hours of hard intellectual work per day maximum and that you need 28 years to grow scientist and that there is only so many people actually capable to produce science and that science progresses one funeral at time, et cetera, et cetera. The same with adoption: even if innovation exists, there is only so many people actually capable to understand it and implement while integrating into existing workflow in maintainable way and they have only so many productive working hours per day.
Even if we dial every other productivity factor into infinity, there is a still hard cap on productivity growth from inherent human limitations and population. When we achieve AI automation, we substitute these limitations with what is achievable inside physical limitations.
Your assumption “everybody thinks that RnD is going to be automated first because RnD is mostly abstract reasoning” is wrong, because actual reason why RnD is going to be automated relatively early is because it’s very valuable, especially if you count opportunity costs. Every time you pay AI researcher money to automate narrow task, you waste money which could be spent on creating an army on 100k artificial researchers who could automate 100k tasks. I think this holds even if you assume pretty long timelines, because w.r.t. of mentioned human bottlenecks everything you can automate before RnD automation is going to be automated rather poorly (and, unlike science, you can’t run self-improvement from here).

quetzal_rainbow Apr 20, 2025, 9:04 AM
2 points
0
in reply to: Lucius Bushnaq’s comment on: quetzal_rainbow’s Shortform
I mostly think about alignment methods like “model-based RL which maximizes reward iff it outputs action which is provably good under our specification of good”.

quetzal_rainbow Apr 19, 2025, 1:15 PM
2 points
0
on: Probability Theory Fundamentals 102: Territory that Probability is in the Map of
I think the problem here is that you do not quite understand the problem.
It’s not that we “imagine that we’ve imagined the whole world, do not notice any contradictions and call it a day”. It’s that we know there exists idealized procedure which doesn’t produce stupid answers, like, it can’t be money-pumped. We also know that if we try to approximate this procedure harder (consider more hypotheses, compute more inferences) we are going to get in expectation better results. It is not, say, property of null hypothesis testing—the more hypotheses you consider, the more likely you to either p-hack or drive p-value into statistical insignificance due to excessive multiple testing correction.
The whole computationally unbounded Bayesian business is more about “here is an idealized procedure X, and if we don’t do anything visibly for us stupid from perspective of X, then we can hope that our losses won’t be unbounded from certain notion of boundedness”. It is not obvious that your procedure can be understood this way.

quetzal_rainbow Apr 18, 2025, 7:17 PM
2 points
0
in reply to: tailcalled’s comment on: johnswentworth’s Shortform
No, my point is that in worlds where intelligence is possible, almost all obstacles are common.

quetzal_rainbow Apr 17, 2025, 1:52 PM
16 points
3
on: quetzal_rainbow’s Shortform
We can probably survive in the following way:
1. RL becomes the main way to get new, especially superhuman, capabilities.
2. Because RL pushes models hard to do reward hacking, it’s difficult to reliably get models to do something difficult to verify. Models can do impressive feats, but nobody is stupid enough to put AI into positions which usually imply responsibility.
3. This situation conveys how difficult alignment is and everybody moves toward verifiable rewards or similar approaches. Capabilities progress becomes dependent on alignment progress.

quetzal_rainbow Apr 17, 2025, 1:29 PM
2 points
0
in reply to: tailcalled’s comment on: johnswentworth’s Shortform
I feel like I’m failing to convey the level of abstraction I intend to.

I’m not saying that durability of object implies intelligence of object. I’m saying that if the world is ordered in a way that allows existence of distinct durable and non-durable objects, that means the possibility of intelligence which can notice that some objects are durable and some are not and exploit this fact.

If the environment is not ordered enough to contain intelligent beings, it’s probably not ordered enough to contain distinct durable objects too.

To be clear, by “environment” I mean “the entire physics”. When I say “environment not ordered enough” I mean “environment with physical laws chaotic enough to not contain ordered patterns”.

quetzal_rainbow Apr 16, 2025, 10:31 AM
2 points
0
in reply to: tailcalled’s comment on: johnswentworth’s Shortform
I mean, we exist and we are at least somewhat intelligent, which implies strong upper bound on heterogenity of environment.

On the other hand, words like “durability” imply possibility of categorization, which itself implies intelligence. If environment is sufficiently heterogenous, you are durable at one second and evaporate at another.

quetzal_rainbow Apr 16, 2025, 9:12 AM
3 points
0
in reply to: samuelshadrach’s comment on: Joseph Miller’s Shortform
I’m trying to write post, but, well, it’s hard.

Many people wrongly assume that the main way to use bioweapons is to create small amount of pathogen to release it in environment with outbreak as an intended outcome. (I assume that where your sentence about DNA synthesis comes from.) The problem is that creating outbreaks in practice is very hard, we, thankfully, don’t know reliable way to do that. In practice, the way that bioweapons work reliably is “bomb-saturate the entire area with anthrax such that first wave of death is going to be from anaphylactic shock rather than infection” and to create necessary amount of pathogen you need industrial infrastructure which doesn’t exist, because nobody in our civilization cultivates anthrax at industrial scale.

quetzal_rainbow Apr 16, 2025, 8:40 AM
2 points
0
in reply to: tailcalled’s comment on: johnswentworth’s Shortform
Filter for homogenity of environment is anthropic selection—if environment is sufficiently heterogeneous, it kills everyone who tries to reach out of its ecological niche, general intelligence doesn’t develop and we are not here to have this conversation.

quetzal_rainbow Mar 29, 2025, 6:06 AM
14 points
0
in reply to: Richard_Ngo’s comment on: ricraz’s Shortform
In some sense this is a core idea of UDT: when coordinating with forks of yourself, you defer to your unique last common ancestor. When it’s not literally a fork of yourself, there’s more arbitrariness but you can still often find a way to use history to narrow down on coordination Schelling points (e.g. “what would Jesus do”)
I think this is wholly incorrect line of thinking. UDT operates on your logical ancestor, not literal.
Say, if you know enough science, you know that normal distribution is a maxentropy distribution for fixed mean and variance, and therefore, optimal prior distribution under certain set of assumptions. You can ask yourself question “let’s suppose that I haven’t seen this evidence, what would be my prior probability?” and get an answer and cooperate with your counterfactual versions which have seen other versions of evidence. But you can’t cooperate with your hypothetical version which doesn’t know what normal distribution is, because, if it doesn’t know about normal distribution, it can’t predict how you would behave and account for this in cooperation.
Sufficiently different versions of yourself are just logically uncorrelated with you and there is no game-theoretic reason to account for them.

quetzal_rainbow Mar 24, 2025, 8:59 AM
3 points
1
on: Solving willpower seems easier than solving aging
I’m quite skeptical about “easier” due to usual Algernon argument.
Roughly, conceptually, animal activity can be divided into 3 phases:
1. Active, when animal is looking for food or mate or something.
2. Sleep (which seems to be a requirement for Earth animals design).
3. Passive phase in-between when animal doesn’t do anything particular, but it shouldn’t sleep because of predators (the main problem with sleep as an evolutionary adaptation).
Large herbivores have very short sleep phase and active/passive phase are not very meaningfully different, because food doesn’t hide from them and is not very energy-dense, and they should always look out for predators. Large predators have very long sleep phase, short active phase and don’t have passive phase—because their search for food is very energy-intense and they don’t need to look out for predators.
Humans seem to be comfortably between these two extremes, but I think that unproductive phase in your day is just what it is—unproductive phase, meant only for you to be alert about predators (and fellow tribesmen). You can probably extend your productive phase somewhat by doing known things, like healthy diet, normal sleep, exercise, but it is likely that you have strong biological limit on what you can productively do per day which you can’t cross without degradation in quality of life.
I don’t think caffeine and amphetamines help here. They just forcefully redistribute activity—if you have ADHD, you are going to work instead of doing whatever you feel like and if you are night owl, you can function better at morning.
You can play on difference between “activity which requires you to be biologically active” and “meaningful activity”, like, I don’t think watching TV with family is very energetically demanding. Although, it seems to be a time organization problem?
I think that actual solution to “lack of biological willpower” is something like “large bionanotech system which resets organism in the way like sleep resets it” and honestly, I think if you can develop such systems you are not far from actual aging treatment.