quetzal_rainbow

• quetzal_rainbowJun 1, 2025, 11:19 AM

  7 points

  4

  on: quet­zal_rain­bow’s Shortform

  For your information, Ukraine seems to have attacked airfields in Murmansk and Irkutsk Oblast’s. It’s approximately 1800 and 4500 km from Ukraine border respectively. Suspected method of attack is drones, transported on truck.

• quetzal_rainbowMay 28, 2025, 10:12 AM

  4 points

  0

  on: Does the Univer­sal Geom­e­try of Embed­dings pa­per have big im­pli­ca­tions for in­ter­pretabil­ity?

  The problem here is that sequence embeddings should have tons of side-channels which should convey non-semantic information (like, say, frequencies of tokens in sequence) and you can come a long way with this sort of information.

  What would be really interesting is to train embedding models in different languages and check whether you can translate highly metaphorical sentences with no correspondence other than semantic, or train embedding models on different representations of the same math (for example, matrix mechanics vs wave mechanics formulations of quantum mechanics) and see if they recognize equivalent theorems.

• quetzal_rainbowMay 25, 2025, 9:34 PM

  2 points

  0

  in reply to: tailcalled’s comment on: tail­called’s Shortform

  Who’s gonna do that? It’s not like we have enough young people for rapid cultural evolution.

• quetzal_rainbowMay 23, 2025, 10:22 AM

  6 points

  4

  in reply to: silentbob’s comment on: silent­bob’s Shortform

  I think you would appreciate this post

• quetzal_rainbowMay 22, 2025, 8:16 PM

  2 points

  0

  in reply to: fencebuilder’s comment on: quet­zal_rain­bow’s Shortform

  Yeah, they both made up some stuff in response to the same question.

• quetzal_rainbowMay 22, 2025, 7:57 PM

  9 points

  2

  on: quet­zal_rain­bow’s Shortform

  I’m so far not impressed with Claude 4s. They are trying to make up superficially plausible stuff for my math questions as fast as possible. Sonnet 3.7, at least, explored a lot of genuinely interesting venues before making an error. “Making up superficially plausible stuff” sounds like a good strategy for hacking not very robust verifiers.

• quetzal_rainbowMay 19, 2025, 2:58 PM

  9 points

  0

  on: $500 + $500 Bounty Prob­lem: Does An (Ap­prox­i­mately) Deter­minis­tic Max­i­mal Re­dund Always Ex­ist?

  I dunno how much it’s obvious for people who want to try for bounty, but I only now realized that you can express criteria for redund as inequality with mutual information and I find mutual information to be much nicer to work with, even if from pure convenience of notation. Proof:

  Let’s take criterion for redund $\Gamma$ w.r.t. of $X_1$, $X_2$

  $ϵ\ge D_{KL}\left(P\right[X_1,X_2,\Gamma ]\parallel P[X_1\left]P\right[X_2|X_1\left]P\right[\Gamma |X_2\left]\right)$

  expand expression for KL divergence:$D_{KL}\left(P\right[X_1,X_2,\Gamma ]\parallel P[X_1\left]P\right[X_2|X_1\left]P\right[\Gamma |X_2\left]\right)={\sum }_{x_1,x_2,\gamma }P(x_1,x_2,\gamma )\log \frac{P(x_1,x_2,\gamma )}{P\left(x_1\right)P\left(x_2|x_1\right)P\left(\gamma |x_2\right)}$

  expand joint distribution:

  $D_{KL}\left(P\right[X_1,X_2,\Gamma ]\parallel P[X_1\left]P\right[X_2|X_1\left]P\right[\Gamma |X_2\left]\right)={\sum }_{x_1,x_2,\gamma }P(x_1,x_2,\gamma )\log \frac{P\left(x_1\right)P\left(x_2|x_1\right)P(\gamma |x_1,x_2)}{P\left(x_1\right)P\left(x_2|x_1\right)P\left(\gamma |x_2\right)}$

  simplify:

  $D_{KL}\left(P\right[X_1,X_2,\Gamma ]\parallel P[X_1\left]P\right[X_2|X_1\left]P\right[\Gamma |X_2\left]\right)={\sum }_{x_1,x_2,\gamma }P(x_1,x_2,\gamma )\log \frac{P(\gamma |x_1,x_2)}{P\left(\gamma |x_2\right)}$

  which is a conditional mutual information:

  ${\sum }_{x_1,x_2,\gamma }P(x_1,x_2,\gamma )\log \frac{P(\gamma |x_1,x_2)}{P\left(\gamma |x_2\right)}=I(X_1;\Gamma |X_2)$

  what results in:

  $D_{KL}\left(P\right[X_1,X_2,\Gamma ]\parallel P[X_1\left]P\right[X_2|X_1\left]P\right[\Gamma |X_2\left]\right)=I(X_1;\Gamma |X_2)\le ϵ$

• quetzal_rainbowMay 18, 2025, 4:31 AM

  8 points

  0

  in reply to: leogao’s comment on: leogao’s Shortform

  how should you pick which reference class to use

  You shouldn’t. This epistemic bath has no baby in it and we should throw water out of it.

• quetzal_rainbowMay 17, 2025, 12:23 PM

  9 points

  7

  in reply to: Towards_Keeperhood’s comment on: Si­mon Skade’s Shortform

  It’s really sad that we still don’t have bookmarks for comments

• quetzal_rainbowMay 15, 2025, 7:03 AM

  5 points

  3

  in reply to: Matthew Barnett’s comment on: AI Doomerism in 1879

  No, the point is that AI x-risk is commonsensical. “If you drink much from a bottle marked poison it is certain to disagree with you sooner or later” even if you don’t know mechanism of action of poison. We don’t expect Newtonian mechanics to prove that hitting yourself with a brick is quite safe, if we’d found that Newtonian mechanics predicts hitting yourself with a brick to be safe, it would be a huge evidence for Newtonian mechanics to be wrong. Good theories usually support common intuitions.

  The other thing here is an isolated demand for rigor: there is no “technical understanding of today’s deep learning systems” which predicts, say, success of AGI labs or that their final products are going to be safe.

• quetzal_rainbowMay 14, 2025, 7:09 AM

  2 points

  0

  in reply to: cubefox’s comment on: quet­zal_rain­bow’s Shortform

  “For a while” is usually, like, day for me. Sometimes even hours. I don’t think that whatever damage other addictions inflict on cognitive function is that much easy to reverse.

• quetzal_rainbowMay 14, 2025, 3:17 AM

  2 points

  0

  in reply to: cubefox’s comment on: quet­zal_rain­bow’s Shortform

  It doesn’t explain why I fully regain concentration ability after abstaining for a while?

• quetzal_rainbowMay 13, 2025, 6:37 AM

  5 points

  0

  on: quet­zal_rain­bow’s Shortform

  In my personal experience, the main reason why social media causes cognitive decline is fatigue. Evidence from personal experience: like many social media addicts, I struggle with maintaining concentration on books. If I stop using social media for a while, I regain the full ability to concentrate without drawbacks—in a sense, “I suddenly become capable of reading 1,000 pages of a book in two days, which I had been trying to start for two months.”

  The reason why social media is addictive to me, I think, is the following process:

  1. Social media is entertaining;

  2. It becomes cognitively taxing very quickly, such that I lose the stamina to do anything else;

  3. I’m bored but incapable of doing anything else;

  4. I’m stuck in social media.

  “Social media causes fatigue” is also a useful frame to motivate myself not to use social media. “It’s going to have bad long-term consequences” doesn’t motivate me much; “you are going to be so, so tired afterwards” immediately triggers aversion.

• quetzal_rainbowMay 12, 2025, 12:20 PM

  2 points

  0

  in reply to: CronoDAS’s comment on: Game the­ory of “Nu­clear Pri­soner’s Dilemma”—on nuk­ing rocks

  almost all the other agents it expects to encounter are CDT agents

  Given this particular setup (you both get source codes of each other and make decision simultaneously without any means to verify choices of counterparty until outcomes happen), you shouldn’t self-modify into extortionist, because CDT agents always defect, because no amount of reasoning about source code can causally affect your decision and D-D is Nash equilibrium. CDT agents can expect with high probability to meet extortionist in the future and self-modify into weird Son-of-CDT agent, which gives in to extortion, but for this setup to work in any non-trivial way you should be at least EDT-ish.

  But yes, general principle here is “evaluate how much other player decision procedure is logically influenced by my decision procedure, calculate expected value, act accordingly”. The same is true for situation when you decide about self-modification.

  For example, if you think that modifying into extortionist is a good policy, it can lead to situation where everyone is extortionist and everybody nukes each other.

• quetzal_rainbowMay 12, 2025, 11:45 AM

  4 points

  0

  on: Game the­ory of “Nu­clear Pri­soner’s Dilemma”—on nuk­ing rocks

  Let’s suppose that you look at the code of your counterparty which says “I’ll Nuke you unless you Cooperate in which case I Defect”, call it “extortionist”. You have two hypotheses here:

  1. Your counterparty deliberately modified its decision-making procedures in hope to extort more utility;

  2. This decision-making procedure is a genuine result of some weird evolutionary/​learning process.

  If you can’t actually get any evidence in favor of each hypothesis, you go with your prior and do whatever is the best from the standpoint of UDT/​FDT/​LDT counterfactual operationalization. I.e., let’s suppose payoffs are:

  1. Cooperate:Cooperate − 10:10

  2. Cooperate:Defect − 20:2 15:2

  3. Defect:Defect − 5:5

  4. Nuke - −100:-100.

  You are playing against extortionist counterparty. Prior probability of extortionist from hypothesis 2 is x. Extortionists from hypothesis 1 can perfectly predict your responce in their decision to self-modify and cover tracks. If they decide to not self-modify, they choose to cooperate conditional on your cooperation. Let’s call policy “Nuke extortionist, cooperate with non-extortionist” ${\pi }_1$ and “Cooperate with both” ${\pi }_2$

  From here, your UDT-expected utility is:

  $EU\left({\pi }_1\right)=-100x+10(1-x)$

  $EU\left({\pi }_2\right)=2$

  Therefore, you should choose ${\pi }_1$ if

  $-100x+10(1-x)>2$

  i.e.

  $x<\frac{4}{55}\approx 7.27\%$

  And 7.27% is a really high frequency of “natural” extortionists, I won’t expect it to be this high.

• quetzal_rainbowMay 10, 2025, 8:23 AM

  2 points

  0

  in reply to: Noosphere89’s comment on: A Bear Case: My Pre­dic­tions Re­gard­ing AI Progress

  I think the difference between real bureaucracies and HCH is that in real functioning bureaucracies should be elements capable to say “screw this arbitrary problem factorization, I’m doing what’s useful” and bosses of bureaucracy should be able to say “we all understand that otherwise system wouldn’t be able to work”.

• quetzal_rainbowMay 4, 2025, 7:48 PM

  11 points

  6

  on: In­ter­pretabil­ity Will Not Reli­ably Find De­cep­tive AI

  There is a conceptual path for interpretability to lead to reliability: you can understand model in sufficient details to know how it produces intelligence and then make another model out of interpreted details. Obviously, it’s not something that we can expect to happen anytime soon, but it’s something that army of interpretability geniuses in datacenter could do.

• quetzal_rainbowMay 4, 2025, 8:58 AM

  16 points

  8

  on: The Ukraine War and the Kill Market

  I think current Russia-Ukraine war is a perfect place to implement such system. It’s an attrition war, there is not many goals which are not reduced to “kill and destroy as many as you can”. There is a strategic aspect: Russia pays exorbitant compensations to families of killed soldiers, decades of income for poor regions. So, when Russian soldier dies, two things can happen:

  1. Russian government dumps unreasonable amount of money on the market, contributing to inflation;

  2. Russian government fails to pay (for 1000 and 1 stupid bureaucratic reasons), which erodes trust of would-be soldiers and reduces Russia’s mobilization potential.

  I can easily see how such system could fail terribly in, say, Afghanistan (if you paid for every killed terrorist, there is an easy loophole “kill civilian, say they are terrorist”). It’s fine for current stage of Ukraine war.

  Also I don’t see how kill markets contribute to ability of military to coup. Payments are made in purely virtual points, soldiers can’t spend them on something else.

• quetzal_rainbowMay 3, 2025, 10:10 PM

  3 points

  0

  in reply to: tailcalled’s comment on: Ju­lian Brad­shaw’s Shortform

  By “artificial employee” I mean “something than can fully replace human employee, including their agentic capabilities”. And, of course, it should be much more useful than generic AI chatbot, it should be useful like owning Walmart (1,200,000 employees) is useful.

• quetzal_rainbowMay 3, 2025, 8:04 PM

  2 points

  0

  in reply to: tailcalled’s comment on: Ju­lian Brad­shaw’s Shortform

  “Set up a corporation with a million of artificial employees” is pretty legible, but human amount of agency is catastrophically insufficient for it.