Mikhail Samin

Karma: 1,049

My name is Mikhail Samin (diminutive Misha, @Mihonarium on Twitter, @misha in Telegram).

Humanity’s future can be huge and awesome; losing it would mean our lightcone (and maybe the universe) losing most of its potential value.

My research is currently focused on AI governance and improving the understanding of AI and AI risks among stakeholders. I also have takes on what seems to me to be the very obvious shallow stuff about the technical AI notkilleveryoneism; but many AI Safety researchers told me our conversations improved their understanding of the alignment problem.

I believe a capacity for global regulation is necessary to mitigate the risks posed by future general AI systems. I’m happy to talk to policymakers and researchers about ensuring AI benefits society.

I took the Giving What We Can pledge to donate at least 10% of my income for the rest of my life or until the day I retire (why?).

In the past, I’ve launched the most funded crowdfunding campaign in the history of Russia (it was to print HPMOR! we printed 21 000 copies =63k books) and founded audd.io, which allowed me to donate >$100k to EA causes, including >$60k to MIRI.

[Less important: I’ve also started a project to translate 80,000 Hours, a career guide that helps to find a fulfilling career that does good, into Russian. The impact and the effectiveness aside, for a year, I was the head of the Russian Pastafarian Church: a movement claiming to be a parody religion, with 200 000 members in Russia at the time, trying to increase separation between religious organisations and the state. I was a political activist and a human rights advocate. I studied relevant Russian and international law and wrote appeals that won cases against the Russian government in courts; I was able to protect people from unlawful police action. I co-founded the Moscow branch of the “Vesna” democratic movement, coordinated election observers in a Moscow district, wrote dissenting opinions for members of electoral commissions, helped Navalny’s Anti-Corruption Foundation, helped Telegram with internet censorship circumvention, and participated in and organized protests and campaigns. The large-scale goal was to build a civil society and turn Russia into a democracy through nonviolent resistance. This goal wasn’t achieved, but some of the more local campaigns were successful. That felt important and was also mostly fun- except for being detained by the police. I think it’s likely the Russian authorities will imprison me if I ever visit Russia.]

Mikhail Samin 6 Dec 2024 9:25 UTC
4 points
0
on: I Finally Worked Through Bayes’ Theorem (Personal Achievement)
If you want to understand Bayes theorem, know why you’re applying it, and use it intuitively, try https://arbital.com/p/bayes_rule/?l=1zq

Mikhail Samin 3 Dec 2024 11:07 UTC
9 points
0
on: (The) Lightcone is nothing without its people: LW + Lighthaven’s big fundraiser
I’ve donated $1000. Thank you for your work.

Mikhail Samin 25 Nov 2024 14:00 UTC
1 point
1
on: “The Solomonoff Prior is Malign” is a special case of a simpler argument
I’d bet 1:1 that, conditional on building a CEV-aligned AGI, we won’t consider this type of problem to have been among the top-5 hardest to solve.

Reality-fluid in our universe should pretty much add up to normality, to the extent it’s Tegmark IV (and it’d be somewhat weird for your assumed amount of compute and simulations to exist but not for all computations/maths objects to exist).

If a small fraction of computers simulating this branch stop, this doesn’t make you stop. All configurations of you are computed; simulators might slightly change the relative likelihood of currently being in one branch or another, but they can’t really terminate you

Furthermore, our physics seems very simple, and most places that compute us probably do it faithfully, on the level of the underlying physics, with no interventions.

I feel like thinking of reality-fluid as just inverse relationship to the description length might produce wrong intuitions. In Tegmark IV, you still get more reality-fluid if someone simulates you; and it’s less intuitive why this translates into shorter description length. It might be better to think of it as: if all computation/maths exists and I open my eyes in a random place, how often would that happen here? All the places run this world give some of their reality-fluid to this world. If a place visible from a bunch of other places starts to simulate this universe, it will be visible from slightly more places.

You can think of the entire object of everything, with all of its parts being simulated in countless other parts; or imagine a Markov process, but with worlds giving each other reality-fluid.

In that sense, the resource that we have is the reality-fluid of our future lightcone; it is our endowment, and we can use it to maximize the overall flourishing in the entire structure.

If we make decisions based on how good the overall/average use of the reality-fluid would be, you’ll gain less reality-fluid by manipulating our world the way described in the post than you’ll spend on the manipulation. It’s probably better for you to trade with us instead.

(I also feel like there might be a reasonable way to talk about causal descendants, where the probabilities are whatever abides the math of probability theory and causality down the nodes we care about, instead of being the likelihoods of opening eyes in different branches in a particular moment of evaluation.)

Mikhail Samin 7 Nov 2024 15:56 UTC
2 points
0
in reply to: Christopher King’s comment on: LDT (and everything else) can be irrational
It’s reasonable to consider two agents playing against each other. “Playing against your copy” is a reasonable problem. ($9 rocks get 0 in this problem, LDTs probably get $5.)

Newcomb, Parfit’s hitchhiker, smoking, etc. are all very reasonable problems that essentially depend on the buttons you press when you play the game. It is important to get these problems right.

But playing against LDT is not necessarily in the “fair problem class” because the game might behave differently depending on your algorithm/on how you arrive at taking actions, and not just depending on your actions.

Your version of it- playing against an LDT- is indeed different from playing against a game that looks at whether we’re an alphabetizing agent and pick X instead of Y because X<Y and not because we looked at the expected utility: we would want LDT to perform optimally in this game. But the reason LDT-created-rock loses to a natural rock here isn’t fundamentally different from the reason LDT loses to an alphabetizing agent in the other game and it is known that you can construct a game like that where LDT will lose to something else. You can make the game description sound more natural, but I feel like there’s a sharp divide between the “fair problem class” problems and others.

(I also think that in real life, where this game might play out, there isn’t really a choice we can make, to make our AI a $9 rock instead of an LDT agent; because when we do that due to the rock’s better performance in this game, our rock gets slightly less than $5 in EV instead of getting $9; LDT doesn’t perform worse than other agents we could’ve chosen in this game.)

Mikhail Samin 7 Nov 2024 10:52 UTC
6 points
3
on: LDT (and everything else) can be irrational
Playing ultimatum game against an agent that gives in to $9 from rocks but not from us is not in the fair problem class, as the payoffs depend directly on our algorithm and not just on our choices and policies.

https://arbital.com/p/fair_problem_class/

A simpler game is “if you implement or have ever implemented LDT, you get $0; otherwise, you get $100”.

LDT decision theories are probably the best decision theories for problems in the fair problem class.

(Very cool that you’ve arrived at the idea of this post independently!)

Mikhail Samin 5 Oct 2024 7:42 UTC
6 points
4
on: If I have some money, whom should I donate it to in order to reduce expected P(doom) the most?
Do you want to donate to alignment specifically? IMO AI governance efforts are significantly more p(doom)-reducing than technical alignment research; it might be a good idea to, e.g., donate to MIRI, as they’re now focused on comms & governance.

Mikhail Samin 1 Oct 2024 10:46 UTC
2 points
0
in reply to: Alexander Gietelink Oldenziel’s comment on: Alexander Gietelink Oldenziel’s Shortform
- Probability is in the mind. There’s no way to achieve entanglement between what’s necessary to make these predictions and the state of your brain, so for you, some of these are random.
- In multi-worlds, the Turing machine will compute many copies of you, and there might be more of those who see one thing when they open their eyes than of those who see another thing. When you open your eyes, there’s some probability of being a copy that sees one thing and a copy that sees the other thing. In a deterministic world with many copies of you, there’s “true” randomness in where you end up opening your eyes.

Mikhail Samin 30 Sep 2024 15:03 UTC
3 points
0
in reply to: Bunthut’s comment on: How to Give in to Threats (without incentivizing them)
If you are a smart individual in todays society, you shouldn’t ignore threats of punishment
If today’s society consisted mostly of smart individuals, they would overthrow the government that does something unfair instead of giving in to its threats.
Should you update your idea of fairness if you get rejected often?
Only if you’re a kid who’s playing with other human kids (which is the scenario described in the quoted text), and converging on fairness possibly includes getting some idea of how much effort various things take different people.
If you’re an actual grown-up (not that we have those) and you’re playing with aliens, you probably don’t update, and you certainly don’t update in the direction of anything asymmetric.

Mikhail Samin 29 Sep 2024 10:09 UTC
2 points
0
in reply to: Thomas Kwa’s comment on: How to Give in to Threats (without incentivizing them)
Very funny that we had this conversation a couple of weeks prior to transparently deciding that we should retaliate with p=.7!

Mikhail Samin 28 Sep 2024 0:00 UTC
3 points
0
in reply to: quetzal_rainbow’s comment on: [Completed] The 2024 Petrov Day Scenario
huh, are you saying my name doesn’t sound WestWrongian

Mikhail Samin 27 Sep 2024 11:05 UTC
11 points
0
on: [Completed] The 2024 Petrov Day Scenario
The game was very fun! I played General Carter.
Some reflections:
- I looked at the citizens’ comments, and while some of them were notable (@Jesse Hoogland calling for the other side to nuke us <3), I didn’t find anything important after the game started- I considered the overall change in their karma if one or two sides get nuked, but comments from the citizens were not relevant to decision-making (including threats around reputation or post downvotes).
- It was great to see the other side sharing my post internally to calculate the probability of retaliation if we nuke them
- It was a good idea to ask whether looking at the source code is ok and then share it, which made it clear Petrovs won’t necessarily have much information on whether the missiles they see are real.
- The incentives (+350..1000 LW karma) weren’t strong enough to make the generals try to win by making moves instead of winning by not playing, but I’m pretty happy with the outcome.
- It’s awesome to be able to have transparent and legible decision-making processes and trust each other’s commitments.
- One of the Petrovs preferred defeat to mutual destruction- I’m curious whether they’d report nukes if they were sure the nukes were real.
- In real life, diplomatic channels would not be visible to the public. I think with stronger incentives, the privacy of diplomatic channels could’ve made the outcomes more interesting (though for everyone else, there’d be less entertainment throughout the game).
- It was a good idea to ask the organizers if it’s ok to look at the source code and then post the link in the comments. Transparency into the fact that a side knows if they launched nukes meant we were able to complete the game peacefully.
I’d claim that we kinda won the soft power competition:
- we proposed commitments to not first-strike;
- we bribed everyone (and then the whole website went down, but funnily enough, that didn’t affect our war room and diplomatic channel- deep in our bunkers, we were somehow protected from the LW downtime);
- we proposed commitments to report through the diplomatic channel if someone on our side made a launch, which disincentivized individual generals from unilaterally launching the nukes, allowed Petrovs to ignore scary incoming missiles, and possibly was necessary to win the game;
- finally, after a general on their side said they’ll triumph economically and culturally, General Brooks wrote a poem, and I generated a cultural gift, which made generals on the other side feel inspired. That was very wholesome and was highlighted in Ben Paces’s comment and the subsequent post with a retrospective after the game ended. I think our side triumphed here!
Thanks everyone for the experience!

Mikhail Samin 26 Sep 2024 17:24 UTC
2 points
0
in reply to: Eli Tyre’s comment on: How to Give in to Threats (without incentivizing them)
Thanks!

The post is mostly trying to imply things about AI systems and agents in a larger universe, like “aliens and AIs usually coordinate with other aliens annd AIs, and ~no commitment races happen”.

For humans, it’s applicable to bargaining and threat-shape situations. I think bargaining situations are common; clearly threat-shaped situations are rarer.

I think while taxes in our world are somewhat threat-shaped, it’s not clear they’re “unfair”- I think we want everyone to pay them so that good governments work and provide value. But if you think taxes are unfair, you can leave the country and pay some different taxes somewhere else instead of going to jail.

The society’s stance towards crime- preventing it via the threat of punishment- is not what would work on smarter people: it makes sense to prevent people from committing more crimes by putting them in jails or not trading with them, but the threat of punishment that exists only to prevent an agent from doing something won’t work on smarter agents.

Mikhail Samin 14 Sep 2024 12:02 UTC
3 points
0
in reply to: Richard_Kennaway’s comment on: How to Give in to Threats (without incentivizing them)
A smart agent can simply make decisions like a negotiator with restrictions on the kinds of terms it can accept, without having to spawn a “boulder” to do that.
You can just do the correct thing, without having to separate yourself into parts that do things correctly and a part that tries to not look at the world and spawns correct-thing-doers.
In Parfit’s Hitchhiker, you can just pay once you’re there, without precommiting/rewriting yourself into an agent that pays. You can just do the thing that wins.
Some agents can’t do the things that win and would have to rewrite themselves into something better and still lose in some problems, but you can be an agent that wins, and gradient descent probably crystallizes something that wins into what is making the decisions in smart enough things.

Mikhail Samin 13 Sep 2024 15:56 UTC
2 points
2
in reply to: Multicore’s comment on: How to Give in to Threats (without incentivizing them)
Yep! If someone is doing things because it’s in their best interests and not to make you do something (and they’re not a result of someone else shaping themselves into them to cause you do something, whereas some previous agent wouldn’t actually prefer the thing the new one prefers, that you don’t want to happen), then this is not a threat.

Mikhail Samin 13 Sep 2024 11:25 UTC
3 points
1
in reply to: Wei Dai’s comment on: How to Give in to Threats (without incentivizing them)
By a stone, I meant a player with very deterministic behavior in a game with known payoffs, named this way after the idea of cooperate-stones in prisoner’s dilemma (with known payoffs).

I think to the extent there’s no relationship between giving in to a boulder/implemeting some particular decision theory and having this and other boulders thrown at you, UDT and FDT by default swerve (and probably don’t consider the boulders to be threatening them, and it’s not very clear in what sense this is “giving in”); to the extent it sends more boulders their way, they don’t swerve.

If making decisions some way incentivizes other agents to become less like LDTs and more like uncooperative boulders, you can simply not make decisions that way. (If some agents actually have an ability to turn into animals and you can’t distinguish the causes behind an animal running at you, you can sometimes probabilistically take out your anti-animal gun and put them to sleep.)

Do you maybe have a realistic example where this would realistically be a problem?

I’d be moderately surprised if UDT/FDT consider something to be a better policy than what’s described in the post.

Edit: to add, LDTs don’t swerve to boulders that were created to influence the LDT agent’s responses. If you turn into a boulder because you expect some agents among all possible agents to swerve, this is a threat, and LDTs don’t give in to those boulders (and it doesn’t matter whether or not you tried to predict the behavior of LDTs in particular). If you believed LDT agents or agents in general would swerve against a boulder, and that made you become a boulder, LDT agents obviously don’t swerve to that boulder. They might swerve to boulders that are actually natural boulders caused by the very simple physics no one influenced to cause the agents to do something. They also pay their rent- because they’d be evicted otherwise, not for the reason of getting rent from them under the threat of eviction but for the reason of getting rent from someone else, and they’re sure there were no self-modifications to make it look this way.

Mikhail Samin 12 Sep 2024 20:37 UTC
1 point
0
in reply to: Thomas Kwa’s comment on: How to Give in to Threats (without incentivizing them)
(It is pretty important to very transparently respond with a nuclear strike to a nuclear strike. I think both Russia and the US are not really unpredictable in this question. But yeah, if you have nuclear weapons and your opponents don’t, you might want to be unpredictable, so your opponent is more scared of using conventional weapons to destroy you. In real-life cases with potentially dumb agents, it might make sense to do this.)

Mikhail Samin 12 Sep 2024 19:49 UTC
5 points
0
in reply to: Richard_Kennaway’s comment on: How to Give in to Threats (without incentivizing them)
Your solution works! It’s not exploitable, and you get much more than 0 in expectation! Congrats!
Eliezer’s solution is better/optimal in the sense that it accepts with the highest probability a strategy can use without becoming exploitable. If offered ⁴⁄₁₀, you accept with p=40%; the optimal solution accepts with p=83% (or slightly less than ⁵⁄₆); if offered ¹⁄₁₀, it’s p=10% vs. p=55%. The other player’s payout is still maximized at 5, but everyone gets the payout a lot more often!

Mikhail Samin 12 Sep 2024 19:40 UTC
1 point
0
in reply to: Dagon’s comment on: How to Give in to Threats (without incentivizing them)
It’s not how the game would be played between dath ilan and true aliens
This is a reference to “Sometimes they accept your offer and then toss a jellychip back to you”. Between dath ilan and true aliens, you do the same except for tossing the jellychip when you think you got more than what would’ve been fair. See True Prisoner’s Dilemma.

Mikhail Samin 12 Sep 2024 19:24 UTC
1 point
0
in reply to: IC Rainbow’s comment on: How to Give in to Threats (without incentivizing them)
I guess when criminals and booing bystanders are not as educated as dath ilani children, some real-world situations might get complicated. Possibly, transparent stats about the actions you’ve taken in similar situations might serve the same purpose even if you don’t broadcast throwing your dice on live TV. Or it might make sense to transparently never give in to some kinds of threats in some sorts of real-life situations.

Mikhail Samin 26 Aug 2024 2:11 UTC
1 point
0
in reply to: Andrew Currall’s comment on: Why you should be using a retinoid
There’s certainly a huge difference in the UV levels between winters and summers. Even during winters, if you go out while the UV index isn’t 0, you should wear sunscreen if you’re on tretinoin. (I’m deferring to a dermatologist and haven’t actually checked the sources though.)