RSS

Optimization

TagLast edit: Dec 30, 2024, 9:33 AM by Dakara

Optimization is any kind of process that systematically comes up with solutions that are better than the solution used before. More technically, this kind of process moves the world into a specific and unexpected set of states by searching through a large search space, hitting small and low probability targets. When this process is gradually guided by some agent into some specific state, through searching specific targets, we can say it prefers that state.

The best way to exemplify an optimization process is through a simple example: Eliezer Yudkowsky suggests natural selection is such a process. Through an implicit preference – better replicators – natural selection searches all the genetic landscape space and hit small targets: efficient mutations.

Consider the human being. We are a highly complex object with a low probability to have been created by chance—natural selection, however, over millions of years, built up the infrastructure needed to build such a functioning body. This body, as well as other organisms, had the chance (was selected) to develop because it is in itself a rather efficient replicator suitable for the environment where it came up.

Or consider the famous chessplaying computer, Deep Blue. Outside of the narrow domain of selecting moves for chess games, it can’t do anything impressive: but as a chessplayer, it was massively more effective than virtually all humans. It has a high optimization power in the chess domain but almost none in any other field. Humans or evolution, on the other hand, are more domain-general optimization processes than Deep Blue, but that doesn’t mean they’re more effective at chess specifically. (Although note in what contexts this optimization process abstraction is useful and where it fails to be useful: it’s not obvious what it would mean for “evolution” to play chess, and yet it is useful to talk about the optimization power of natural selection, or of Deep Blue.)

Measuring Optimization Power

One way to think mathematically about optimization, like evidence, is in information-theoretic bits. The optimization power is the amount of surprise we would have in the result if there were no optimization process present. Therefore we take the base-two logarithm of the reciprocal of the probability of the result. A one-in-a-million solution (a solution so good relative to your preference ordering that it would take a million random tries to find something that good or better) can be said to have log_2(1,000,000) = 19.9 bits of optimization. Compared to a random configuration of matter, any artifact you see is going to be much more optimized than this. The math describes only laws and general principles for reasoning about optimization; as with probability theory, you oftentimes can’t apply the math directly.

Further Reading & References

See also

The ground of optimization

Alex FlintJun 20, 2020, 12:38 AM
248 points
80 comments27 min readLW link1 review

Mea­sur­ing Op­ti­miza­tion Power

Eliezer YudkowskyOct 27, 2008, 9:44 PM
89 points
37 comments6 min readLW link

Optimization

Eliezer YudkowskySep 13, 2008, 4:00 PM
55 points
45 comments5 min readLW link

Op­ti­miza­tion Amplifies

Scott GarrabrantJun 27, 2018, 1:51 AM
114 points
12 comments4 min readLW link

Selec­tion vs Control

abramdemskiJun 2, 2019, 7:01 AM
172 points
26 comments11 min readLW link2 reviews

DL to­wards the un­al­igned Re­cur­sive Self-Op­ti­miza­tion attractor

jacob_cannellDec 18, 2021, 2:15 AM
32 points
22 comments4 min readLW link

Aiming at the Target

Eliezer YudkowskyOct 26, 2008, 4:47 PM
40 points
40 comments5 min readLW link

Risks from Learned Op­ti­miza­tion: Introduction

May 31, 2019, 11:44 PM
187 points
42 comments12 min readLW link3 reviews

Thoughts and prob­lems with Eliezer’s mea­sure of op­ti­miza­tion power

Stuart_ArmstrongJun 8, 2012, 9:44 AM
36 points
24 comments5 min readLW link

The Op­ti­mizer’s Curse and How to Beat It

lukeprogSep 16, 2011, 2:46 AM
100 points
84 comments3 min readLW link

Bot­tle Caps Aren’t Optimisers

DanielFilanAug 31, 2018, 6:30 PM
100 points
23 comments3 min readLW link1 review
(danielfilan.com)

Beren’s “De­con­fus­ing Direct vs Amor­tised Op­ti­mi­sa­tion”

DragonGodApr 7, 2023, 8:57 AM
52 points
10 comments3 min readLW link

Op­ti­mal­ity is the tiger, and agents are its teeth

VeedracApr 2, 2022, 12:46 AM
327 points
44 comments16 min readLW link1 review

Steer­ing systems

Max HApr 4, 2023, 12:56 AM
50 points
1 comment15 min readLW link

Good­hart’s Curse and Limi­ta­tions on AI Alignment

Gordon Seidoh WorleyAug 19, 2019, 7:57 AM
25 points
18 comments10 min readLW link

Op­ti­miza­tion Con­cepts in the Game of Life

Oct 16, 2021, 8:51 PM
75 points
16 comments10 min readLW link

Towards Mea­sures of Optimisation

May 12, 2023, 3:29 PM
53 points
37 comments4 min readLW link

Re­quire­ments for a STEM-ca­pa­ble AGI Value Learner (my Case for Less Doom)

RogerDearnaleyMay 25, 2023, 9:26 AM
33 points
3 comments15 min readLW link

Draft: In­fer­ring minimizers

Alex_AltairApr 1, 2023, 8:20 PM
9 points
0 comments1 min readLW link

Mean­ing & Agency

abramdemskiDec 19, 2023, 10:27 PM
91 points
17 comments14 min readLW link

Gaia Net­work: a prac­ti­cal, in­cre­men­tal path­way to Open Agency Architecture

Dec 20, 2023, 5:11 PM
22 points
8 comments16 min readLW link

Search­ing for Search­ing for Search

Rubi J. HudsonFeb 14, 2024, 11:51 PM
21 points
4 comments7 min readLW link

Difficulty classes for al­ign­ment properties

JozdienFeb 20, 2024, 9:08 AM
34 points
5 comments2 min readLW link

Towards a for­mal­iza­tion of the agent struc­ture problem

Alex_AltairApr 29, 2024, 8:28 PM
55 points
5 comments14 min readLW link

Fat Tails Dis­cour­age Compromise

niplavJun 17, 2024, 9:39 AM
53 points
5 comments1 min readLW link

Clar­ify­ing mesa-optimization

Mar 21, 2023, 3:53 PM
38 points
6 comments10 min readLW link

Draft: In­tro­duc­tion to optimization

Alex_AltairMar 26, 2023, 5:25 PM
43 points
8 comments16 min readLW link

Draft: The op­ti­miza­tion toolbox

Alex_AltairMar 28, 2023, 8:40 PM
20 points
1 comment7 min readLW link

Draft: De­tect­ing optimization

Alex_AltairMar 29, 2023, 8:17 PM
23 points
2 comments6 min readLW link

Con­se­quen­tial­ism is in the Stars not Ourselves

DragonGodApr 24, 2023, 12:02 AM
7 points
19 comments5 min readLW link

[Question] How Many Bits Of Op­ti­miza­tion Can One Bit Of Ob­ser­va­tion Un­lock?

johnswentworthApr 26, 2023, 12:26 AM
62 points
32 comments3 min readLW link

Game The­ory with­out Argmax [Part 1]

Cleo NardoNov 11, 2023, 3:59 PM
70 points
18 comments19 min readLW link

Game The­ory with­out Argmax [Part 2]

Cleo NardoNov 11, 2023, 4:02 PM
31 points
14 comments13 min readLW link

What is op­ti­miza­tion power, for­mally?

sbenthallOct 18, 2014, 6:37 PM
18 points
16 comments2 min readLW link

Math­e­mat­i­cal Mea­sures of Op­ti­miza­tion Power

Alex_AltairNov 24, 2012, 10:55 AM
8 points
16 comments5 min readLW link

Op­ti­miza­tion Provenance

Adele LopezAug 23, 2019, 8:08 PM
38 points
5 comments5 min readLW link

Two senses of “op­ti­mizer”

Joar SkalseAug 21, 2019, 4:02 PM
35 points
41 comments3 min readLW link

Is the term mesa op­ti­mizer too nar­row?

Matthew BarnettDec 14, 2019, 11:20 PM
39 points
21 comments1 min readLW link

Mesa-Op­ti­miz­ers vs “Steered Op­ti­miz­ers”

Steven ByrnesJul 10, 2020, 4:49 PM
45 points
7 comments8 min readLW link

Mesa-Op­ti­miz­ers and Over-op­ti­miza­tion Failure (Op­ti­miz­ing and Good­hart Effects, Clar­ify­ing Thoughts—Part 4)

DavidmanheimAug 12, 2019, 8:07 AM
15 points
3 comments4 min readLW link

The Credit As­sign­ment Problem

abramdemskiNov 8, 2019, 2:50 AM
103 points
40 comments17 min readLW link1 review

Fake Op­ti­miza­tion Criteria

Eliezer YudkowskyNov 10, 2007, 12:10 AM
73 points
21 comments3 min readLW link

Search ver­sus design

Alex FlintAug 16, 2020, 4:53 PM
109 points
40 comments36 min readLW link1 review

The First World Takeover

Eliezer YudkowskyNov 19, 2008, 3:00 PM
42 points
24 comments6 min readLW link

Life’s Story Continues

Eliezer YudkowskyNov 21, 2008, 11:05 PM
24 points
14 comments5 min readLW link

Utility Max­i­miza­tion = De­scrip­tion Length Minimization

johnswentworthFeb 18, 2021, 6:04 PM
213 points
44 comments5 min readLW link

Ap­pli­ca­tions for De­con­fus­ing Goal-Directedness

adamShimiAug 8, 2021, 1:05 PM
38 points
3 comments5 min readLW link1 review

A new defi­ni­tion of “op­ti­mizer”

ChantielAug 9, 2021, 1:42 PM
5 points
0 comments7 min readLW link

Mea­sure­ment, Op­ti­miza­tion, and Take-off Speed

jsteinhardtSep 10, 2021, 7:30 PM
48 points
4 comments13 min readLW link

In Defence of Op­ti­miz­ing Rou­tine Tasks

leogaoNov 9, 2021, 5:09 AM
47 points
6 comments3 min readLW link1 review

Ngo and Yud­kowsky on AI ca­pa­bil­ity gains

Nov 18, 2021, 10:19 PM
130 points
61 comments39 min readLW link1 review

Defin­ing “op­ti­mizer”

ChantielApr 17, 2021, 3:38 PM
9 points
6 comments1 min readLW link

Bits of Op­ti­miza­tion Can Only Be Lost Over A Distance

johnswentworthMay 23, 2022, 6:55 PM
31 points
18 comments2 min readLW link

Distributed Decisions

johnswentworthMay 29, 2022, 2:43 AM
66 points
6 comments6 min readLW link

Quan­tify­ing Gen­eral Intelligence

JasonBrownJun 17, 2022, 9:57 PM
9 points
6 comments13 min readLW link

Defin­ing Op­ti­miza­tion in a Deeper Way Part 1

J BostockJul 1, 2022, 2:03 PM
7 points
0 comments2 min readLW link

Notes on Simplicity

David GrossDec 2, 2020, 11:14 PM
9 points
0 comments7 min readLW link

Defin­ing Op­ti­miza­tion in a Deeper Way Part 2

J BostockJul 11, 2022, 8:29 PM
7 points
0 comments4 min readLW link

Defin­ing Op­ti­miza­tion in a Deeper Way Part 3

J BostockJul 20, 2022, 10:06 PM
8 points
0 comments2 min readLW link

Defin­ing Op­ti­miza­tion in a Deeper Way Part 4

J BostockJul 28, 2022, 5:02 PM
7 points
0 comments5 min readLW link

Ab­stract­ing The Hard­ness of Align­ment: Un­bounded Atomic Optimization

adamShimiJul 29, 2022, 6:59 PM
72 points
3 comments16 min readLW link

Vingean Agency

abramdemskiAug 24, 2022, 8:08 PM
62 points
14 comments3 min readLW link

“Nor­mal” is the equil­ibrium state of past op­ti­miza­tion processes

Alex_AltairOct 30, 2022, 7:03 PM
81 points
5 comments5 min readLW link

What I Learned Run­ning Refine

adamShimiNov 24, 2022, 2:49 PM
108 points
5 comments4 min readLW link

Don’t al­ign agents to eval­u­a­tions of plans

TurnTroutNov 26, 2022, 9:16 PM
45 points
49 comments18 min readLW link

Op­por­tu­nity Cost Blackmail

adamShimiJan 2, 2023, 1:48 PM
70 points
11 comments2 min readLW link
(epistemologicalvigilance.substack.com)

De­con­fus­ing Direct vs Amor­tised Optimization

berenDec 2, 2022, 11:30 AM
124 points
19 comments10 min readLW link

[Question] Do the Safety Prop­er­ties of Pow­er­ful AI Sys­tems Need to be Ad­ver­sar­i­ally Ro­bust? Why?

DragonGodFeb 9, 2023, 1:36 PM
22 points
42 comments2 min readLW link

Fun­da­men­tal Uncer­tainty: Chap­ter 4 - Why don’t we do what we think we should?

Gordon Seidoh WorleyAug 29, 2022, 7:25 PM
15 points
6 comments13 min readLW link

Op­ti­miza­tion Markets

StrivingForLegibilityDec 30, 2023, 1:24 AM
13 points
2 comments2 min readLW link

Op­ti­miza­tion and the Singularity

Eliezer YudkowskyJun 23, 2008, 5:55 AM
41 points
21 comments9 min readLW link

Op­ti­miza­tion hap­pens in­side the mind, not in the world

azsantoskJun 3, 2023, 9:36 PM
17 points
10 comments5 min readLW link

Ru­n­away Op­ti­miz­ers in Mind Space

silentbobJul 16, 2023, 2:26 PM
16 points
0 comments12 min readLW link

Go­ing Beyond Lin­ear Mode Con­nec­tivity: The Lay­er­wise Lin­ear Fea­ture Connectivity

zhanpeng_zhouJul 20, 2023, 5:38 PM
22 points
13 comments3 min readLW link
(openreview.net)

[Question] What are ex­am­ples of some­one do­ing a lot of work to find the best of some­thing?

chanamessingerJul 27, 2023, 3:58 PM
29 points
16 comments1 min readLW link

Op­ti­mi­sa­tion Mea­sures: Desider­ata, Im­pos­si­bil­ity, Proposals

Aug 7, 2023, 3:52 PM
36 points
9 comments1 min readLW link

Goldilocks and the Three Optimisers

dkl9Aug 17, 2023, 6:15 PM
−10 points
0 comments5 min readLW link
(dkl9.net)

Ob­serv­ing Optimization

Eliezer YudkowskyNov 21, 2008, 5:39 AM
12 points
28 comments6 min readLW link

Satis­ficers want to be­come maximisers

Stuart_ArmstrongOct 21, 2011, 4:27 PM
38 points
70 comments1 min readLW link

Evolu­tions Build­ing Evolu­tions: Lay­ers of Gen­er­ate and Test

plexFeb 5, 2021, 6:21 PM
12 points
1 comment6 min readLW link

Sur­pris­ing ex­am­ples of non-hu­man optimization

Jan_RzymkowskiJun 14, 2015, 5:05 PM
31 points
9 comments1 min readLW link

Bridg­ing Ex­pected Utility Max­i­miza­tion and Optimization

Daniel HerrmannAug 5, 2022, 8:18 AM
25 points
5 comments14 min readLW link

I missed the crux of the al­ign­ment prob­lem the whole time

zeshenAug 13, 2022, 10:11 AM
53 points
7 comments3 min readLW link

What’s Gen­eral-Pur­pose Search, And Why Might We Ex­pect To See It In Trained ML Sys­tems?

johnswentworthAug 15, 2022, 10:48 PM
153 points
18 comments10 min readLW link

Dis­cov­er­ing Agents

zac_kentonAug 18, 2022, 5:33 PM
73 points
11 comments6 min readLW link

Ac­ci­den­tal Optimizers

aysajanSep 22, 2021, 1:27 PM
7 points
2 comments3 min readLW link

The Carnot Eng­ine of Economics

StrivingForLegibilityAug 9, 2024, 3:59 PM
5 points
0 comments5 min readLW link

The sling­shot helps with learning

Wilson WuOct 31, 2024, 11:18 PM
33 points
0 comments8 min readLW link

In­ter­view with Bill O’Rourke—Rus­sian Cor­rup­tion, Putin, Ap­plied Ethics, and More

JohnGreerOct 27, 2024, 5:11 PM
3 points
0 comments6 min readLW link

The Three Warn­ings of the Zentradi

Trevor Hill-HandNov 21, 2024, 8:28 PM
11 points
1 comment5 min readLW link

Vi­sual demon­stra­tion of Op­ti­mizer’s curse

Roman MalovNov 30, 2024, 7:34 PM
25 points
3 comments7 min readLW link

MONA: Man­aged My­opia with Ap­proval Feedback

Jan 23, 2025, 12:24 PM
76 points
29 comments9 min readLW link

Safety Data Sheets for Op­ti­miza­tion Processes

StrivingForLegibilityJan 4, 2024, 11:30 PM
15 points
1 comment4 min readLW link

The Gears of Argmax

StrivingForLegibilityJan 4, 2024, 11:30 PM
11 points
0 comments3 min readLW link

Un­der­stand­ing Gra­di­ent Hacking

peterbarnettDec 10, 2021, 3:58 PM
41 points
5 comments30 min readLW link

Wild­fire of strategicness

TsviBTJun 5, 2023, 1:59 PM
38 points
19 comments1 min readLW link

Trans­form­ing my­opic op­ti­miza­tion to or­di­nary op­ti­miza­tion—Do we want to seek con­ver­gence for my­opic op­ti­miza­tion prob­lems?

tailcalledDec 11, 2021, 8:38 PM
12 points
1 comment5 min readLW link

Could Things Be Very Differ­ent?—How His­tor­i­cal In­er­tia Might Blind Us To Op­ti­mal Solutions

James Stephen BrownSep 11, 2024, 9:53 AM
5 points
0 comments8 min readLW link
(nonzerosum.games)

Hy­poth­e­sis: gra­di­ent de­scent prefers gen­eral circuits

Quintin PopeFeb 8, 2022, 9:12 PM
46 points
26 comments11 min readLW link

Op­ti­miz­ing crop plant­ing with mixed in­te­ger lin­ear pro­gram­ming in Stardew Valley

hapaninApr 5, 2022, 6:42 PM
68 points
4 comments7 min readLW link

Think­ing about max­i­miza­tion and corrigibility

James PayorApr 21, 2023, 9:22 PM
63 points
4 comments5 min readLW link

In­ter­lude: But Who Op­ti­mizes The Op­ti­mizer?

Paul BricmanSep 23, 2022, 3:30 PM
15 points
0 comments10 min readLW link

Ar­chi­tec­ture-aware op­ti­mi­sa­tion: train ImageNet and more with­out hyperparameters

Chris MingardApr 22, 2023, 9:50 PM
6 points
2 comments2 min readLW link

Ad­ver­sar­ial at­tacks and op­ti­mal control

JanMay 22, 2022, 6:22 PM
17 points
7 comments8 min readLW link
(universalprior.substack.com)

Non-re­solve as Resolve

Linda LinseforsJul 10, 2018, 11:31 PM
15 points
1 comment2 min readLW link

(Struc­tural) Sta­bil­ity of Cou­pled Optimizers

Paul BricmanSep 30, 2022, 11:28 AM
25 points
0 comments10 min readLW link

One bit of ob­ser­va­tion can un­lock many of op­ti­miza­tion—but at what cost?

dr_sApr 29, 2023, 10:53 AM
42 points
4 comments5 min readLW link

De­gen­era­cies are sticky for SGD

Jun 16, 2024, 9:19 PM
56 points
1 comment16 min readLW link

Adam Op­ti­mizer Causes Priv­ileged Ba­sis in Trans­former LM Resi­d­ual Stream

Sep 6, 2024, 5:55 PM
70 points
7 comments4 min readLW link

The Hu­man’s Role in Mesa Optimization

silentbobMay 9, 2024, 12:07 PM
5 points
0 comments2 min readLW link

No free lunch the­o­rem is irrelevant

CatneeOct 4, 2022, 12:21 AM
18 points
7 comments1 min readLW link

Op­ti­miza­tion and Ad­e­quacy in Five Bullets

james.lucassenJun 6, 2022, 5:48 AM
35 points
2 comments4 min readLW link
(jlucassen.com)

Break­ing Down Goal-Directed Behaviour

Oliver SourbutJun 16, 2022, 6:45 PM
11 points
1 comment2 min readLW link

Per­ils of op­ti­miz­ing in so­cial contexts

owencbJun 16, 2022, 5:40 PM
50 points
1 comment2 min readLW link

Plans Are Pre­dic­tions, Not Op­ti­miza­tion Targets

johnswentworthOct 20, 2022, 9:17 PM
108 points
20 comments4 min readLW link1 review

The Limits of Automation

milkandcigarettesJun 23, 2022, 6:03 PM
5 points
1 comment5 min readLW link
(milkandcigarettes.com)

Some Prob­lems with Or­di­nal Op­ti­miza­tion Frame

Mateusz BagińskiMay 6, 2024, 5:28 AM
9 points
0 comments7 min readLW link

Is Gen­eral In­tel­li­gence “Com­pact”?

DragonGodJul 4, 2022, 1:27 PM
27 points
6 comments22 min readLW link

De­grees of Freedom

sarahconstantinApr 2, 2019, 9:10 PM
103 points
31 comments11 min readLW link
(srconstantin.wordpress.com)

He­donic asymmetries

paulfchristianoJan 26, 2020, 2:10 AM
98 points
22 comments2 min readLW link
(sideways-view.com)

De­mons in Im­perfect Search

johnswentworthFeb 11, 2020, 8:25 PM
107 points
21 comments3 min readLW link

Tes­sel­lat­ing Hills: a toy model for demons in im­perfect search

DaemonicSigilFeb 20, 2020, 12:12 AM
97 points
18 comments2 min readLW link

Align­ing a toy model of optimization

paulfchristianoJun 28, 2019, 8:23 PM
53 points
25 comments3 min readLW link

Siren wor­lds and the per­ils of over-op­ti­mised search

Stuart_ArmstrongApr 7, 2014, 11:00 AM
83 points
418 comments7 min readLW link

When Can Op­ti­miza­tion Be Done Safely?

StrivingForLegibilityDec 30, 2023, 1:24 AM
12 points
0 comments3 min readLW link

Notes on Antelligence

AurigenaMay 13, 2023, 6:38 PM
2 points
0 comments9 min readLW link

Worse Than Random

Eliezer YudkowskyNov 11, 2008, 7:01 PM
46 points
102 comments12 min readLW link

Don’t de­sign agents which ex­ploit ad­ver­sar­ial inputs

Nov 18, 2022, 1:48 AM
72 points
64 comments12 min readLW link

Effi­cient Cross-Do­main Optimization

Eliezer YudkowskyOct 28, 2008, 4:33 PM
54 points
38 comments5 min readLW link

Break­ing the Op­ti­mizer’s Curse, and Con­se­quences for Ex­is­ten­tial Risks and Value Learning

Roger DearnaleyFeb 21, 2023, 9:05 AM
10 points
1 comment23 min readLW link

Ex­tinc­tion Risks from AI: In­visi­ble to Science?

Feb 21, 2024, 6:07 PM
24 points
7 comments1 min readLW link
(arxiv.org)
No comments.