RSS

Sin­gu­lar Learn­ing Theory

TagLast edit: Jun 20, 2023, 11:25 PM by DanielFilan

Singluar learning theory is a theory that applies algebraic geometry to statistical learning theory, developed by Sumio Watanabe. Reference textbooks are “the grey book”, Algebraic Geometry and Statistical Learning Theory, and “the green book”, Mathematical Theory of Bayesian Statistics.

DSLT 2. Why Neu­ral Net­works obey Oc­cam’s Razor

Liam CarrollJun 18, 2023, 12:23 AM
24 points
14 comments17 min readLW link

DSLT 3. Neu­ral Net­works are Singular

Liam CarrollJun 20, 2023, 8:20 AM
29 points
5 comments19 min readLW link

DSLT 1. The RLCT Mea­sures the Effec­tive Di­men­sion of Neu­ral Networks

Liam CarrollJun 16, 2023, 9:50 AM
54 points
10 comments13 min readLW link

DSLT 0. Distill­ing Sin­gu­lar Learn­ing Theory

Liam CarrollJun 16, 2023, 9:50 AM
80 points
7 comments5 min readLW link

Neu­ral net­works gen­er­al­ize be­cause of this one weird trick

Jesse HooglandJan 18, 2023, 12:10 AM
181 points
34 comments15 min readLW link1 review
(www.jessehoogland.com)

Sin­gu­lar learn­ing the­ory: exercises

Zach FurmanAug 30, 2024, 8:00 PM
90 points
5 comments14 min readLW link

In­ves­ti­gat­ing the learn­ing co­effi­cient of mod­u­lar ad­di­tion: hackathon project

Oct 17, 2023, 7:51 PM
94 points
5 comments12 min readLW link

Growth and Form in a Toy Model of Superposition

Nov 8, 2023, 11:08 AM
89 points
7 comments14 min readLW link

An­nounc­ing Timaeus

Oct 22, 2023, 11:59 AM
188 points
15 comments4 min readLW link

Spooky ac­tion at a dis­tance in the loss landscape

Jan 28, 2023, 12:22 AM
61 points
4 comments7 min readLW link
(www.jessehoogland.com)

Ti­maeus’s First Four Months

Feb 28, 2024, 5:01 PM
173 points
6 comments6 min readLW link

DSLT 4. Phase Tran­si­tions in Neu­ral Networks

Liam CarrollJun 24, 2023, 5:22 PM
30 points
3 comments16 min readLW link

Gra­di­ent sur­fing: the hid­den role of regularization

Jesse HooglandFeb 6, 2023, 3:50 AM
37 points
9 comments14 min readLW link
(www.jessehoogland.com)

Gen­er­al­iza­tion, from ther­mo­dy­nam­ics to statis­ti­cal physics

Jesse HooglandNov 30, 2023, 9:28 PM
64 points
9 comments28 min readLW link

Stage­wise Devel­op­ment in Neu­ral Networks

Mar 20, 2024, 7:54 PM
96 points
1 comment11 min readLW link

Sim­ple ver­sus Short: Higher-or­der de­gen­er­acy and er­ror-correction

Daniel MurfetMar 11, 2024, 7:52 AM
110 points
8 comments13 min readLW link

AXRP Epi­sode 31 - Sin­gu­lar Learn­ing The­ory with Daniel Murfet

DanielFilanMay 7, 2024, 3:50 AM
72 points
4 comments71 min readLW link

Dialogue in­tro­duc­tion to Sin­gu­lar Learn­ing Theory

Olli JärviniemiJul 8, 2024, 4:58 PM
100 points
15 comments8 min readLW link

Ti­maeus is hiring!

Jul 12, 2024, 11:42 PM
67 points
6 comments2 min readLW link

AXRP Epi­sode 38.2 - Jesse Hoogland on Sin­gu­lar Learn­ing Theory

DanielFilanNov 27, 2024, 6:30 AM
34 points
0 comments10 min readLW link

Deep Learn­ing is cheap Solomonoff in­duc­tion?

Dec 7, 2024, 11:00 AM
45 points
1 comment17 min readLW link

In­ter­pret­ing Complexity

Maxwell AdamMar 14, 2025, 4:52 AM
53 points
8 comments26 min readLW link

The gen­er­al­iza­tion phase diagram

Dmitry VaintrobJan 26, 2025, 8:30 PM
26 points
2 comments16 min readLW link

Proof idea: SLT to AIT

Lucius BushnaqFeb 10, 2025, 11:14 PM
40 points
15 comments6 min readLW link

Epoch wise crit­i­cal pe­ri­ods, and sin­gu­lar learn­ing theory

Garrett BakerDec 14, 2023, 8:55 PM
16 points
1 comment5 min readLW link

Towards Devel­op­men­tal Interpretability

Jul 12, 2023, 7:33 PM
192 points
10 comments9 min readLW link1 review

Ap­ply for the 2023 Devel­op­men­tal In­ter­pretabil­ity Con­fer­ence!

Aug 25, 2023, 7:12 AM
33 points
0 comments2 min readLW link

You’re Mea­sur­ing Model Com­plex­ity Wrong

Oct 11, 2023, 11:46 AM
93 points
17 comments13 min readLW link

[Question] A few Align­ment ques­tions: util­ity op­ti­miz­ers, SLT, sharp left turn and identifiability

Igor TimofeevSep 26, 2023, 12:27 AM
6 points
1 comment2 min readLW link

My hopes for al­ign­ment: Sin­gu­lar learn­ing the­ory and whole brain emulation

Garrett BakerOct 25, 2023, 6:31 PM
61 points
5 comments12 min readLW link

Open Call for Re­search As­sis­tants in Devel­op­men­tal Interpretability

Aug 30, 2023, 9:02 AM
55 points
11 comments4 min readLW link

Sin­gu­lar learn­ing the­ory and bridg­ing from ML to brain emulations

Nov 1, 2023, 9:31 PM
26 points
16 comments29 min readLW link

Es­ti­mat­ing effec­tive di­men­sion­al­ity of MNIST models

Arjun PanicksseryNov 2, 2023, 2:13 PM
41 points
3 comments1 min readLW link

De­gen­era­cies are sticky for SGD

Jun 16, 2024, 9:19 PM
56 points
1 comment16 min readLW link

A short ‘deriva­tion’ of Watan­abe’s Free En­ergy Formula

Wuschel SchulzJan 29, 2024, 11:41 PM
13 points
6 comments7 min readLW link

Learn­ing co­effi­cient es­ti­ma­tion: the details

Zach FurmanNov 16, 2023, 3:19 AM
36 points
0 comments2 min readLW link
(colab.research.google.com)

My Crit­i­cism of Sin­gu­lar Learn­ing Theory

Joar SkalseNov 19, 2023, 3:19 PM
83 points
56 comments12 min readLW link

In­ter­view Daniel Mur­fet on Univer­sal Phenom­ena in Learn­ing Machines

Alexander Gietelink OldenzielFeb 6, 2023, 12:00 AM
50 points
1 comment16 min readLW link

Minor in­ter­pretabil­ity ex­plo­ra­tion #3: Ex­tend­ing su­per­po­si­tion to differ­ent ac­ti­va­tion func­tions (loss land­scape)

Rareș BaronMar 14, 2025, 3:45 PM
3 points
0 comments3 min readLW link

Sin­gu­lar Learn­ing The­ory for Dummies

Rahul ChandOct 15, 2024, 9:13 PM
1 point
0 comments8 min readLW link

rough draft on what hap­pens in the brain when you have an insight

EmrikMay 21, 2024, 6:02 PM
11 points
2 comments1 min readLW link

My im­pres­sion of sin­gu­lar learn­ing theory

Ege ErdilJun 18, 2023, 3:34 PM
47 points
30 comments2 min readLW link

Sin­gu­lar­i­ties against the Sin­gu­lar­ity: An­nounc­ing Work­shop on Sin­gu­lar Learn­ing The­ory and Alignment

Apr 1, 2023, 9:58 AM
87 points
0 comments1 min readLW link
(singularlearningtheory.com)

Es­ti­mat­ing the Prob­a­bil­ity of Sam­pling a Trained Neu­ral Net­work at Random

Mar 1, 2025, 2:11 AM
32 points
10 comments1 min readLW link
(arxiv.org)

Minor in­ter­pretabil­ity ex­plo­ra­tion #4: Lay­erNorm and the learn­ing coefficient

Rareș BaronMar 20, 2025, 4:18 PM
2 points
0 comments1 min readLW link

The Hes­sian rank bounds the learn­ing coefficient

Lucius BushnaqAug 8, 2024, 8:55 PM
68 points
10 comments4 min readLW link

The The­ory Be­hind Loss Curves

James CamachoMay 6, 2025, 10:22 PM
16 points
1 comment4 min readLW link
(github.com)

Fea­ture Tar­geted LLC Es­ti­ma­tion Dist­in­guishes SAE Fea­tures from Ran­dom Directions

Jul 19, 2024, 8:32 PM
59 points
6 comments16 min readLW link

Jesse Hoogland on Devel­op­men­tal In­ter­pretabil­ity and Sin­gu­lar Learn­ing Theory

Michaël TrazziJul 6, 2023, 3:46 PM
42 points
2 comments4 min readLW link
(theinsideview.ai)
No comments.