Hi, I am a Physicist, an Effective Altruist and AI Safety student/researcher.
Linda Linsefors
Karma: 2,465
The time and space bounded SI is then approximately optimal in the sense that its total error compared to this efficient predictor, as measured by KL-divergence from the predictions made by , will be bits summed across all data points.[7]
I think that classical Solomonoff induction gives zero posterior to any program with less than perfect prediction record? I can see why this works for Solomonoff with unbounded description length, this is solved by the DH(h) term you mention above.
But for bounded Solomonoff you have to allow some non-perfect programs to stay in the posterior, or you might be left with no candidates at all?
Is there an easy way to deal with this?
This is not a problem if the programs are giving probabilistic outputs, but that’s a different class of programs than used in classical Solomonoff induction.
Since Bayesian statistics is both fundamental and theoretically tractable
What do you mean by “tractable” here?
New related post:
Theory of Change for AI Safety Camp
Why AISC needs Remmelt
For ASIC to survive long term, it needs at leasat one person who is long term committed to running the program. Without such a person I estimate the half-life of of AISC to be ~1 year. I.e, there would be be ~50% chance of AISC dying out every year, simply because there isn’t an organiser team to run it.
Since the start, this person has been Remmelt. Because of Remmelt AISC has continued to exist. Other organiser have come and gone, but Remmelt has stayed and held things together. I don’t know if there is anyone to replace Remmelt in this role. Maybe Robert? But I think it’s too early to say. I’m definitely not available for this role, I’m too restless.
Hiring for long term commitment is very hard.
Why AISC would currently have had even less funding without Remmelt
For a while AISC was just me and Remmelt. During this time Remmelt took care of all the fundraising, and still mostly does, because Robert is still new, and I don’t do grant applications.
I had several bad experiences around grant applications in the past. The last one was in 2021, when me and JJ applied for money for AI Safety Support. The LTFF committee decided that they didn’t like me personally and agreed to fund JJ’s salary but not mine. This is a decision they were allowed to make, of course. But on my side, it was more than I could take emotionally, and it led me to quit EA and AI Safety entirely for a year, and I’m still not willing to do grant applications. Maybe someday, but not yet.
I’m very grateful to Remmelt for being willing to take on this responsibility, and for hiring me at a time when I was the one who was toxic to grant makers.
I have less triggers for crowdfunding and private donations, than for grant applications, but I still find it personally very stressfull. I’m not saying my trauma around this is anyone’s fault, or anyone else’s problem. But I do think it’s relevant context for understanding AISC funding situation. Organisations are made of people, and these people may have constraints that are invisible to you.
Some other things about Remmelt and his role in AISC
I know Remmelt get’s into argument on Twitter, but i’m not on Twitter, so I’m not paying attention to that. I know Remmelt as a friend and as a great co-organiser. Remmelt is one of the rare people I work really well with.
Within AISC, Remmelt is overseeing the Stop/Pause AI projects. For all the other projects, Remmelt is only involved in a logistical capacity.
For the current AISC there are
4 Pause/Stopp AI projects. Remmelt is in charge of accepting or rejecting projects in this chathegory, and also supporting them if they need any help.
1 project which Remmelt is running personally as the research lead. (Not counted as one of the 4 above)
26 other projects where Remmelt only have purely logistical involvement. E.g. Remmelt is in charge of stipends and compute reimbursement, but his personal opinions abut AI Safety is not a factor in this. I see most of the requests, and I’ve never seen Remmelt discriminate based on what he think of the project.
Each of us organisers (Remmelt, Robert, me) can unilaterally decide to accept any project we like, and once a project is accepted to AISC, we all support it in our roles as organisers. We have all agreed to this, because we all thinks that having this diversity is worth it, even if not all of us likes every single project the other ones accept.
I vouch for Robert as a good replacement for me.
Hopefully there is enough funding to onboard a third person for next camp. Running AISC at the current scale is a three person job. But I need to take a break from organising.
Is this because they think it would hurt their reputation, or because they think Remmelt would make the program a bad experience for them?
This comment has two disagree votes, which I interpret as other people seeing the flowchart. I see it too. If it still doesn’t work for you for some reason, you can also see it here: AISC ToC Graph—Google Drawings
Each organiser on the team are allowed to accept projects independently. So far Remmelt hasn’t accepted any projects that I would have rejected, so I’m not sure how his unorthodox views could have affected project quality.
Do you think people are avoiding AISC because of Remmelt? I’d be surprised if that was a significant effect.
After we accept projects, the project is pretty much in the hands of each research lead, with very lite involvement from the organisers.
I’d be interested to learn more about in what way you think or have heard that the program have gotten worse.
Not on sci-hub or Anna’s Archive, so I’m just going off the abstract and summary here; would love a PDF if anyone has one.
If you email the authors they will probably send you the full article.
It looks related, but these are not the plots I remember from the talk.
ϕt=Utϕ0U−1t.
I think you mean here, not
One of the talks at ILIAD had a set for PCA plots where the PC2 turned around at different points for different training setups. I think the turning point corresponded to when the model started to overfit. I don’t quite remember. But what ever the meaning of the turning point was, I think they also verified this with some other observation. Given that this was ILIAD the other observation was probably the LLC.
If you want to look it up I can try to find the talk among the recordings.
This looks like a typo?
Did you just mean “CS”?
In standard form, a natural latent is always approximately a deterministic function of . Specifically: .
What does the arrow mean in this expression?
I think the qualitive difference is not as large as you think it is. But I also don’t think this is very crux-y for anything, so I will not try to figure out how to translate my reasoning to words, sorry.
I guess the modern equivalent that’s relevant to AI alignment would be Singular Learning Theory which proposes a novel theory to explain how deep learning generalizes.
I think Singular Learning Theory was developed independently of deep learning, and is not specifically about deep learning. It’s about any learning system, under some assumptions, which are more general than the assumptions for normal Learning Theory. This is why you can use SLT but not normal Learning Theory when analysing NNs. NNs break the assumptions for normal Learning Theory but not for SLT.
Ok, in that case I want to give you this post as inspiration.
Changing the world through slack & hobbies — LessWrong
That’s still pretty good. Most reading lists are not updated at all after publication.
Is it an option to keep your current job but but spend your research hours on AI Safety instead of quarks? Is this something that would appealing to you + acceptable to your employer?
Given the current AI safety funding situation, I would strongly reccomend not giving up your current income.
I think that a lot of the pressure towards street light research comes from the funding situation. The grants are short and to stay in the game you need visible results quickly.
I think MATS could be good, if you can treat it as exploration, but not so good if you’re in a hurry to get a job or a grant directly afterwards. Since MATS is 3 months of full time, it might not fit into your schedule (without quitting your job). Maybe instead try SPAR. Or see here for more options.
Or you can skip the training program route, and just start reading on your own. There’s lots and lots of AI safety reding lists. I reccomend this one for you. @Lucius Bushnaq who created and maintains it, also did quark physics, before switching to AI Safety. But if you don’t like it, there are more options here under “Self studies”.
In general, the funding situation in AI safety is pretty shit right now, but other than that, there are so many resources to help people get started. It’s just a matter of choosing where to start.
According to my calculation, this embedding will result in too much compounding noise. I get the same noise results as you for one layer, but the noise grows too much from layer to layer.
However, Lucius suggested a different embedding, which seems to work.
We’ll have some publication on this eventually. If you want to see the details sooner you can message me.