Software engineer transitioned into AI safety, teaching and strategy. Particularly interested in psychology, game theory, system design, economics.
Jonathan Claybrough
Karma: 384
I don’t actualy think your post was hostile, but I think I get where deepthoughtlife is coming from. At the least, I can share about how I felt reading this post and point out to why, since you seem keen on avoiding the negative side. Btw I don’t think you avoid causing any frustration in readers, they are too diverse, so don’t worry too much about it either.
The title of the piece is strongly worded and there’s no epistimic status disclaimer to state this is exploratory, so I actually came in expecting much stronger arguments. Your post is good as an exposition of your thoughts and conversation started, but it’s not a good counter argument to NAH imo, so shouldn’t be worded as such. Like deepthoughtlife, I feel your post is confused re NAH, which is totally fine when stated as such, but a bit grating when I came in expecting more rigor or knowledge of NAH.
Here’s a reaction to the first part :
- in “Systems must have similar observational apparatus” you argue that different apparatus lead to different abstractions and claim a blind deaf person is such an example, yet in practice blind deaf people can manipulate all the abstractions others can (with perhaps a different inner representation), that’s what general intelligence is about. You can check out this wiki page and video for some of how it’s done https://en.wikipedia.org/wiki/Tadoma . The point is that all the abstractions can be understood and must be understood by a general intelligence trying to act effectively, and in practice Helen Keler could learn to speak by using other senses than hearing, in the same way we learn all of physics despite limited native instruments.
I think I had similar reactions to other parts, feeling they were missing the point about NAH and some background assumptions.
Thanks for posting!
Putting this short rant here for no particularly good reason but I dislike that people claim constraints here or there in a way where I guess their intended meaning is only that “the derivative with respect to that input is higher than for the other inputs”.
On factory floors there exist hard constraints, the throughput is limited by the slowest machine (when everything has to go through this). The AI Safety world is obviously not like that. Increase funding and more work gets done, increase talent and more work gets done. None are hard constraints.
If I’m right that people are really only claiming the weak version, then I’d like to see somewhat more backing to their claims, especially if you say “definitely”. Since none are constraints, the derivatives could plausibly be really close to one another. In fact, they kind of have to be, because there are smart optimizers who are deciding where to spend their funding and trying to actively manage the proportion of money sent to field building (getting more talent) vs direct work.
Interesting thoughts, ty.
A difficulty to common understanding I see here is that you’re talking of “good” or “bad” paragraphs in the absolute, but didn’t particularly define “good” or “bad” paragraph by some objective standard, so you’re relying on your own understanding of what’s good or bad. If you were defining good or bad relatively, you’d look for a 100 paragraphs, and post the worse 10 as bad. I’d be interested in seeing what were the worse paragraphs you found, some 50 percentile ones, and what were the best, then I’d tell you if I have the same absolute standards as you have.
Enjoyed this post.
Fyi, from the front page I just hovered this post “The shallow bench” and was immediately spoiled on Project Hail Mary (which I had started listening to, but didn’t get far into). Maybe add some spoiler tag or warning directly after the title?
Without removing from the importance of getting the default right, and with some deliberate daring to feature creep, I think adding a customization feature (select colour) in personal profiles is relatively low effort and maintenance, so would solve the accessibility problem.
There’s tacit knowledge in bay rationalist conversation norms that I’m discovering and thinking about, here’s an observation and related thought. (I put the example later after the generalisation because that’s my preferred style, feel free to read the other way).
Willingness to argue righteously and hash out things to the end, repeated over many conversations, makes it more salient when you’re going for a dead end argument. This salience can inspire you to do argue more concisely and to the point over time.
Going to the end of things generates ground data on which to update your models of arguing and conversation paths, instead of leaving things unanswered.
So, though it’s skilful to know when not to “waste” time on details and unimportant disagreements, the norm of “frequently enough going through til everyone agrees on things” seems profoundly virtuous.
Short example from today, I say “good morning”. They point out it’s not morning (it’s 12:02). I comment about how 2 minutes is not that much. They argue that 2 minutes is definitely more than zero and that’s the important cut-off.
I realize that “2 minutes is not that much” was not my true rebuttal, that this next token my brain generated was mostly defensive reasoning rather than curious exploration of why they disagreed with my statement. Next time I could instead note they’re using “morning” to have a different definition/central cluster than I, appreciate that they pointed this out, and decide if I want to explore this discrepancy or not.
Many things don’t make sense if you’re just doing them for local effect, but do when you consider long term gains. (something something naive consequentialism vs virtue ethics flavored stuff)
I don’t strongly disagree but do weakly disagree on some points so I guess I’ll answer
Re first- if you buy into automated alignment work by human level AGI, then trying to align ASI now seems less worth it. The strongest counterargument to this I see is that “human level AGI” is impossible to get with our current understanding, as it will be superhuman in some things and weirdly bad at others.
Re second- disagreements might be nitpicking on “few other approaches” vs “few currently pursued approaches”. There are probably a bunch of things that would allow fundamental understanding if they panned out (various agent foundations agendas, probably safe ai agendas like davidad’s), though one can argue they won’t apply to deep learning or are less promising to explore than SLT
I don’t think your second footnote sufficiently addresses the large variance in 3D visualization abilities (note that I do say visualization, which includes seeing 2D video in your mind of a 3D object and manipulating that smoothly), and overall I’m not sure where you’re getting at if you don’t ground your post in specific predictions about what you expect people can and cannot do thanks to their ability to visualize 3D.
You might be ~conceptually right that our eyes see “2D” and add depth, but *um ackshually*, two eyes each receiving 2D data means you’ve received 4D input (using ML standards, you’ve got 4 input dimensions per time unit, 5 overall in your tensor). It’s very redundant, and that redundancy mostly allows you to extract depth using a local algo, which allows you to create a 3D map in your mental representation. I don’t get why you claim we don’t have a 3D map at the end.
Back to concrete predictions, are there things you expect a strong human visualizer couldn’t do? To give intuition I’d say a strong visualizer has at least the equivalent visualizing, modifying and measuring capabilities of solidworks/blender in their mind. You tell one to visualize a 3D object they know, and they can tell you anything about it.
It seems to me the most important thing you noticed is that in real life we don’t that often see past the surface of things (because the spectrum of light we see doesn’t penetrate most material) and thus most people don’t know the inside of 3D things very well, but that can be explained by lack of exposure rather than inability to understand 3D.
Fwiw looking at the spheres I guessed an approx 2.5 volume ratio. I’m curious, if you visualized yourself picking up these two spheres, imagining them made of a dense metal, one after the other, could you feel one is 2.3 times heavier than the previous?
I’ll give fake internet points to whoever actually follows the instructions and posts photographic proof.
The naming might be confusing because pivotal act sounds like a one time action, but in most cases getting to a stable world without any threat from AI requires constant pivotal processes. This makes almost all the destructive approaches moot (and they’re probably already bad for ethical concerns and many others already discussed) because you’ll make yourself a pariah.
The most promising venue for a pivotal act/pivotal process that I know of is doing good research so that ASI risks are known and proven, doing good outreach and education so most world leaders and decision makers are well aware of this, and helping setup good governance worldwide to monitor and limit the development of AGI and ASI until we can control it.
I recently played Outer Wilds and Subnautica, and the exercise I recommend for both of these games is : Get to the end of the game without ever failing.
In subnautica that’s dying once, in Outer Wilds it’s a spoiler to describe what failing is (successfully getting to the end could certainly be argued to be a fail).
I failed in both of these. I played Outer Wilds first and was surprised at my fail, which inspired me to play Subnautica without dying. I got pretty far but also died from a mix of 1 unexpected game mechanic, uncareful measure of another mechanic, lack of redundancy in my contingency plans.
Oh wow, makes sense. It felt weird that you’d spend so much time on posts, yet if you didn’t spend much time it would mean you write at least as fast as Scott Alexander. Well, thanks for putting in the work. I probably don’t publish much because I want it to not be much work to do good posts but you’re reassuring it’s normal it does.
(aside : I generally like your posts’ scope and clarity, mind saying how long it takes you to write something of this length?)
Self modeling is a really important skill, and you can measure how good you are at it by writing predictions about yourself. (Modelling A notably important one for people who have difficulty with motivation is predicting your own motivation—will you be motivated to do X in situation Y?
If you can answer that one generally, you can plan to actually anything you could theoretically do, using the following algorithm : from current situation A, to achieve wanted outcome Z, find a predecessor situation Y from which you’ll be motivated to get to Z (eg. have written 3 paragraphs of 4 of an essay), and a predecessor situational X from which you’ll get to Y, iterate til you get to A (or forward chain, from A to Z). Check that indeed you’ll be motivated each step of the way.
How can the above plan fail? Either you were mistaken about yourself, or about the world. Figure out which and iterate.
Appreciate the highlight of identity as this import/crucial self fulfilling prophecy, I use that frame a lot.
What does the title mean? Since they all disagree I don’t see one as being more of a minority than the other.
Nice talk!
When you talk about the most important interventions for the three scenarios, I wanna highlight that in the case of nationalization, you can also, if you’re a citizen of one of these countries nationalizing AI, work for the government and be on those teams working and advocating for safe AI.
In my case I should have measurable results like higher salary, higher life satisfaction, more activity, more productivity as measured by myself and friends/flatmates. I was very low so it’ll be easy to see progress. The difficulty was finding something that’d work, but it won’t be measuring if it does.
Some people have short ai timelines based inner models that don’t communicate well. They might say “I think if company X trains according to new technique Y it should scale well and lead to AGI, and I expect them to use technique Y in the next few years”, and the reasons for why they think technique Y should work are some kind of deep understanding built from years of reading ml papers, that’s not particularly easy to transmit or debate.
In those cases, I want to avoid going into details and arguing directly, but would suggest that they use their deep knowledge of ML to predict existing recent results before looking at them. This would be easy to cheat, so I mostly suggest this for people to check themselves, or check people you trust to be honorable. Concretely, it’d be nice if when some new ml paper with a new technique comes out, someone compilés a list of questions answered by that paper (eg is technique A better than technique B for a particular result) and posts it to LW so people can track how well they understand ML, and thus (to some extent) short timelines.
For example a recent paper examinés how data affects performance on a bunch of benchmarks, and notably tested training either on an duplicated dataset (a bunch of common crawls), or deduplixated (the same except remove same documents that were shared between crawls). Do you expect deduplication in this case raises or lowers performance on benchmarks? If we could have similar questions when new results come out it’s be nice.
Thank you for sharing, it really helps to pile on these stories (and nice to have some trust they’re real, more difficult to get from reddit—on which note are there non doxing receipts you can show for this story being true? I have no reason to doubt you in particular but I guess it’s good hygiene when on the internet to ask for evidence)
It also makes me wanna share a bit of my story. I read The Mind Illuminated, I did only small amounts of meditation, yet the framing the book offers has been changing my thinking and motivational systems. There aren’t many things I’d call info hazards, but in my experience even just reading the book seems to be enough to contribute to profound changes, that would not be obviously be considered positive by the previous me. (They’re not obviously negative either, I happen to be hopeful, but I’m waiting on results another year later to say)
Congratz on your successes and thank you for publishing this impact report.
It leaves me unsatiated related to cost effectiveness though. With no idea of how much money was invested in this project to get this outcome, I don’t know if Arena is cost effective compared to other training programs and counterfactual opportunities. Would you mind sharing at least something about the amount of funding this got?
Re
it doesn’t strike me that a 5 week all expenses paid program is a particularly low cost way to find out AI Safety isn’t for you (as compared to for example participating in an Apart Hackathon)