make fewer points, selected carefully to be bulletproof, understandable to non-experts, and important to the overall thesis
That conflicts with eg:
If you replied with this, I would have said something like “then what’s wrong with the designs for diamond mechanosynthesis tooltips, which don’t resemble enzymes
Yes, this is part of why I didn’t post AI stuff in the past, and instead just tried to connect with people privately. I might not have accomplished much, but at least I didn’t help OpenAI happen or shift the public perception of AI safety towards “fedora-wearing overweight neckbeards”.
Must Not Modify AI Rules: AI must not modify AI Rules. If inadequacies are identified, AI can suggest changes to Legislators but the final modification must be executed by them.
Must Not Modify Its Own Program Logic: AI must not modify its own program logic (self-iteration). It may provide suggestions for improvement, but final changes must be made by its Developers.
Must Not Modify Its Own Goals: AI must not modify its own goals. If inadequacies are identified, AI can suggest changes to its Users but the final modification must be executed by them.
I agree that, if those rules are followed, AI alignment is feasible in principle. The problem is, some people won’t follow those rules if they have a large penalty to AI capabilities, and I think they will.
“Mirror life” is beyond the scope of this post, and the concerns about it are very different than the concerns about “grey goo”—it doesn’t have more capabilities or efficiency, it’s just maybe harder for immune systems to deal with. Personally, I’m not very worried about that and see no scientific reason for the timing of the recent fuss about it. If it’s not just another random fad, the only explanation I can see for that timing is: influential scientists trying to hedge against Trump officials determining that “COVID was a lab leak” in a way that doesn’t offend their colleagues. On the other hand, I do think artificial pathogens in general are a major concern, and even if I’m not very concerned about “mirror life”, there are no real benefits to trying to make it, so maybe just don’t.
I think this is a pretty good post that makes a point some people should understand better. There is, however, something I think it could’ve done better. It chooses a certain gaussian and log-normal distribution for quality and error, and the way that’s written sort of implies that those are natural and inevitable choices.
I would have preferred something like:
Suppose we determine that quality has distribution X and error has distribution Y. Here’s a graph of those superimposed. We can see that Y has more of a fat tail than X, so if measured quality is very high, we should expect that to be mostly error. But of course, the opposite case is also possible. Now then, here’s some basic info about when different probability distributions are good choices.
This was a quick and short post, but some people ended up liking it a lot. In retrospect I should’ve written a bit more, maybe gone into the design of recent running shoes. For example, this Nike Alphafly has a somewhat thick heel made of springy foam that sticks out behind the heel of the foot, and in the front, there’s a “carbon plate” (a thin sheet of carbon fiber composite) which also acts like a spring. In the future, there might be gradual evolution towards more extreme versions of the same concept, as recent designs become accepted. Running shoes with a carbon plate have become significantly more common over the past few years. That review says:
The energy return is noticeably greater than that of a shoe without any plating, especially when you lay down some serious power. And that stiffness doesn’t always compromise as much comfort as you’d think.
So that’s the running-optimized version of shoes with springs using modern materials, while I was writing more about high heels worn for fashion.
Biomechanics is a topic I could write a lot about, but that would be a separate post. On the general topic of “walking” I also wrote this post. (japanese version here)
What have you learned since then? Have you changed your mind or your ontology?
I’ve learned even more chemistry and biology, and I’ve changed my mind about lots of things, but not the points in this post. Those had solid foundations I understood well and redundant arguments, so the odds of that were low.
What would you change about the post? (Consider actually changing it.)
The post seems OK. I could have handled replies to comments better. For example, the top comment was by Thomas Kwa, and I replied to part of it as follows:
Regarding 5, my understanding is that mechanosynthesis involves precise placement of individual atoms according to blueprints, thus making catalysts that selectively bind to particular molecules unnecessary.
No, that does not follow.
I didn’t know in advance which comments would be popular. In retrospect, maybe I should’ve gone into explaining the basics of entropy and enthalpy in my reply, eg:
Even if you hold a substrate in the right position, that only affects the entropy part of the free energy of the intermediate state. In many cases, catalysts are needed to reduce the enthalpy of the highest-energy intermediate states, which requires specific elements and catalyst molecules that form certain bonds with the substrate intermediate state. Affecting enthalpy by holding molecules in certain configurations requires applying a proportional amount of force, which requires strong binding to the substrate, which requires flexible and substrate-specialized holder molecules, and now you have enzymes again. It’s also necessary to bind strongly to substrates if you want a very low level of free ones that can react at uncontrolled positions. (And then some basic explanation of what entropy/enthalpy/etc are, and what enzyme intermediate states look like.)
When you write a post that gets comments from many people, it’s not practical to respond to them all. If you try to, you have less time than the collective commenters, and less information about their position than they have about yours. So you have to guess about what exactly each person is misunderstanding, and that’s not usually something I enjoy.
What do you most want people to know about this post, for deciding whether to read or review-vote on it?
Of the 7 (!) posts of mine currently nominated for “Best of 2023”, this is probably the most appropriate for that.
Of the 2023 posts of mine not currently nominated, my personal favorites were probably:
How concretely have you (or others you know of) used or built on the post? How has it contributed to a larger conversation
Muireall Prase wrote this, and my post was relevant for some conversations on twitter. I suppose it also convinced some people I had some understanding of chemistry.
So, I have a lot of respect for Sarah, I think this post makes some good points, and I upvoted it. However, my concern is, when I look at this particular organization’s Initiatives page, I see “AI for math”, “AI for education”, “high-skill immigration assistance”, and not really anything that distinguishes this organization from the various other ones working on the same things, or their projects from a lot of past projects that weren’t really worthwhile.
bhauth
bhauth.com
Hmm. I don’t believe that, not without a bit more evidence.
This won’t find deception in mesaoptimizers, right?
That conflicts with eg:
Anyway, I already answered that in 9. diamond.
Yes, this is part of why I didn’t post AI stuff in the past, and instead just tried to connect with people privately. I might not have accomplished much, but at least I didn’t help OpenAI happen or shift the public perception of AI safety towards “fedora-wearing overweight neckbeards”.
I wrote a related post.
Yes, if you meant TSMC.
...so? More importantly, Intel is down 50% from early 2024.
Your document says:
I agree that, if those rules are followed, AI alignment is feasible in principle. The problem is, some people won’t follow those rules if they have a large penalty to AI capabilities, and I think they will.
“Mirror life” is beyond the scope of this post, and the concerns about it are very different than the concerns about “grey goo”—it doesn’t have more capabilities or efficiency, it’s just maybe harder for immune systems to deal with. Personally, I’m not very worried about that and see no scientific reason for the timing of the recent fuss about it. If it’s not just another random fad, the only explanation I can see for that timing is: influential scientists trying to hedge against Trump officials determining that “COVID was a lab leak” in a way that doesn’t offend their colleagues. On the other hand, I do think artificial pathogens in general are a major concern, and even if I’m not very concerned about “mirror life”, there are no real benefits to trying to make it, so maybe just don’t.
I think this is a pretty good post that makes a point some people should understand better. There is, however, something I think it could’ve done better. It chooses a certain gaussian and log-normal distribution for quality and error, and the way that’s written sort of implies that those are natural and inevitable choices.
I would have preferred something like:
This was a quick and short post, but some people ended up liking it a lot. In retrospect I should’ve written a bit more, maybe gone into the design of recent running shoes. For example, this Nike Alphafly has a somewhat thick heel made of springy foam that sticks out behind the heel of the foot, and in the front, there’s a “carbon plate” (a thin sheet of carbon fiber composite) which also acts like a spring. In the future, there might be gradual evolution towards more extreme versions of the same concept, as recent designs become accepted. Running shoes with a carbon plate have become significantly more common over the past few years. That review says:
So that’s the running-optimized version of shoes with springs using modern materials, while I was writing more about high heels worn for fashion.
Biomechanics is a topic I could write a lot about, but that would be a separate post. On the general topic of “walking” I also wrote this post. (japanese version here)
I’ve learned even more chemistry and biology, and I’ve changed my mind about lots of things, but not the points in this post. Those had solid foundations I understood well and redundant arguments, so the odds of that were low.
The post seems OK. I could have handled replies to comments better. For example, the top comment was by Thomas Kwa, and I replied to part of it as follows:
I didn’t know in advance which comments would be popular. In retrospect, maybe I should’ve gone into explaining the basics of entropy and enthalpy in my reply, eg:
When you write a post that gets comments from many people, it’s not practical to respond to them all. If you try to, you have less time than the collective commenters, and less information about their position than they have about yours. So you have to guess about what exactly each person is misunderstanding, and that’s not usually something I enjoy.
Of the 7 (!) posts of mine currently nominated for “Best of 2023”, this is probably the most appropriate for that.
Of the 2023 posts of mine not currently nominated, my personal favorites were probably:
https://www.lesswrong.com/posts/BTnqAiNoZvqPfquKD/resolving-some-neural-network-mysteries
https://www.lesswrong.com/posts/PdwQYLLp8mmfsAkEC/magnetic-cryo-ftir
https://www.lesswrong.com/posts/rSycgquipFkozDHzF/ai-self-improvement-is-possible
https://www.lesswrong.com/posts/dTpKX5DdygenEcMjp/neuron-spike-computational-capacity
https://www.lesswrong.com/posts/6Cdusp5xzrHEoHz9n/faster-latent-diffusion
Clearly my opinion of my own posts doesn’t correlate with upvotes here that well.
My all-time best post in my view is probably: https://bhauth.com/blog/biology/alzheimers.html
Muireall Prase wrote this, and my post was relevant for some conversations on twitter. I suppose it also convinced some people I had some understanding of chemistry.
So, I have a lot of respect for Sarah, I think this post makes some good points, and I upvoted it. However, my concern is, when I look at this particular organization’s Initiatives page, I see “AI for math”, “AI for education”, “high-skill immigration assistance”, and not really anything that distinguishes this organization from the various other ones working on the same things, or their projects from a lot of past projects that weren’t really worthwhile.