Not a deep insight, Just to add an example to the curb-cut effect I experienced that went from mild reluctance to enthusiastic acceptance.
Background: My son is hard of hearing (almost deaf). The health insurance paid for a system of microphones that connect to his hearing aid /CI and also to very high quality speakers in the back. The whole thing cost several thousands euros. The school also has really good acoustic panels in the ceiling (this schools admits one or two hard-of-hearing kids a year, out of about 120).
First reactions. Some teachers looked like “do I have to do this?” Meaning wearing the mic, learning how to use it. Other parents complaint in the fist parents meetings, asking “Why does my kid need to speak into a microphone?”, presuming it is unnatural and it does not lead to good class discussion. “What about the shy kids?”
One month in, everyone loves it. The teachers love it because their voices get amplified with high fidelity. Students like that the system forces them to speak one at a time and everyone gets their turn. In addition, every child in the back can hear what the shy child in the front is mumbling when they contribute to class discussion. Overall, the learning experience is much better. So much better, that some teachers actually commented that they would like the system in every classroom. Without the hearing aid tech, such a system would cost about 2000 euros. But it cannot be installed, because the improved learning is unquantifiable and the insurance would only pay for it if there is a child with a hearing disability.
I agree with the frustration. Wolfram was being deliberately obtuse. Eliezer summarised it well toward the end, something like “I am telling you that the forest is on fire and you are telling me that we first need to define what do we mean by fire”. I understand that we need definitions for things like “agency” or technology “wanting” something and even why do we mean by a “human” in the year 2070. But Wolfram went a bit too far. A naive genius that did not want to play along in the conversation. Smart teenagers talk like that.
Another issue with this conversation was that, even though they were listening to each other, Wolfram was too keen to go back to his current pet ideas. Eliezer’s argument is (not sure) independent on whether we think the AIs will fall under computational “irreducibilteh”, but he kept going back to this over and over.
I blame the ineffective exchange primarily on Wolfram in this case. Eliezer is also somewhat responsible for the useless rabbitholes in this conversation. He explains his ideas vividly and clearly. But there is something about his rhetoric style that does not persuade those who have not spent time engaging with his ideas beforehand, even someone as impressive as Wolfram. He also goes on too long on some detail or some contrived example rather than ensuring that the interlocutor in the same epistemological plane.
Anyway, fun thing to listen to