Views my own, not my employers.
nc
Karma: 27
Currently I’m not convinced that the memetic analogy has done more to clarify than to occlude cultural evolution/opinion dynamics. That’s not to say that work in genetics is useless, but I think that the terminology has taken precedence above what the actual concepts mean, and I read a lot of conversations that feel like people just trading the information that they read The Selfish Gene 40 years ago.
There’s certainly scope for an applied “memetics” but it’s really crying out for a good predictive (even if simplistic) model.
I’ve noticed they perform much better on graduate-level ecology/evolution questions (in a qualitative sense—they provide answers that are more ‘full’ as well as technically accurate). I think translating that into a “usefulness” metric is always going to be difficult though.
I would have found it helpful in your report for there to be a ROSES-type diagram or other flowchart showing the steps in your paper collation. This would bring it closer in line with other scoping reviews and would have made it easier to understand your methodology.
Linguistic Drift, Neuralese, and Steganography
In this section you use these terms implying there’s a body of research underneath these terms. I’m very interested in understanding this behaviour but I wasn’t aware it was being measured. Is anyone currently working on models of linguistic drift/measuring it with manuscripts you could link?
My impression is that’s a little simplistic, but I also don’t have the best knowledge of the market outside WGS/WES and related tools. That particular market is a bloodbath. Maybe there’s better scope in proteomics/metabolomics/stuff I know nothing about.
My impression is that much of this style of innovation is happening inside research institutes and then diffusing outward. There are plenty of people doing “boring” infrastructure work at the Sanger Institute, EMBL-EBI, etc. And you all get it for free! I can however see that on-demand services for biotech are a little different.
This fail-state is particularly worrying to me, although it is not obvious whether there is enough time for such an effect to actually intervene on the future outcome.
Are you aware of anyone else working on the same topic?
I was reading the UK National Risk Register earlier today and thinking about this. Notable to me that the top-level disaster severity has a very low cap of ~thousands of casualties, or billions of economic loss. Although it does note in the register that AI is a chronic risk that is being managed under a new framework (that I can’t find precedent for).
I do think this comes back to the messages in On Green and also why the post went down like a cup of cold sick—rationality is about winning. Obviously nobody on LW wants to “win” in the sense you describe, but more winning over more harmony on the margin, I think.
The future will probably contain less of the way of life I value (or something entirely orthogonal), but then that’s the nature of things.
Your general argument rings true to my ears—except the part about AI safety. It is very hard to interact with AI safety without entering the x-risk sphere, as shown by this piece of research by the Cosmos Institute, where the x-risk sphere is almost 2/3rds total funding (I have some doubts about the accounting). Your argument about Mustafa Suleyman strikes me as a “just-so” story—I do wish it were replicable, but I would be surprised, particularly with AI safety’s sense of urgency.
I’m here because truly there is no better place, and I mean that in both a praiseworthy and an upsetting sense. If you think it’s misguided, we, on the same side, need to show the strength of our alternative, don’t we?
I think you’re getting at something fairly close to the Piranha theorem from a different (ecological?) angle.
Advice for journalists was a bit more polemic which I think naturally leads to more engagement. But I’d like to say that I strongly upvoted the mapping discussions post and played around with the site quite a bit when it was first posted—it’s really valuable to me.
Karma’s a bit of a blunt tool—yes I think it’s good to have posts with broad appeal but some posts are going to be comparatively more useful to a smaller group of people, and that’s OK too.
Your points are true and insightful, but you’ve written them in a way that won’t gain much cachet here.
I wrote a similar piece to the Cosmos Institute competition, which hopefully I can share here when that is finished, and maybe we can bounce the idea off each other?
I think this effect will be more wide-spread than targeting only already-vulnerable people, and it is particularly hard to measure because the causes will be decentralised and the effects will be diffuse. I predict it being a larger problem if, in the run-up between narrow AI and ASI, we have a longer period of necessary public discourse and decision-making. If the period is very short then it doesn’t matter. It may not affect many people given how much penetration AI chatbots have in the market before takeoff too.
This is not an obvious continuation of the prompt to me—maybe there are just a lot more examples of explicit refusal on the internet than there are in (e.g.) real life.
thinking maybe we owe something to our former selves, but future people probably won’t think this
This is a very strong assertion. Aren’t most people on this forum, when making present claims about what they would like to happen in the future, trying to form this contract? (This comes back to the value lock-in debate.)
Is there a reason to expect this kind of behaviour to appear from base models with no fine-tuning?
Unfortunately you did nerdsnipe me with the ‘biologists think’ statement so I am forced to keep replying!
It’s worth noting that the original derivations of natural selection do use absolute fitness—relative fitness is simply a reparameterization when you have constant N (source: any population genetics textbook). This was why I brought up density-dependent selection, as under that framework N (and s) is changing, and selection in those circumstances is more complicated.
In fact, even under typical models, relative fitness and absolute fitness show interesting relations. See this paper by Orr where alleles (which in this model only affect relative fitness by increasing absolute fitness) show diminishing returns on excess fecundity. The first paper I sent you also explicitly says [that absolute fitness is required under N-varying s or T-varying N] in the abstract.
I thought you were making a more subtle point about the additional demands of life history theory vs. the pure allele-eye view, which I agree is interesting. I hope I have convinced you that biologists are already doing fruitful work in this area. I don’t understand the mesa-optimization arguments well enough to tell whether such an analogy is useful (to people who work in AI), but I do think it is true in at least a trivial sense.
There’s a connection to the idea of irony poisoning here, and I do not think it is good for the person in question to pretend to hold extremist views. This is a parallel issue with the fact that it’s terrible optics and creates a difficult tension with this website’s newfound interest in doing communications/policy/outreach work.