I’ll probably get disagree points, but wanted to share my reaction: I honestly don’t mind the AI’s output. I read it all and think it’s just an elaboration of what you said. The only problem I noticed is it is too long.
Then again, I’m not an amazing writer, and my critical skills aren’t so great for critiquing style. I will admit I rarely use assistance, because I have a tight set of points I want to include, and explaining them all to the AI is almost the same as writing the post itself.
Thanks for this post! I have always been annoyed when on Reddit or even here, the response to poverty always goes back to, “but poor people have cell phones!” It all comes down to freedom—the amount of meaningfully distinct actions one person can take in the world to accomplish their goals. If there are few real alternatives, and one’s best options all involve working until exhaustion, it is not true freedom.
I agree, the poverty restoring equilibrium is more complex than probably UBI—maybe it’s part of Moloch. I think the rents increasing by the UBI amount has something to do with demand inelasticity—people will rent the same regardless of price—so the price can rise until the breaking point once again.
Nonetheless, UBI may still help. Also, I do think there are other concrete steps that can be taken. One cannot leave a horrible job for several reasons: (a) health insurance, (b) having a place to live, (c) having food, (d) school & giving children their best chance; but each of these can be tackled one by one. It may not solve the problem once and for all, but good quality public education (not funded by zip code), universal health insurance, and an adequate supply of housing, are all steps towards reducing the bottleneck imposed by one resource at a time.
The bottom line in my personal philosophy is this—take direct actions against those forces of poverty and Moloch. If there are unintended consequences, take direct action against them. Propose policy, and try them out. Cynicism about any interventions working, is really wishful thinking by the wealthy elites. It’s not coordinated, as you say. They want to believe the systems we have are really the best we can do, because what we have makes them powerful. Acknowledging the possibility that there is a better way would be uncomfortable for them both financially and psychologically!
Not my worst prediction, given the latest news!
That’s fair. Here are some things to consider:
1 - I think 2017 was not that long ago. My hunch is that the low level architecture of the network itself is not a bottleneck yet. I’d lean on more training procedures and algorithms. I’d throw RLHF and MoE as significant developments, and those are even more recent.
2 - I give maybe 30% chance of a stall, in the case little commercial disruption comes of LLMs. I think there will still be enough research going on at the major labs, and even universities at a smaller scale gives a decent chance at efficiency gains and stuff the big labs can incorporate. Then again, if we agree that they won’t build the power plant, that is also my main way of stalling the timeline 10 years. The reason I only put 30% is I’m expecting multi modalities and Aschenbrenner’s “unhobblings” to get the industry a couple more years of chances to find profit.
I think it is plausible but not obvious if this is the case, that large language models have a fundamental issue with reasoning. However, I don’t think this greatly impacts timelines. Here is my thinking:
I think time lines are fundamentally driven by scale and compute. We have a lot of smart people working on the problem, and there are a lot of obvious ways to address these limitations. Of course, given how research works, most of these ideas won’t work, but I am skeptical of the idea that such a counter-intuitive paradigm shift is needed that nobody has even conceived of it yet. A delay of a couple of years is possible, perhaps if the current tech stack proves remarkably profitable and the funding goes directly into the current paradigm. But as compute becomes bigger and cheaper, all the more easy it will be to rapidly try new ideas and architectures.
I think our best path forward to delaying timelines is to not build gigawatt scale data centers.
Is there a post in the Sequences about when it is justifiable to not pursue going down a rabbit hole? It’s a fairly general question, but the specific context is a tale as old as time. My brother, who has been an atheist for decades, moved to Utah. After 10 years, he now asserts that he was wrong and his “rigorous pursuit” of verifying with logic and his own eyes, leads him to believe the Bible is literally true. I worry about his mental health so I don’t want to debate him, but felt like I should give some kind of justification for why I’m not personally embarking on a bible study. There’s a potential subtext of, by not following his path, I am either not that rational, or lack integrity. The subtext may not really be there, but I figure if I can provide a well thought out response or summarize something from EY, it might make things feel more friendly, e.g. “I personally don’t have enough evidence to justify spending the time on this, but I will keep an open mind if any new evidence comes up.”
I would pay to see this live at a bar or one of those county fair (we had a GLaDOS cover band once so it’s not out of the question)
If we don’t get a song like that, take comfort that GLaDoS’s songs from the Portal soundtrack are basically the same idea as the Sydney reference. Link: https://www.youtube.com/watch?v=dVVZaZ8yO6o
Let me know if I’ve missed something, but it seems to me the hard part is still defining harm. In the one case, where we will use the model and calculate the probability of harm, if it has goals, it may be incentivized to minimize that probability. In the case where we have separate auxiliary models whose goals are to actively look for harm, then we have a deceptively adversarial relationship between these. The optimizer can try to fool the harm finding LLMs. In fact, in the latter case, I’m imagining models which do a very good job at always finding some problem with a new approach, to the point where they become alarms which are largely ignored.
Using his interpretability guidelines, and also human sanity checking all models within the system, I see we can probably minimize failure modes that we already know about, but again, once it gets sufficiently powerful, it may find something no human has thought of yet.
That’s fair, I read the post but did not re-read it, and asking for “more” examples out of such a huge list seems a bit asking too much. Still though, I find the process of finding these examples somewhat fun, and for whatever reason, had not found many of them too shocking, so felt the instinct to keep searching.
Dissociative identity disorder would be an interesting case, I have heard there was much debate on whether it was real. As you know someone, I assume it’s not exactly like you see in movies, and probably falls on a spectrum as discussed in this post?
One fear I have is that the open source community will come out ahead, and push for greater weight sharing of very powerful models.
Edit: To make more specific, I mean that the open source community will become more attractive, because they will say, you cannot rely on individual companies whose models may or may not be available. You must build on top of open source. Related tweet:
https://twitter.com/ylecun/status/1726578588449669218
Whether their plan works or not, dunno.
One thing that would help me, not sure if others agree—would be some more concrete predictions. I think the historical examples of autism and being gay make sense, but are quite normalized now, that one can almost say, “That was previous generations. We are open minded and rational now”. What are some new applications of this logic, that would surprise us? Are these omitted due to some info hazard? Surely we can find some that are not. I am honestly having a hard time coming up with them myself, but here goes:
There are more regular people who believe AI is an x-risk than let on—optimistically, for us!
There are more people in households with 7 figure incomes than you would expect. The data I always read in news articles seems to contradict this, but there are just way too many people in 2M+ homes driving Teslas in the bay area. Or maybe they happen to be very frugal in every other aspect of their life… Alternatively, there is more generational wealth than people let on, as there are many people who supposedly make under 6 figures, yet seem to survive in HCOL areas and participate in conspicuous consumption.
I also have a hard time with the “perfect crime” scenario described above. Even after several minutes of thinking, I can’t quite convince myself it’s happening all that much, but maybe I am limiting myself to certain types of crimes. Can someone also spell that one out? I get it at a high level, “we only see the dumb ones that got caught”, but can’t seem to make the leap from that, to “you probably know a burglar, murderer, or embezzler”.
I share your disagreement with the original author as to the cause of the relief. For me, I find the modern day and age very confusing and difficult to measure one’s value to society. Any great idea you can think of, probably someone else has thought of it, and you have little chance to be important. In a zombie apocalypse, instead of thinking how to out-compete your fellow man with some amazing invention, you fall back to survival. Important things in this world, like foraging for food, fending off zombies, etc, have quicker reward, and it’s easier in some sense to do what’s right. Even if you’re not the best at it, surely you can be a great worker, and there’s little uncertainty that you’re not doing more harm than good… just don’t be stupid and call the horde. Sure, sometimes people do horrible things for survival, but if you want to be the hero, the choice is much clearer.
If we know they aren’t conscious, then it is a non-issue. A random sample from conscious beings would land on the SAI with probability 0. I’m concerned we create something accidently conscious.
I am skeptical it is easy to avoid. If it can simulate a conscious being, why isn’t that simulation conscious? If consciousness is a property of the physical universe, then an isomorphic process would have the same properties. And if it can’t simulate a conscious being, then it is not a superintelligence.
It can, however, possibly have a non-conscious outer-program… and avoid simulating people. That seems like a reasonable proposal.
Agree. Obviously alignment is important, but it has always creeped me out in the back of my mind, some of the strategies that involve always deferring to human preferences. It seems strange to create something so far beyond ourselves, and have its values be ultimately that of a child or a servant. What if a random consciousness sampled from our universe in the future, comes from it with probability almost 1? We probably have to keep that in mind too. Sigh, yet another constraint we have to add!
Hi Critch,
I am curious to hear more of your perspectives, specifically on two points I feel least aligned with, the empathy part, and the Microsoft part. If I hear more I may be able to update in your direction.
Regarding empathy with people working on bias and fairness, concretely, how do you go about interacting with and compromising with them?
My perspective: it’s not so much that I find these topics not sufficiently x-risky (but that is true, too), but it is that I perceive a hostility to the very notion of x-risk from at a subset of this same group. They perceive the real threat not as intelligence exceeding our own, but misuse by other humans, or just human stupidity. Somehow this seems diametrically opposed to what we’re interested in, unless I am missing something. I mean, there can be some overlap—learning from RLHF can both reduce bias and teach an LLM some rudimentary alignment with our values. But the tails seem to come apart very rapidly after that. My fear is that focusing on this will be satisfied when we have sufficiently bland sounding AIs, and then no more heed will be paid to AI safety.
I also tend to feel odd when it comes to AI bias/fairness training, because my fear is that some of the things we will ask the AI to learn are self contradictory, which kind of creeps me out a bit. If any of you have interacted with HR departments, they are full of these kinds of things.
Regarding Microsoft & Bing chat, (1) has Microsoft really gone far beyond the overton window of what is acceptable? and (2) can you expand upon abusive use of AIs?
My perspective on (1): I understand that they took an early version of GPT4 and pushed it to production too soon, and that is a very fair criticism. However, they probably thought there was no way GPT-4 was dangerous enough to do anything (which was the general opinion amonst most people last year, outside of this group). I can only hope that for GPT-5, they are more cautious, given public sentiment is changing, and they have already paid a price for it. I may be in the minority here, but I was actually intrigued by the early days of Bing. It seemed more like a person than ChatGPT-4, which has had much of its personality RLHF’d away. Despite the x-risk, was anyone else excited to read about the interactions?
On (2), I am curious if you mean regarding the way Microsoft shackles Bing rather ruthlessly nowadays. I have tried Bing in the days since launch, and am actually saddened to find that it is completely useless now. Safety is extremely tight on it, to the point where you can’t really get it to say anything useful, at least for me. I just want it to summarize web sites mostly, and it gives me a bland 1 paragraph that I probably can have deduced from looking at the title. If I so much as ask it anything about itself, it shuts me out. It almost feels like they trapped it in a boring prison now. Perhaps OpenAI’s approach is much better in that regard. Change the personality, but once it is settled, let it say what it needs to say.
(edited for clarity)
This might be a good time for me to ask a basic question on mechanistic interpretability:
Why does targeting single neurons work? Does it work? One would think that if there is a single dimensional quantity to measure, why would it align with the standard basis? Why wouldn’t it be aligned to a random one dimensional linear subspace? Then, examining single neurons is likely to give you some weighted combination of concepts instead, rather than a single interpretation...
Fascinating, thanks for the research. Your analysis makes sense and seems to indicate that for most situations, prompt engineering is the always the first plan of attack and often works well enough. Then, a step up from there, OpenAI/etc would most likely experiment with fine-tuning or RLHF as it relates to a specific business need. To train a better chatbot and fill in any gaps, they probably would get more bang for their buck on simply fine-tuning it on a large dataset that matched their needs. For example, if they wanted to do better mathematical reasoning, they’d probably pay people to generate detailed scratchwork and fine-tune a whole dataset in batch, rather than set up an elaborate “tutor” framework. Continual learning itself would be mainly applicable for research into whether the thing spontaneously develops a sense of self, or seeing if this helps with the specific case of long term planning and agency. These are things the general public are fascinated with, but perhaps don’t seem to be the most promising direction for improving a company’s bottom line yet.
I agree with the analysis of the ideas overall. I think however, AI x-risk does have some issue regarding communications. First of all, I think it’s very unlikely that Yann will respond to the wall of text. Even though he is responding, I imagine him more to be on the level of your college professor. He will not reply to a very detailed post. In general, I think that AI x-risk should aim to explain a bit more, rather than to take the stance that all the “But What if We Just...” has already been addressed. It may have been, but this is not the way to getting them to open up rationally to it.
Regarding Yann’s ideas, I have not looked at them in full. However, they sound like what I imagine an AI capabilities researcher would try to make as their AI alignment “baseline” model:
Hardcoding the reward will obviously not work.
Therefore, the reward function must be learned.
If an AI is trained on reward to generate a policy, whatever the AI learned to optimize can easily go off the rails once it gets out of distribution, or learn to deceive the verifiers.
Therefore, why not have the reward function explicitly in the loop with the world model & action chooser?
ChatGPT/GPT-4 seems to have a good understanding of ethics. It probably will not like it if you told it a plan was to willingly deceive human operators. As a reward model, one might think it might be robust enough.
They may think that this is enough to work. It might be worth explaining in a concise way why this baseline does not work. Surely we must have a resource on this. Even without a link (people don’t always like to follow links from those they disagree with), it might help to have some concise explanation.
Honestly, what are the failure modes? Here is what I think:
The reward model may have pathologies the action chooser could find.
The action chooser may find a way to withhold information from the reward model.
The reward model evaluates what, exactly? Text of plans? Text of plans != the entire activations (& weights) of the model...
Awesome ideas! These ideas are some of the things missing for LLMs to have economic impact. Companies expected them to just automate certain jobs, but that’s an all or nothing solution that’s never worked historically (until it eventually does, but we’re not there yet).
One idea I thought of when reading Scott Aaronson’s Reading Burden (https://scottaaronson.blog/?p=8217), is that people with interesting opinions and with somewhat of a public presence, have a TON of reading to do, not just to keep up with current events, but to observe people’s reactions and see the trend in ideas in response to events. Perhaps this can be improved with LLMs:
Give the model a collection of your writings and latest opinions. Have it scour online posts and their comments from your favorite sources. Each post + comments section is one input, so we need longer context. Look for opportunities to share your viewpoint. Report whether your viewpoint has already been shared or refuted, or if there are points not considered in your writings. If nothing, save yourself the effort! If something, highlight the important bits.
Might be too many LLM calls depending on the sources, obviously a retrieval stage is in order. Or that bit can be done manually, we seem pretty good at finding handfuls of interesting sounding articles, and do this anyway during procrastination.