ThomasCederborg

Karma: 186

My research focus is Alignment Target Analysis (ATA). I noticed that the most recently published version of CEV (Parliamentarian CEV, or PCEV) gives a large amount of extra influence to people that intrinsically value hurting other individuals. For Yudkowsky’s description of the issue you can search the CEV arbital page for ADDED 2023.

The fact that no one noticed this issue for over a decade shows that ATA is difficult. If PCEV had been successfully implemented, the outcome would have been massively worse than extinction. I think that this illustrates that scenarios where someone successfully hits a bad alignment target pose a serious risk. I also think that it illustrates that ATA can reduce these risks (noticing the issue reduced the probability of PCEV getting successfully implemented). The reason that more ATA is needed is that PCEV is not the only bad alignment target that might end up getting implemented. ATA is however very neglected. There does not exist a single research project dedicated to ATA. In other words: the reason that I am doing ATA is that it is a tractable and neglected way of reducing risks.

I am currently looking for collaborators. I am also looking for a grant or a position that would allow me to focus entirely on ATA for an extended period of time. Please don’t hesitate to get in touch if you are curious and would like to have a chat, or if you have any feedback, comments, or questions. You can for example PM me here, or PM me on the EA Forum, or email me at thomascederborgsemail@gmail.com (that really is my email address. It’s a Gavagai / Word and Object joke from my grad student days)

My background is physics as an undergrad and then AI research. Links to some papers: P1 P2 P3 P4 P5 P6 P7 P8. (no connection to any form of deep learning)

ThomasCederborg Sep 21, 2024, 2:29 PM
1 point
0
in reply to: Tamsin Leake’s comment on: The case for more Alignment Target Analysis (ATA)
I think I see your point. Attempting to design a good alignment target could lead to developing intuitions that would be useful for ATA. A project trying to design an alignment target might result in people learning skills that allows them to notice flaws in alignment targets proposed by others. Such projects can therefore contribute to the type of risk mitigation that I think is lacking. I think that this is true. But I do not think that such projects can be a substitute for an ATA project with a risk mitigation focus.

Regarding Orthogonal:
It is difficult for me to estimate how much effort Orthogonal spends on different types of work. But it seems to me that your published results are mostly about methods for hitting alignment targets. This also seems to me to be the case for your research goals. If you are successful, it seems to me that your methods could be used to hit almost any alignment target (subject to constraints related to finding individuals that want to hit specific alignment targets).
I appreciate you engaging on this, and I would be very interested in hearing more about how the work done by Orthogonal could contribute to the type of risk mitigation effort discussed in the post. I would, for example, be very happy to have a voice chat with you about this.

ThomasCederborg Sep 20, 2024, 7:31 AM
3 points
0
in reply to: Nathan Helm-Burger’s comment on: The case for more Alignment Target Analysis (ATA)
The proposed research project would indeed be focused on a certain type of alignment target. For example proposals along the lines of PCEV. But not proposals along the lines of a tool-AI. Referring to this as Value-Alignment Target Analysis (VATA) would also be a possible notation. I will adopt this notation for the rest of this comment.

The proposed VATA research project would be aiming for risk mitigation. It would not be aiming for an answer:
There is a big difference between proposing an alignment target on the one hand. And pointing out problems with alignment targets on the other hand. For example: it is entirely possible to reduce risks from a dangerous alignment target, without having any idea how one might find a good alignment target. One can actually reduce risks without having any idea, what it even means for an alignment target to be a good alignment target.
The feature of PCEV mentioned in the post is an example of this. The threat posed by PCEV has presumably been mostly removed. This did not require anything along the lines of an answer. The analysis of Condorcet AI (CAI) is similar. The analysis simply describes a feature shared by all CAI proposals (the feature that a barely caring solid majority can do whatever they want with everyone else). Pointing this out presumably reduces the probability that a CAI will be launched by designers that never considered this feature. All claims made in the post about a VATA research project being tractable is referring to this type of risk mitigation being tractable. There is definitely no claim that a VATA research project can (i): find a good alignment target, (ii): somehow verify that this alignment target does not have any hidden flaws, and (iii): convince whoever is in charge to launch this target.
One can also go a bit beyond analysis of individual proposals, even if one does not have any idea how to find an answer. One can mitigate risk by describing necessary features (for example along the lines of this necessary Membrane formalism feature). This reduces risks from all proposal that clearly does not have such a necessary feature.
(and just to be extra clear: the post is not arguing that launching a Sovereign AI is a good idea. The post is assuming an audience that agree that it is possible that a Sovereign AI might be launched. And then the post is arguing that if this does happen, then there is a risk that such a Sovereign AI project will be aiming at a bad value alignment target. The post then further argues that this particular risk can be reduced by doing VATA)

Regarding people being skeptical of Value Alignment Target proposals:
If someone ends up with the capability to launch a Sovereign AI, then I certainly hope that they will be skeptical of proposed Value Alignment Targets. Such skepticism can avert catastrophe even if the proposed alignment target has a flaw that no one has noticed.
The issue is that a situation might arise where (i): someone has the ability to launch a Sovereign AI, (ii): there exists a Sovereign AI proposal that no one can find any flaws with, and (iii): there is a time crunch.

Regarding the possibility that there exists people trying to find an answer without telling anyone:
I’m not sure how to estimate the probability of this. From a risk mitigation standpoint, this is certainly not the optimal way of doing things (if a proposed alignment target has a flaw, then it will be a lot easier to notice that flaw, if the proposal is not kept secret). I really don’t think that this is a reasonable way of doing things. But I think that you have a point. If Bob is about to launch an AI Sovereign with some critical flaw that would lead to some horrific outcome. Then secretly working Steve might be able to notice this flaw. And if Bob is just about to launch his AI, and speaking up is the only way for Steve to prevent Bob from causing a catastrophe, then Steve will presumably speak up. In other words: the existence of people like secretly working Steve would indeed offer some level of protection. It would mean that the lack of people with relevant intuitions is not as bad as it appears (and when allocating resources, this possibility would indeed point to less resources for VATA). But I think that what is really needed is at least some people doing VATA with a clear risk mitigation focus. And discussing their finding with each other. This does not appear to exist.

Regarding other risks, and the issue that findings might be ignored:
A VATA research project would not help with misalignment. In other words: even if the field of VATA was somehow completely solved tomorrow, AI could still lead to extinction. So the proposed research project is definitely not dealing with all risks. The point of the post is that the field of VATA is basically empty. I don’t know of anyone that is doing VATA full time with a clear risk mitigation focus. And I don’t know if you personally should switch to focusing on VATA. It would not surprise me at all if some other project is a better use of your time. It just seems like there should exist some form of VATA research project with a clear risk mitigation focus.
It is also possible that a VATA finding will be completely ignored (by leading labs, or by governments, or by someone else). It is possible that a Sovereign AI will be launched, leading to catastrophe, even though it has a known flaw (because the people launching it is just refusing to listen). But finding a flaw at least means that it is possible to avert catastrophe.
PS:
Thanks for the links! I will look into this. (I think that there are many fields of research that are relevant to VATA. It’s just that one has to be careful. A concept can behave very differently when it is transferred to the AI context)

The case for more Alignment Target Analysis (ATA)

Chi Nguyen and ThomasCederborg

Sep 20, 2024, 1:14 AM

27 points

13 comments17 min readLW link

ThomasCederborg Sep 18, 2024, 4:41 AM
5 points
2
in reply to: Nathan Helm-Burger’s comment on: A problem with the most recently published version of CEV
Regarding the political feasibility of PCEV:
PCEV gives a lot of extra power to some people, specifically because those people intrinsically value hurting other humans. This presumably makes PCEV politically impossible in a wide range of political contexts (including negotiations between a few governments). More generally: now that it has been pointed out that PCEV has this feature, the risks from scenarios where PCEV gets successfully implemented has presumably been mostly removed. Because PCEV is probably off the table as a potential alignment target, pretty much regardless of who ends up deciding what alignment target to aim an AI Sovereign at (the CEO of a tech company, a designs team, a few governments, the UN, a global electorate, etc).
PCEV is however just one example of a bad alignment target. Let’s take the perspective of Steve, an ordinary human individual with no special influence over an AI project. The reason that PCEV is dangerous for Steve, is that PCEV (i): adopts preferences that refer to Steve, (ii): in a way that gives Steve no meaningful influence over the decision, of which Steve-referring preferences PCEV will adopt. PCEV is just one possible AI that would adopt preferences about Steve, in a way that Steve would have no meaningful influence over. So, even fully removing the all risks associated with PCEV in particular, does not remove all risks from this more general class of dangerous alignment targets. From Steve’s perspective, the PCEV thought experiment is illustrating a more general danger: risks from scenarios where an AI will adopt preferences that refer to Steve, in a way that Steve will have no meaningful influence over.
Even more generally: scenarios where someone successfully implements some type of bad alignment target still pose a very real risk. Alignment Target Analysis (ATA) is still at a very early stage of development, and these risks are not well understood. ATA is also a very neglected field of research. In other words: there are serious risks that could be mitigated. But those risks are not currently being mitigated. (As a tangent, I think that the best way of looking at ATA is: risk mitigation through the identification of necessary features. As discussed here, identifying features that are necessary can be a useful risk mitigation tool, even if those features are far from sufficient, and even if one is not close to any form of solution)

ThomasCederborg Sep 14, 2024, 1:49 AM
3 points
0
in reply to: Nathan Helm-Burger’s comment on: A necessary Membrane formalism feature
Regarding Corrigibility as an alternative safety measure:
I think that exploring the Corrigibility concept sounds like a valuable thing to do. I also think that Corrigibility formalisms can be quite tricky (for similar reasons that Membrane formalisms can be tricky: I think that they are both vulnerable to difficult-to-notice definitional issues). Consider a powerful and clever tool-AI. It is built using a Corrigibility formalism that works very well when the tool-AI is used to shut down competing AI projects. This formalism relies on a definition of Explanation, that is designed to prevent any form of undue influence. When talking with this tool-AI about shutting down computing AI projects, the definition of Explanation holds up fine. In this scenario, it could be the case that asking this seemingly corrigible tool-AI about a Sovereign AI proposal, is essentially equivalent to implementing that proposal.
Any definition of Explanation will necessarily be built on top of a lot of assumptions. Many of these will be unexamined implicit assumptions that the designers will not be aware of. In general, it would not be particularly surprising if one of these assumptions turns out to hold when discussing things along the lines of shutting down competing AI projects. But turns out to break when discussing a Sovereign AI proposal.
Let’s take one specific example. Consider the case where the tool-AI will try to Explain any topic that it is asked about, until the person asking Understands the topic sufficiently. When asked about a Sovereign AI proposal, the tool-AI will ensure that two separate aspects of the proposal will be Understood, (i): an alignment target, and (ii): a normative moral theory according to which this alignment target is the thing that a Sovereign AI project should aim at. It turns out that Explaining a normative moral theory until the person asking Understands it, is functionally equivalent to convincing the person to adopt this normative moral theory. If the tool-AI is very good at convincing, then the tool-AI could be essentially equivalent to an AI that will implement whatever Sovereign AI proposal it is first asked to explain (with a few extra steps).
(I discussed this issue with Max Harms here)
What links here?

ThomasCederborg Sep 13, 2024, 4:53 AM
3 points
0
in reply to: Nathan Helm-Burger’s comment on: A necessary Membrane formalism feature
I agree that focus should be on preventing the existence of a Sovereign AI that seeks to harm people (as opposed to trying to deal with such an AI after it has already been built). The main reason for trying to find necessary features, is actually that it might stop a dangerous AI project from being pursued in the first place. In particular: it might convince the design team to abandon an AI project, that clearly lacks a feature that has been found to be necessary. An AI project that would (if successfully implemented) result in an AI Sovereign that would seek to harm people. For example a Sovereign AI that wants to respect a Membrane. But where the Membrane formalism does not actually prevent the AI from wanting to hurt individuals, because the formalism lacks a necessary feature.
One reason we might end up with a Sovereign AI that seeks to harm people is that someone makes two separate errors. Let’s say that Bob gains control over a tool-AI, and uses it to shut down unauthorised AI projects (Bob might for example be a single individual, or a design team, or a government, or a coalition of governments, or the UN, or a democratic world government, or something else along those lines). Bob gains the ability to launch a Sovereign AI. And Bob settles on a specific Sovereign AI design: Bob’s Sovereign AI (BSAI).
Bob knows that BSAI might contain a hidden flaw. And Bob is not being completely reckless about launching BSAI. So Bob designs a Membrane, whose function is to protect individuals (in case BSAI does have a hidden flaw). And Bob figures out how to make sure that BSAI will want to avoid piercing this Membrane (in other words: Bob makes sure that the Membrane will be internal to BSAI).
Consider the case where both BSAI, and the Membrane formalism in question, each have a hidden flaw. If both BSAI and the Membrane is successfully implemented, then the result would be a Sovereign AI that seeks to harm people (the resulting AI would want to both, (i): harm people, and (ii): respect the Membrane of every individual). One way to reduce the probability that such a project would go ahead, is to describe necessary features.
For example: if it is clear that the Membrane that Bob is planning to use, does not have the necessary Extended Membrane feature described in the post, then Bob should be able to see that this Membrane will not offer reliable protection from BSAI (which Bob knows might be needed, because Bob knows that BSAI might be flawed).
For a given AI project, it is not certain that there exists a realistically findable necessary feature, that can be used to illustrate the dangers of the project in question. And even if such a feature is found, it is not certain that Bob will listen. But looking for necessary features is still a tractable way of reducing the probability of a Sovereign AI that seeks to harm people.
A project to find necessary features is not really a quest for a solution to AI. It is more informative to see such a project as analogous to a quest to design a bulletproof vest for Bob, who will be going into a gunfight (and who might decide to put on the vest). Even if very successful, the bulletproof vest project will not offer full protection (Bob might get shot in the head). A vest is also not a solution. Whether Bob is a medic trying to evacuate wounded people from the gunfight, or Bob is a soldier trying to win the gunfight, the vest cannot be used to achieve Bob’s objective. Vests are not solutions. Vests are still very popular amongst people who know that they will be going into a gunfight.
So if you will share the fate of Bob. And if you might fail to persuade Bob to avoid a gunfight. Then it makes sense to try to design a bulletproof vest for Bob (because if you succeed, then he might decide to wear it. And that would be very good if he ends up getting shot in the stomach). (the vest in this analogy is analogous to descriptions of necessary features, that might be used to convince designers to abandon a dangerous AI project. The vest in this analogy is not analogous to a Membrane)

ThomasCederborg Sep 11, 2024, 10:31 PM
2 points
0
in reply to: Ben Pace’s comment on: A necessary Membrane formalism feature
Thanks for the feedback! I see what you mean and I edited the post. (I turned a single paragraph abstract into a three paragraph Summary section. The text itself has not been changed)

A necessary Membrane formalism feature

ThomasCederborgSep 10, 2024, 9:33 PM

20 points

6 comments11 min readLW link

ThomasCederborg Jun 12, 2024, 7:29 PM
LW: 3 AF: 2
3
AF
in reply to: Max Harms’s comment on: Corrigibility could make things worse
Thank you for engaging. If this was unclear for you, then I’m sure it was also unclear for others.
The post outlined a scenario where a Corrigibility method works perfectly for one type of AI (an AI that does not imply an identifiable outcome, for example a PAAI). The same Corrigibility method fails completely for another type of AI (an AI that does imply an identifiable outcome, for example PCEV). So the second AI, that does have an IO, is indeed not corrigible.
This Corrigibility method leads to an outcome that is massively worse than extinction. This bad outcome is the result of two things being true, (i): the fully Corrigible first AI made this outcome possible to reach, and (ii): since the Corrigibility method worked perfectly for the first AI, the designers falsely believed that the Corrigibility method would also work for the second AI.
The second AI wants many things. It wants to get an outcome, as close as possible to IO. The Corrigibility method resulted in the second AI also wanting many additional things (such as wanting all explanations it gives to count as AE, even if this makes the explanations less efficient. And wanting to avoid implementing anything, unless informed designers want that thing to be implemented). But in practice the Corrigibility method does not change the outcome in any way (it just adds an ``explanation step″). So I think it makes sense to say that the second AI has ``zero Corrigibility″. The first AI is completely corrigible. And if the designers had only used the Corrigibility method for the first AI, then the Corrigibility method would have worked perfectly.
This is what I was trying to communicate with the first sentence of the post: ``A Corrigibility method that works for a Pivotal Act AI (PAAI) but fails for a CEV style AI could make things worse.″. I could have used that sentence as a title, but I decided against trying to include everything in the title. (I think it is ok to leave information out of the title, as long as the title is accurate. And the title is, in fact, accurate (because the first AI is genuinely corrigible. And things would have turned out a lot better, in the counterfactual situation where no one had developed any form of Corrigibility).)
One possible source of confusion, could be that you are perhaps interpreting this post as referring to some specific example scenario from your sequence. My post was mostly written before you posted your sequence. It is not meant as a comment on any specific AI in your sequence (which is why I don’t link to your sequence). But given that you had just published your sequence, maybe you were expecting my scenario to contain a single, partially corrigible AI (without an IO). That is not the scenario that I was describing in my post.
However, I could actually make the same point using a scenario with a single AI (without an IO), that is partially Corrigible. (there exists a more general danger here, that is not strongly related to the number of AI designs involved). So, here is an attempt to make the same point using such a scenario instead. A possible title for such an alternative post would be: ``a partially corrigible AI could make things worse″. (this is also a standalone scenario. It is also not meant as a response to anything specific in your sequence)
I think that one could reasonably describe Corrigibility as being context dependent. A given AI could be fully Corrigible in one context (such as preventing competing AI projects), and not Corrigible at all in another context (such as discussing Alignment Target Analysis). I think that one could reasonably refer to such an AI as being partially Corrigible. And, as will be shown below, such an AI could lead to a massively worse than extinction outcome.
Summary: Consider the case where a design team uses a Corrigibility method to build an AI Assistant (AIA). The resulting AIA does not have an IO. When the design team try to use the AIA to prevent competing AI projects, everything works perfectly. However, when they try to use the AIA to understand Alignment Target Analysis (ATA), the Corrigibility method fails completely. Let’s try two very rough analogies. Talking to the AIA about shutting down competing AI projects is very roughly analogous to using a djinn that grants wishes, while caring fully about intentions in exactly the right way. But talking to the AIA about ATA is very roughly analogous to using a djinn that grants wishes, while not caring about intentions at all (so, not a djinn with any form of preferred outcome. And not any form of ``malicious story djinn″. But also not a safe djinn).
The AIA always wants to interact in ways that counts as Acceptable Explanation (AE). When the designers ask it to shut down all hardware that is capable of running a powerful AI, it disobeys the order, and explains that human brains can, in theory, be used to run a powerful AI. When they ask it to shut down all non biological hardware that is capable of running a powerful AI, it first asks if it should delete itself, or if it should transfer itself to biological hardware. Etc. In short: it is Corrigible in this context. In particular: while talking about this topic, the definition of AE holds up.
When one of the designers asks the AIA to explain PCEV however, the definition of AE does not hold up. When the AIA was discussing the potential action of shutting down hardware, there were two components that it wanted the designers to understand, (i): the reasons for shutting down this hardware, and (ii): the effects of shutting down this hardware. In the hardware case, the designers already understood (i) reasonably well. So there was no need to explain it. When AIA is asked about PCEV, there are again two components that AIA wants the designers to understand, (i): the reasons for building PCEV, and (ii): the effects of building PCEV. PCEV is a two component thing. It is simultaneously an alignment target, and also a normative moral theory. It turns out that in this case, the designers do not actually understand (i) at all. They do not understand the normative moral theory behind PCEV. So the AIA explains this normative moral theory to the designers. If the AE definition had been perfect, this would not have been a problem. In other words: if the AIA had been completely Corrigible, then this would not have been a problem.
But it turns out that the people that designed the Corrigibility method did not in fact have a sufficiently good understanding of concepts along the lines of: ``normative moral theories″, ``explanations″, ``understanding″, etc (understanding these concepts sufficiently well was a realistic outcome. But in this scenario the designers failed to do so). As a result, the AE definition is not perfect, and the AIA is only partly Corrigible. So, the AIA ``explains″ the ``normative moral theory of PCEV″ until the designers ``understand″ it (using an explanation that counts as AE). This results in designers that feel a moral obligation to implement PCEV, regardless of what the result is. This new moral framework is robust to learning what happened. So, the result is a set of fully informed designers that are fully committed to implementing PCEV. So, the outcome is massively worse than extinction. (one way to reduce the probability of scenarios along these lines, is to make progress on ATA. Thus: even the existence of an AI that seems to be completely corrigible, is not an argument against the need to make progress on ATA. This also works as a way of gesturing at the more general point, that I was trying to gesture at in the first post)
(Wei Dai has been talking about related things for quite a while)
What links here?
- ThomasCederborg's comment on A necessary Membrane formalism feature by ThomasCederborg (Sep 14, 2024, 1:49 AM; 3 points)

ThomasCederborg Jun 11, 2024, 1:22 AM
3 points
0
in reply to: Thomas Kwa’s comment on: Corrigibility could make things worse
The first AI is genuinely Corrigible. The second AI is not Corrigible at all. This leads to a worse outcome, compared to the case where there was no Corrigible AI. Do you disagree with the statement that the first AI is genuinely Corrigible? Or do you disagree with the statement that the outcome is worse, compared to the case where there was no Corrigible AI?

Corrigibility could make things worse

ThomasCederborgJun 11, 2024, 12:55 AM

9 points

6 comments6 min readLW link

ThomasCederborg May 3, 2024, 9:44 PM
11 points
4
in reply to: davidad’s comment on: A list of core AI safety problems and how I hope to solve them
Thank you for the clarification. This proposal is indeed importantly different from the PCEV proposal. But since hurting heretics is a moral imperative, any AI that allows heretics to escape punishment, will also be seen as unacceptable by at least some people. This means that the set of Pareto improvements is empty.
In other words: hurting heretics is indeed off the table in your proposal (which is an important difference compared to PCEV). However, any scenario that includes the existence of an AI, that allow heretics to escape punishment, is also off the table. The existence of such an AI, would be seen as intrinsically bad, by people that see hurting heretics as a moral imperative (for example: Gregg really does not want a world, where Gregg has agreed to tolerate the existence of an unethical AI, that disregards its moral duty, to punish heretics). More generally: anything that improves the lives of heretics, is off the table. If an outcome improves the lives of heretics (compared to the no-AI-baseline), then this outcome is also not a Pareto improvement. Because improving the lives of heretics, makes things worse from the point of view, of those that are deeply committed to hurting heretics.
In yet other words: it only takes two individuals, to rule out any outcome, that contains any improvement, for any person. Gregg and Jeff are both deeply committed to hurting heretics. But their definitions of ``heretic″ differ. Every individual is seen as a heretic by at least one of them. So, any outcome, that makes life better for any person, is off the table. Gregg and Jeff does have to be very committed to the moral position, that the existence of any AI, that neglects its duty to punish heretics, is unacceptable. It must for example be impossible to get them to agree to tolerate the existence of such an AI, in exchange for increased influence over the far future. But a population of billions only has to contain two such people, for the set of Pareto improvements to be empty.
Another way to approach this would be to ask: What would have happened, if someone had successfully implemented a Gatekeeper AI, built on top of a set of definitions, such that the set of Pareto improvements is empty?
For the version of the random dictator negotiation baseline that you describe, this comment might actually be more relevant, than the PCEV thought experiment. It is a comment on the suggestion by Andrew Critch, that it might be possible to view a Boundaries / Membranes based BATNA, as having been agreed to acausally. It is impossible to reach such an acausal agreement when a group include people like Gregg and Jeff, for the same reason that it is impossible to find an outcome that is a Pareto improvement, when a group include people like Gregg and Jeff. (that comment also discuss ideas, for how one might deal with the dangers that arise, when one combines people like Gregg and Jeff, with a powerful and clever AI)
Another way to look at this, would be to consider what it would mean to find a Pareto improvement, with respect to only Bob and Dave. Bob wants to hurt heretics, and Bob considers half of all people to be heretics. Dave is an altruist, that just wants people to have as good a life as possible. The set of Pareto improvements would now be made up entirely of different variations of the general situation: make the lives of non heretics much better, and make the lives of heretics much worse. For Bob to agree, heretics must be punished. And for Dave to agree, Dave must see the average life quality, as an improvement on the ``no superintelligence″ outcome. If the ``no superintelligence″ outcome is bad for everyone, then the lives of heretics in this scenario could get very bad.
More generally: people like Bob (with aspects of morality along the lines of: ``heretics deserve eternal torture in hell″) will have dramatically increased power over the far future, when one uses this type of negotiation baseline (assuming that things have been patched, in a way that results in a non empty set of Pareto improvements). If everyone is included in the calculation of what counts as Pareto improvements, then the set of Pareto improvements is empty (due to people like Gregg and Jeff). And if everyone is not included, then the outcome could get very bad, for many people (compared to whatever would have happened otherwise).
(adding the SPADI feature to your proposal would remove these issues, and would prevent people like Dave from being dis-empowered, relative to people like Bob. The details are importantly different from PCEV, but it is no coincidence that adding the SPADI feature removes this particular problem, for both proposals. The common denominator is that from the perspective of Steve, it is in general dangerous to encounter an AI, that has taken ``unwelcome″ or ``hostile″ preferences about Steve into account)
Also: my general point about the concept of ``fair Pareto improvements″ having counterintuitive implications in this novel context still apply (it is not related to the details of any specific proposal).
What links here?

ThomasCederborg Apr 28, 2024, 1:48 AM
13 points
4
on: A list of core AI safety problems and how I hope to solve them
There is a serious issue with your proposed solution to problem 13. Using a random dictator policy as a negotiation baseline is not suitable for the situation, where billions of humans are negotiating about the actions of a clever and powerful AI. One problem with using this solution, in this contexts, is that some people have strong commitments to moral imperatives, along the lines of ``heretics deserve eternal torture in hell″. The combination of these types of sentiments, and a powerful and clever AI (that would be very good at thinking up effective ways of hurting heretics), leads to serious problems when one uses this negotiation baseline. A tiny number of people with sentiments along these lines, can completely dominate the outcome.
Consider a tiny number of fanatics with this type of morality. They consider everyone else to be heretics, and they would like the AI to hurt all heretics as much as possible. Since a powerful and clever AI would be very good at hurting a human individual, this tiny number of fanatics, can completely dominate negotiations. People that would be hurt as much as possible (by a clever and powerful AI), in a scenario where one of the fanatics are selected as dictator, can be forced to agree to very unpleasant negotiated positions, if one uses this negotiation baseline (since agreeing to such an unpleasant outcome, can be the only way to convince a group of fanatics, to agree to not ask the AI to hurt heretics, as much as possible, in the event that a fanatic is selected as dictator).
This post, explore these issues in the context of the most recently published version of CEV: Parliamentarian CEV (PCEV). PCEV has a random dictator negotiation baseline. The post shows that PCEV results in an outcome massively worse than extinction (if PCEV is successfully implemented, and pointed at billions of humans).
Another way to look at this, is to note that the concept of ``fair Pareto improvements″ has counterintuitive implications, when the question is about AI goals, and some of the people involved, has this type of morality. The concept was not designed with this aspect of morality in mind. And it was not designed to apply to negotiations about the actions of a clever and powerful AI. So, it should not be very surprising, to discover that the concept has counterintuitive implications, when used in this novel context. If some change in the world improves the lives of heretics, then this is making the world worse, from the perspective of those people, that would ask an AI to hurt all heretics as much as possible. For example: reducing the excruciating pain of a heretic, in a way that does not affect anyone else in any way, is not a ``fair Pareto improvement″, in this context. If every person is seen as a heretic by at least one group of fanatics, then the concept of ``fair Pareto improvements″ has some very counterintuitive implications, when it is used in this context.
Yet another way of looking at this, is to take the perspective of human individual Steve, who will have no special influence over an AI project. In the case of an AI, that is describable as doing what a group wants, Steve has a serious problem (and this problem is present, regardless of the details of the specific Group AI proposal). From Steve’s perspective, the core problem, is that an arbitrarily defined abstract entity, will adopt preferences, that is about Steve. But, if this is any version of CEV (or any other Group AI), directed at a large group, then Steve has had no meaningful influence, regarding the adoption of those preferences, that refer to Steve. Just like every other decision, the decision of what Steve-preferences the AI will adopt, is determined by the outcome of an arbitrarily defined mapping, that maps large sets of human individuals, into the space of entities that can be said to want things. Different sets of definitions, lead to completely different such ``Group entities″. These entities all want completely different things (changing one detail can for example change which tiny group of fanatics, will end up dominating the AI in question). Since the choice of entity is arbitrary, there is no way for an AI to figure out that the mapping ``is wrong″ (regardless of how smart this AI is). And since the AI is doing what the resulting entity wants, the AI has no reason to object, when that entity wants the AI to hurt an individual. Since Steve does not have any meaningful influence, regarding the adoption of those preferences, that refer to Steve, there is no reason for him to think that such an AI will want to help him, as opposed to want to hurt him. Combined with the vulnerability of a human individual, to a clever AI that tries to hurt that individual as much as possible, this means that any group AI would be worse than extinction, in expectation.
Discovering that doing what a group wants, is bad for human individuals in expectation, should not be particularly surprising. Groups and individuals are completely different types of things. So, this should be no more surprising, than discovering that any reasonable way of extrapolating Dave, will lead to the death of every single one of Dave’s cells. Doing what one type of thing wants, might be bad for a completely different type of thing. And aspects of human morality, along the lines of ``heretics deserve eternal torture in hell″ shows up throughout human history. It is found across cultures, and religions, and continents, and time periods. So, if an AI project is aiming for an alignment target, that is describable as ``doing what a group wants″, then there is really no reason for Steve to think, that the result of a successful project, would want to help him, as opposed to want to hurt him. And given the large ability of an AI to hurt a human individual, the success of such a project would be massively worse than extinction (in expectation).
The core problem, from the perspective of Steve, is that Steve has no control over the adoption of those preferences, that refer to Steve. One can give each person influence over this decision, without giving anyone any preferential treatment (see for example MPCEV in the post about PCEV, mentioned above). Giving each person such influence, does not introduce contradictions, because this influence is defined in ``AI preference adoption space″, not in any form of outcome space. This can be formulated as an alignment target feature that is necessary, but not sufficient, for safety. Let’s refer to this feature as the: Self Preference Adoption Decision Influence (SPADI) feature. (MPCEV is basically what happens, if one adds the SPADI feature to PCEV. Adding the SPADI feature to PCEV, solves the issue, illustrated by that thought experiment)
The SPADI feature is obviously very underspecified. There will be lots of border cases whose classification will be arbitrary. But there still exists many cases, where it is in fact clear, that a given alignment target, does not have the SPADI feature. Since the SPADI feature is necessary, but not sufficient, these clear negatives are actually the most informative cases. In particular, if an AI project is aiming for an alignment target, that clearly does not have the SPADI feature. Then the success of this AI project, would be worse than extinction, in expectation (from the perspective of a human individual, that is not given any special influence over the AI project). While there are many border cases, regarding what alignment targets could be described as having the SPADI feature, CEV is an example of a clear negative (in other words: there exists no reasonable set of definitions, according to which there exists a version of CEV, that has the SPADI feature). This is because building an AI that is describable as ``doing what a group wants″, is inherent in the core concept, of building an AI, that is describable as: ``implementing the Coherent Extrapolated Volition of Humanity″.
In other words: the field of alignment target analysis is essentially an open research question. This question is also (i): very unintuitive, (ii): very under explored, and (iii): very dangerous to get wrong. If one is focusing on necessary, but not sufficient, alignment target features. Then it is possible to mitigate dangers related to someone successfully hitting a bad alignment target, even if one does not have any idea of what it would mean, for an alignment target to be a good alignment target. This comment outlines a proposed research effort, aimed at mitigating this type of risk.
These ideas also have implications for the Membrane concept, as discussed here and here.
(It is worth noting explicitly that the problem is not strongly connected to the specific aspect of human morality discussed in the present comment (the ``heretics deserve eternal torture in hell″ aspect). The problem is about the lack of meaningful influence, regarding the adoption of self referring preferences. In other words, it is about the lack of the SPADI feature. It just happens to be the case, that this particular aspect of human morality is both (i): ubiquitous throughout human history, and also (ii): well suited for constructing thought experiments, that illustrates the dangers of alignment target proposals, that lack the SPADI feature. If this aspect of human morality disappeared tomorrow, the basic situation would not change (the illustrative thought experiments would change. But the underlying problem would remain. And the SPADI feature would still be necessary for safety).)
What links here?

ThomasCederborg Mar 23, 2024, 2:48 PM
9 points
7
on: Agent membranes and formalizing “safety”
I think it is very straightforward to hurt human individual Steve without piercing Steve’s Membrane. Just create and hurt minds that Steve cares about. But don’t tell him about it (in other words: ensure that there is zero effect on predictions of things inside the membrane). If Bob knew Steve before the Membrane enforcing AI was built, and Bob wants to hurt Steve, then Bob presumably knows Steve well enough to know what minds to create (in other words: there is no need to have any form of access, to any form of information, that is within Steve’s Membrane). And if it is possible to build a Membrane enforcing AI, it is presumably possible to build an AI that looks at Bob’s memories of Steve, and creates some set of minds, whose fate Steve would care about. This does not involve any form of blackmail or negotiation (and definitely nothing acausal). Just Bob who wants to hurt Steve, and remembers things about Steve from before the first AI launch.
One can of course patch this. But I think there is a deeper issue in one specific case, that I think is important. Specifically: the case where the Membrane concept is supposed to protect Steve from a clever AI that wants to hurt Steve. Such an AI can think up things that humans can not think up. In this case, patching all human-findable security holes in the Membrane concept, will probably be worthless for Steve. It’s like trying to keep an AI in a box by patching all human findable security holes. Even if it were know, that all human findable security holes, had been fully patched, I don’t think that it changes things, from the perspective of Steve, if a clever AI tries to hurt him (whether the AI is inside a box, or Steve is inside a Membrane). This matters if the end goal is to build CEV. Specifically, it means that if CEV wants to hurt Steve, then the Membrane concept can’t help him.
Let’s consider a specific scenario. Someone builds a Membrane AI, with all human findable safety holes fully patched. Later, someone initiates an AI project, whose ultimate goal is to build an AI, that implements the Coherent Extrapolated Volition of Humanity. This project ends up successfully hitting the alignment target that it is aiming for. Let’s refer the resulting AI as CEV.
One common aspect of human morality, is often expressed in theological terms, along the lines of ``heretics deserve eternal torture in hell″. A tiny group of fanatics, with morality along these lines, can end up completely dominating CEV. I have outlined a thought experiment, where this happens to the most recently published version of CEV (PCEV). It is probably best to first read this comment, that clarifies some things talked about in the post that describes the thought experiment (including important clarifications regarding the topic of the post, and regarding the nature of the claims made, as well as clarifications regarding the terminology used).
So, in this scenario, PCEV will try to implement some outcome along the lines of LP. Now the Membrane concept has to protect Steve from a very clever attacker, that can presumably easily just go around whatever patches was used to plug the human findable safety holes. Against such an attacker, it’s difficult to see how a Membrane will ever offer Steve anything of value (similar to how it is difficult to see how putting PCEV in a human constructed box, would offer Steve anything of value).
I like the Membrane concept. But I think that the intuitions that seems to be motivating it, should instead be used to find an alternative to CEV. In other words, I think the thing to aim for, is an alignment target, such that Steve can feel confident, that the result of a successful project, will not want to hurt Steve. One could for example use these underlying intuitions, to try to explore alignment targets along the lines of MPCEV, mentioned in the above post (MPCEV is based on giving each individual, meaningful influence, regarding the adoption, of those preferences, that refer to her. The idea being that Steve needs to have meaningful influence, regarding the decision, of which Steve-preferences, an AI will adopt). Doing so means that one must abandon the idea of building an AI, that is describable as doing what a group wants (which in turn, means that one must give up on CEV as an alignment target).
In the case of any AI, that is describable as a doing what a group wants, Steve has a serious problem (and this problem is present, regardless of the details of the specific Group AI proposal). Form Steve’s perspective, the core problem, is that an arbitrarily defined abstract entity, will adopt preferences that is about Steve. But, if this is any version of CEV (or any other Group AI), directed at a large group, then Steve has had no meaningful influence, regarding the adoption of those preferences, that refer to Steve. Just like every other decision, the decision of what Steve-preferences to adopt, is determined by the outcome of an arbitrarily defined mapping, that maps large sets of human individuals, into the space of entities that can be said to want things. Different sets of definitions, lead to completely different such ``Group entities″. These entities all want completely different things (changing one detail can for example change which tiny group of fanatics, will end up dominating the AI in question). Since the choice of entity is arbitrary, there is no way for an AI to figure out that the mapping ``is wrong″ (regardless of how smart this AI is). And since the AI is doing what the resulting entity wants, the AI has no reason to object, when that entity wants to hurt an individual. Since Steve does not have any meaningful influence, regarding the adoption of those preferences, that refer to Steve, there is no reason for him to think that this AI will want to help him, as opposed to want to hurt him. Combined with the vulnerability of a human individual, to a clever AI that tries to hurt that individual as much as possible, this means that any group AI would be worse than extinction, in expectation. Discovering that doing what humanity wants, is bad for human individuals in expectation, should not be particularly surprising. Groups and individuals are completely different types of things. So, this should be no more surprising, than discovering that any reasonable way of extrapolating Dave, will lead to the death of every single one of Dave’s cells.
One can of course give every individual meaningful influence, regarding the adoption of those preferences, that refer to her (as in MPCEV, mentioned in the linked post). So, Steve can be given this form of protection, without giving Steve any form of special treatment. But this means that one has to abandon the core concept of CEV.
I like the membrane concept on the intuition level. On the intuition level, it sort of rhymes with the MPCEV idea, of giving each individual, meaningful influence, regarding the adoption of those preferences, that refer to her. I’m just noting that it does not actually protect Steve, from an AI that already wants to hurt Steve. However, if the underlying intuition, that seems to me to be motivating this work, is instead used to look for alternative alignment targets, then I think it might be very useful for safety (by finding an alignment target, such that a successful project would result in an AI, that does not want to hurt Steve in the first place). So, I don’t think the Membrane concept can protect Steve from a successfully implemented CEV, in the unsurprising event that CEV will want to hurt Steve. But if CEV is dropped as an alignment target, and the underlying intuition behind this work, is directed towards looking for alternative alignment targets, then I think the intuitions that seems to be motivating this work, would fit very well with the proposed research effort, described in the comment linked above.
(this is a comment about dangers related to successfully hitting a bad alignment target. It is for example not a comment about dangers related to a less powerful AI, or dangers related to projects that fail to hit an alignment target. These are very different types of dangers. So, my proposed idea of using the underlying intuitions, to look for alternative alignment targets, should be seen as complementary. It can be done, in addition to looking for Membrane related safety measures, that can protect against other forms of AI dangers. In other words: if some scenario does not involve a clever AI, that already wants to hurt Steve, then nothing I have said, implies that the Membrane concept, will be insufficient for protecting Steve. In other words: using the Membrane concept, as a basis for constructing safety measures, might be useful in general. It will however not help Steve, if a clever AI is actively trying to hurt Steve)
What links here?

ThomasCederborg Mar 23, 2024, 2:25 PM
2 points
1
on: «Boundaries», Part 1: a key missing concept from utility theory
For a set of typical humans, that are trying to agree on what an AI should do, there does not exists any fallback option, that is acceptable to almost everyone. For each fallback option, there exists a large number of people, that will find this option completely unacceptable on moral grounds. In other words: when trying to agree on what an AI should do, there exists no place that people can walk away to, that will be seen as safe / acceptable by a large majority of people.
Consider the common aspect of human morality, that is sometimes expressed in theological terms as: ``heretics deserve eternal torture in hell″. This is a common normative position (an aspect of morality), that shows up throughout human history. It is found across cultures, and religions, and regions, and time periods. Consider Steve, for whom this is a central aspect of his morality. His morality is central to his self image, and he classify most people as heretics. The scenario, where the world is organised by an AI, that do not punish heretics, is thus seen as a moral abomination. In other words, such a scenario is a completely unacceptable fallback option for Steve (in other words, Steve would reject any AI, where this is a negotiation baseline). Hurting heretics is a non negotiable moral imperative for Steve. In yet other words: if Steve learns that the only realistic path to heretics being punished, is for the AI to do this, then no fallback position, where the AI allows them avoid punishment, is acceptable.
Bob has an even more common position: not wanting to be subjected to a clever AI, that tries to hurt Bob as much as possible (in many cases, purely normative considerations would be sufficient for strong rejection).
There is simply no way to create a well defined fallback option, that is acceptable for both Steve and Bob. When implementing an AI, that gets its goal from a set of negotiating human individuals, different types of bargaining / negotiation rules, imply a wide variety of different BATNAs. No such AI will be acceptable to both Steve and Bob, because none of the negotiation baselines, will be acceptable to both Steve and Bob. If Bob is not tortured in the BATNA, then the BATNA is completely unacceptable to Steve. And if Bob is tortured, then it is completely unacceptable to Bob. In both cases, the rejection is made on genuinely held, fully normative, non strategic, grounds. In both cases, this normative rejection cannot be changed, by any veil of ignorance (unless that veil transforms people, into something that the original person would find morally abhorrent).
In yet other words: there exists no BATNA, that a set of humans would agree to under a veil of ignorance. If the BATNA involves Bob getting tortured, then Bob will refuse to agree. If the BATNA does not involve Bob getting tortured, then Steve will refuse to agree. Thus, for each possible BATNA, there exists a large number of humans, that will refuse to agree to it (as a basis for AI negotiations), under any coherent veil of ignorance variant.
This conclusion is reached, by just skimming the surface of the many, many, different of types of minds, that exists within a set of billions of humans. It is one of the most common aspects of human morality, that is completely incompatible with the existence of a fallback position, that is acceptable to a large majority. So, I don’t think there is any hope of finding any set of patches, that will work for every unusual type of mind, that exists in a population of billions (even if one is willing to engage in some very creative definitional acrobatics, regarding what counts as agreement).
My analysis here, is that this aspect of human morality, implies a necessary (but not sufficient) feature, that any alignment target must have, for this alignment target to be preferable to extinction. Specifically: the AI in question must give each person meaningful influence, regarding the adoption of those preferences, that refer to her. We can provisionally refer to this feature as: Self Preference Adoption Decision Influence (SPADI). This is obviously very underspecified. There will be lots of border cases whose classification will be arbitrary. But there still exists many cases, where it is in fact clear, that a given alignment target, does not have the SPADI feature. Since the feature is necessary, but not sufficient, these clear negatives are actually the most informative cases. In particular, if an AI project is aiming for an alignment target, that clearly does not have the SPADI feature. Then the success of this AI project, would be worse than extinction, in expectation (from the perspective of a typical human individual that does not share Steve’s type of morality, and that is not given any special influence over the AI project).
If a project has the SPADI feature, then this implies a BATNA, that will be completely unacceptable to Steve on moral grounds (because any given heretic will presumably veto the adoption of those preferences, that demand that she be hurt as much as possible). But I think that disappointing Steve is unavoidable, when constructing a non bad alignment target. Steve is determined to demand that any AI must hurt most people, as much as possible. And this is a deeply held normative position, that is core to Steve’s self image. As long as Steve is still Steve in any coherent sense, then Steve will hold onto this rejection, regardless of what veils of ignorance one puts him under. So, if an AI is implemented, that does satisfy Steve (in any sense), then the outcome is known to be massively worse than extinction, for essentially any human individual, that is classified as a heretic by Steve (in other words: for most people). Thus, we should not look for solutions, that satisfy Steve. In fact, we can actually rephrase this as a necessary feature, that any BATNA must have: Steve must find this BATNA morally abhorrent. And Steve must categorically reject this BATNA, regardless of what type of coherent veil of ignorance, is employed. I think the SPADI feature is more informative, but if one is listing necessary features of a BATNA, then this is one such feature (and this feature can perhaps be useful, for the purpose of illustrating the fact, that we are looking for features, that are not supposed to be sufficient).
Another way to approach this, is to note that there exists many different definitions of heretic. If Gregg and Steve see each other as heretics, satisfying both is simply not possible (the specific definition of heretic is also central to the self image of both Steve and Gregg. So no coherent formulation of a veil of ignorance, will help Steve and Gregg agree on a BATNA). Satisfying any person with a morality along the lines of Steve, implies an outcome far worse than extinction for most people. Satisfying all people along the lines of Steve is also impossible, even in principle. Thus, it is difficult to see, what options there are, other than simply giving up on trying to give Steve an acceptable fallback option (in other words: we should look for a set of negotiation rules, that imply a BATNA, that is completely unacceptable to Steve, on fully normative, moral, grounds. In yet other words: a necessary feature of any veil of ignorance mediated, accusal agreement, is that it is strongly rejected by every person, with a morality along the lines of Steve). Thus, the fact that Steve would find any AI project with the SPADI feature, morally abhorrent, is not an argument against the SPADI feature. (both Steve and Gregg would obviously reject any notion, that they have similar moralities. This is an honest, non strategic, and strong rejection. And this rejection would remain, under any coherent veil of ignorance. But there is not much that could, or should, be done about this)
The SPADI feature is incompatible with building an AI, that is describable as ``doing what a group wants″. Thus, the SPADI feature is incompatible with the core concept of CEV. In other words: the SPADI feature is incompatible with building an AI that, in any sense, is describable as implementing the Coherent Extrapolated Volition of Humanity. So, accepting this feature, means abandoning CEV as an alignment target. In yet other words: if some AI gives each individual meaningful influence, regarding the decision, of which preferences to adopt, that refer to her. Then we know that this AI is not a version of CEV. In still other words: while there are many border cases, regarding what alignment targets could be described as having the SPADI feature, CEV is an example of a clear negative (because doing what a group wants, is inherent in the core concept, of building an AI, that is describable as: implementing the Coherent Extrapolated Volition of Humanity).
Discovering that building an AI that does what a group wants the AI to do, would be bad for the individuals involved, should in general not be particularly surprising (even before taking any facts about those individuals into account). Because groups and individuals are completely different types of things. There is no reason to be surprised, when doing what one type of thing wants, is bad for a completely different type of thing. It would for example not be particularly surprising to discover, that any reasonable way of extrapolating Dave, will lead to all of Dave’s cells dying. In other words, there is no reason to be surprised, if one discovers that Dave would prefer, that Dave’s cells not survive.
Similarly, there is no reason to be surprised, when one discovers that all reasonable ways of defining ``doing what a group wants″ is bad for the individuals involved. A group is an arbitrarily defined abstract entity. Such an entity is pointed at, using an arbitrarily defined mapping, from billions of humans, into the set of entities, that can be said to want things. Different mappings imply completely different entities, that all want completely different things (a slight definition change, can for example lead to a different BATNA, which in turn leads to a different group of fanatics dominating the outcome). Since the choice of which specific entity to point to, is fully arbitrary, no AI can discover that the mapping, that is pointing to such an entity ``is incorrect″ (regardless of how smart this AI is). That doing what this entity wants is bad for individuals, is not particularly surprising (because groups and individuals are completely different types of things). And an AI, that does what such an entity wants it to do, has no reason whatsoever, to object, if that entity wants the AI to hurt individuals. So, discovering that doing what such an entity wants, is bad for individuals, should in general not be surprising (even before we learn anything at all, about the individuals involved).
We now add three known facts about humans and AI designs, (i): a common aspect of human morality, is the moral imperative to hurt other humans (if discovering that no one else will hurt heretics, then presumably the moral imperative will pass to the AI. An obvious way of translating ``eternal torture in hell″ into the real world, is to interpret it as a command to hurt as much as possible), (ii): a human individual is very vulnerable to a clever AI, trying to hurt her as much as possible (and this is true for both selfish, and selfless, humans), and (iii): if the AI is describable as a Group AI, then no human individual can have any meaningful influence, regarding the adoption, of those preferences, that refer to her (if the group is large, and if the individual in question is not given any special treatment). These three facts very strongly imply that any Group AI, would be far worse than extinction, for essentially any human individual, in expectation.
I have outlined a thought experiment that might help to make things a bit more concrete. It shows that a successful implementation of the most recently published version of CEV (PCEV), would lead to an outcome, that would be far, far, worse than extinction. It is probably best to first read this comment, that clarifies some things talked about in the post that describes the thought experiment (the comment includes important clarifications regarding the topic of the post, and regarding the nature of the claims made, as well as clarifications regarding the terminology used).
To me, it looks like the intuitions that are motivating you to explore the membrane concept, is very compatible with the MPCEV proposal, in the linked post (which modifies PCEV, in a way that gives each individual, meaningful influence, regarding the adoption of those preferences, that refer to her). If CEV is abandoned, and the Membrane concept is used to describe / look for, alternative alignment targets, then I think this perspective might fit very well with my proposed research effort (see the comment mentioned above for details on this proposed research effort).
What links here?

ThomasCederborg Mar 23, 2024, 2:02 PM
1 point
0
on: A problem with the most recently published version of CEV
This comment is trying to clarify what the post is about, and by extension clarify which claims are made. Clarifying terminology is an important part of this. Both the post and my research agenda is focused on the dangers of successfully hitting a bad alignment target. This is one specific subset, of the existential threats that humanity face from powerful AI. Let’s distinguish the danger being focused on, from other types of dangers, by looking at a thought experiment, with an alignment target that is very obviously bad. A well intentioned designer named Bill, who shares your values, sincerely believes that it is a good idea to start an AI project, aiming for a Suffering Maximising AI (SMAI). Bill’s project might result in an AI that wants to do something along the lines of building lots of tiny pictures of sad faces, due to a failure to get the AI to want to maximise, a reasonable definition of Suffering. This type of scenario is not what I am focused on. No claim made in the post refer to any scenario along those lines. In this case, Bill’s project is described as having failed. In other words: the aimed for alignment target was not hit. These types of dangers are thus out of scope, of both the post, and my research agenda. It is a very different type of danger, compared to the dangers that come from scenarios where Bill succeeds, and successfully hits the SMAI alignment target. Success would result in an AI, that wants to maximise something, that can be reasonably described as suffering (and little pictures of sad faces, is not a reasonable way of defining suffering). It is also possible that Bill’s project results in an AI that wants to build a lot of molecular Squiggles, due to an issue that has nothing to do with any definition of Suffering (this is also a failure, and it is also out of scope).
Dangers along the lines of Squiggles and pictures of sad faces can be dealt with by improving alignment techniques. Or by explaining to Bill that he does not currently know how to hit an alignment target. All strategies along those lines are out of scope, simply because nothing along those lines, can ever be used to mitigate the dangers, that come from successfully hitting a bad alignment target. Nothing along those lines can help in scenarios, where it would be bad for the project to succeed (which is the case for Bill’s SMAI project). Showing Bill that he does not know how to hit an alignment target, stops working when Bill learns how to hit an alignment target. And helping Bill improve his alignment techniques, dramatically increases the dangers that comes from successfully hitting a bad alignment target. The best way of dealing with such dangers, is by showing Bill, that it would be bad, for his project to succeed. In other words: to show Bill, that his favoured alignment target, is a bad alignment target. My research effort, which the post is a part of, is focused on analysing alignment targets. The reason that such analysis is needed, is that not all bad alignment targets, are as obviously bad as SMAI. The post can be used to illustrate the fact, that alignment target analysis can be genuinely unintuitive.
In other words: the purpose of the proposed research effort, is not to help Bill successfully implement SMAI. Or to help Bill find a good description of SMAI. Or to convince Bill to aim for a different version of SMAI. Instead, the purpose of the proposed alignment target analysis research effort, is to help Bill see that he is aiming for a bad alignment target. So that he will choose to abandon his SMAI project.
One could add a safety measure to Bill’s SMAI project, that might give Bill a chance to try again, in case of a bad outcome. This can reduce many different types of dangers, including dangers from successfully hitting a bad alignment target (less formally: one can add a do over button). But since no such safety measure is guaranteed to work, no such measure can make it ok, for Bill to start his SMAI project. And, crucially, no such safety measure can be relevant to the question, of whether or not SMAI is a bad alignment target. Thus, analysing such safety measures, does not count as analysing alignment targets. Since they are not relevant to the proposed research effort, analysing such safety measures are also out of scope. It would not be particularly surprising, if analysing such safety measures turns out to be more important than analysing alignment targets. But this does not change the fact that proposed safety measures are irrelevant, when analysing alignment targets. In other words: they are a separate, and complementary, method of reducing the type of danger that I am focused on. (less formally: a do over button might save everyone from an outcome massively worse than extinction. So they are definitely not useless in general. They are just useless for the specific task of analysing alignment targets. And since they can never be guaranteed to work, they can never replace alignment target analysis. A do over button can also be added to many different proposed alignment targets. Which means that they are not very helpful when one is comparing two alignment targets)
I don’t use the term alignment target for proposals along the lines of a Pivotal Act AI (PAAI), or any other AI whose function is similar to the typical functions of proposed treaties (such as buying time). So, CEV is an alignment target, but no proposal along the lines of a PAAI, is referred to as an alignment target. To get a bit more concrete, regarding what counts as an alignment target in my terminology, let’s say that a set of designers have an AI proposal, called AIX. If AIX is the type of AI, that implies a decision, regarding which goal to give to a successor AI, then AIX counts as an alignment target. The act of building AIX, implies a choice of alignment target. So, to build AIX, one must be able to analyse alignment targets.
If, on the other hand, the designers do not plan for AIX to play a role in the decision process, regarding what goal to give to a successor AI, then AIX does not count as an alignment target. In this latter case, building AIX might be possible without being able to analyse alignment targets (and building AIX, at a point in time when one does not know how to analyse alignment targets, might be a good idea). But the need to analyse alignment targets would remain, even if this latter type of project is successful.
If AIX is an alignment target, then adding something along the lines of a last judge off switch, to AIX, does not change the fact that AIX is an alignment target. An AI project might add such a feature, specifically because the designers know, that the aimed for alignment target might be bad. In this case, the project is still described as aiming for a bad alignment target. Such designers are not certain about the alignment target in question. And they have taken precautions, in case it turns out to be a bad alignment target. So, it might not be possible to discover anything, that would be very surprising to the designers. But it is still possible to discover that the aimed for alignment target is bad. This might still help a group of not-very-surprised designers, to reach the not-very-surprising-conclusion, that they should abandon the project (since this type of add on might fail, no such add on can ever make it reasonable to start a project that is aiming for a bad alignment target. In other words: it was wrong of these designers, to start this project. But in this scenario, the designers started the project, while thinking that the alignment target might be good. So, there is nothing in this scenario that strongly indicates, that it will be impossible to reason with them, in case one discovers that their alignment target is bad).
A PAAI project might be one part of a larger project plan, that at a later stage calls for some other AI project, that will be trying to hit an agreed upon alignment target. This is similar to how a treaty might be part of such a larger project plan. This larger project counts as aiming for an alignment target. But the alignment target is solely determined by the details of second AI project. The details of any treaty / PAAI / etc is, in my terminology, irrelevant to the question: what alignment target is the larger project aiming for?
Dangers related to successfully hitting a bad alignment target, is a different class of dangers, compared to dangers related to a project, that fails to hit a given alignment target. Preventing dangers related to successfully hitting a bad alignment target, requires a dedicated research effort, because some important measures that can reduce this risk, does not reduce other AI related risks (and are thus unlikely to be found, without a dedicated effort). Specifically, one way to reduce this specific risk, is to analyse alignment targets, with a view to finding some set of features, that are necessary for safety. I propose to take the perspective of one individual that is not given any special treatment. Then one can look for a set of necessary features, that an alignment target of a given project must have, for it to be rational, for such an individual to support this AI project. This should be done, while making as few assumptions as possible regarding this individual. Such a set would help someone, that is trying to construct a new alignment target. Since these features are all of the necessary-but-not-sufficient type, the fact that a project has them all, is not enough to convince anyone, to support the project. But they can help an individual (for whom none of the assumptions made, are false), decide to oppose a project, that is aiming for an alignment target, that lacks one of the features in such a set. The post describes one such necessary feature: that each individual, is given meaningful influence, regarding the adoption of those preferences, that refer to her. In many cases, it will be unclear whether or not some proposal has this feature. But in some cases, it will be clear, that a given proposal does not have this feature. And since it is a necessary but not sufficient feature, it is these clear negatives, that are the most useful. It is for example clear, that no version of CEV, can have this feature. This feature is incompatible with the core concept, of building an AI, that is in any way describable as: implementing the Coherent Extrapolated Volition of Humanity. In other words: in order to build an AI that has this feature, one has to choose some other alignment target. This seems to be a somewhat controversial feature. Possibly because of this incompatibility.
Such features are central to the proposed research effort. And the feature mentioned is the most important part of the post. But, in order to avoid controversies and distractions, this comment will not involve any further discussion of CEV. I will instead make the distinctions that I need to make, by relying exclusively on examples with other alignment targets (whose status as bad alignment targets is not being disputed).
A simple, well understood, and uncontroversial example of a bad alignment target, is a Suffering Reducing AI (SRAI). SRAI cares about what happens to humans, and will not treat humans in an uncaring, strategic way. For example: SRAI would not accept an offer where the solar system is left alone in exchange for resources (with the only intervention being to prevent any other AI from being built). This rejection is not related to any specific detail, or any specific definition. Any reasonable definition of suffering, leads to an AI, that will reject such an offer. Because killing all humans is inherent in the core concept, of reducing suffering. For any reasonable definition of Suffering, accepting the offer, would lead to the continuation of a lot of Suffering. Unless there exists some way of using the offered resources, to reduce some other, larger, source of Suffering, the offer will be rejected (for the rest of this text, it will be assumed that no such opportunity exists).
Regardless of which reasonable set of definitions is used, a successful SRAI project, simply leads to an AI that rejects all such offers (and that kills everyone). In other words, if a SRAI project results in an AI that rejects the offer and kills everyone, then this behaviour is very consistent, with the project having succeeded. The core issue with any SRAI project, is that the project would be aiming for a bad alignment target. The path towards a successfully implemented SRAI project, is simply not relevant to the question, of whether or not this is a bad alignment target. If some treaty, or a Pivotal Act AI, or some other plan for buying time, or some other clever trick, was involved at some point. Then this has no impact whatsoever on the question: is SRAI a bad alignment target? Similarly: if the path to a successful SRAI implementation, involved a last judge off switch, which successfully prevented an issue that would have led to an AI, that would have been glad to accept the offer mentioned above (because that AI would have been able to use the offered resources, to build lots of molecular squiggles), then this is equally irrelevant to the question: is SRAI a bad alignment target? If some other type of do over button, happens to continue to work, until the project is abandoned, then this is equally irrelevant, to the question: is SRAI a bad alignment target? More generally: no try-again-safety-measure can be relevant to this question. And no other detail of the larger project plan, can, be relevant to this question. (it is possible that a failed SRAI project will result in some AI that rejects the offer. Only one positive claim is made: If the resulting AI accepts the offer, then the project did not successfully hit the aimed for alignment target. Or, in other words: if the project does successfully hit the aimed for alignment target, then the resulting AI will reject the offer)
When saying that SRAI is a bad alignment target, there is no intended implication, that an AI project aiming for the SRAI alignment target, will lead to a bad outcome. It could be that some set of safety measures are implemented. And these could hold up until the designers see that the aimed for alignment target is a bad alignment target. Such a project is a bad idea. It is an unnecessary risk. And if it successfully hits the alignment target that it is aiming for, then it will get everyone killed. Success of this project leads to extinction (regardless of path to success, and regardless of the details of the definitions). But there is no implied claim that the act of starting such a project would, in fact, lead to a bad outcome.
It is possible to use other definitions for things such as: ``alignment target″, and ``successfully hitting an alignment target″, and ``the project succeeding″. But the way that these phrases are being defined here, by using the various examples throughout this comment, makes it possible to refer to important concepts in a clear way. The words used to point to these concepts could be changed. But it must be possible to refer to the concepts themselves. Because they are needed in order to make important distinctions.
Now, let’s return to the SMAI alignment target, mentioned at the beginning of this comment. Unless something very strange happens, the outcome of Bill’s SMAI project, would accept the offer mentioned above (this is probably a good deal for the resulting AI, whether or not the project hits the aimed for alignment target. The offered resources can for example be used to create: little pictures of sad faces, or Suffering, or Squiggles). Thus, such an offer can not be used to distinguish different types of scenarios from each other, in the SMAI case. For a SMAI project, it is more informative to ask what it would mean from a human perspective, to give the resulting AI resources. In the SMAI case, this question is better for illustrating the importance of distinguishing between different types of AI related dangers.
So, consider the case where the outcome of this SMAI project is fully contained within some lifeless region. Now, let’s ask what it would mean, to give it more resources. Let’s say that the result of a SMAI project, wants to build lots of little pictures of sad faces (due to a failure to get the AI to want to maximise a reasonable definition of Suffering), or little molecular Squiggles (due to an implementation failure, that is unrelated to the definition of Suffering). In these two cases, it seems like it does not matter much, if resources are destroyed, or given to this AI (assuming that it is fully contained). Dangers related to outcomes along these lines, are out of scope, as they are the results of failed projects. If a SMAI project succeeds however, then it would be far better to destroy those resources, than for SMAI to have those resources. If a SMAI project successfully hits the SMAI alignment target, the details of the definitions would also matter. Possibly a great deal. Successfully hitting the SMAI alignment target, can lead to a very wide range of outcomes, depending on the details of the definitions. Different sets of reasonable definitions, lead to outcomes, that are different in important ways (the SMAI project being successful, means that the resulting AI will want to maximise some reasonable definition of suffering. Thus, the outcome of a successful SMAI project, will be a bad outcome. But different definitions still lead to importantly different outcomes). The dangers involved with successfully hitting the SMAI alignment target, is importantly different from the case where the project fails to hit an alignment target. One very important distinction, is that each scenario, in this very large set, of very diverse types of scenarios, are all best dealt with, by helping Bill see that he is aiming for a bad for alignment target.
(it is possible that a failed SMAI project will result in an AI, such that preventing it from having resources is important (even under a strong containment assumption). Only one positive claim is made: If humans are essentially indifferent to giving the resulting AI resources, then the project did not successfully hit the aimed for alignment target. Or, in other words: if the project does successfully hit the aimed for alignment target, then it is important to keep resources away from the resulting AI)
Shifting resources from the result of one successful SMAI project, to the result of another successful SMAI project, might be very valuable. But issues along those lines are also out of scope of the proposed research effort, of analysing alignment targets. In other words: the point of talking to Bill, is not to help Bill find a better description of SMAI. But instead to help Bill see, that he should aim for a different alignment target. These two objectives are very different. Just as it would be a bad idea to help Bill hit the SMAI alignment target. It would also be a bad idea to help Bill describe SMAI. And while different versions of SMAI might lead to importantly different outcomes, the purpose of the proposed research project, is not to switch from one version of SMAI, to another version of SMAI. The purpose of the proposed research effort, is instead to help Bill see, that SMAI is a bad alignment target.
The reason that the effort to analyse alignment targets needs a dedicated effort, is that the best way to stop the two projects mentioned above, is by pointing out, that it would be a bad thing for them to succeed. If it is possible to stop a project in this way, then this is a better option, than to implement safety measures, that might (or might not) hold up until the project is abandoned. For SRAI and SMAI, the needed insights already exists. So, to stop these two projects, there is no need for further analysis. But if I take an example where there is no consensus, that the alignment target in question is in fact bad. Then the distinctions that I am trying to make in this comment will get lost, in arguments about whether or not the alignment target in question, is in fact bad. So I will stick with SRAI and SMAI.
In other words: To deal with bad alignment targets, that has not currently been discovered to be bad, dedicated effort is needed. Efforts that deal with other types of dangers, does not need the types of insights, that are needed to analyse alignment targets. And such efforts, are thus not likely to lead to such insights. It is not known how long it would take a serious, dedicated, research effort to advance to the point, where the needed insights become possible to see. Partly, this is because it is not known which nice-sounding-but-actually-bad alignment targets, will be proposed in the future. We also don’t know how long there will be to work on this issue. Even if we reason from the assumption, that some alignment target will get successfully hit, there would still be a lot of remaining uncertainty. We would still not know how long time there will be, to find the needed insights. Since we don’t know how much time it would take, or how much time there will be, there is no particular reason to think that the effort will be completed in time. Combined with the stakes involved, this implies urgency. Despite this urgency, currently there exists no serious research project, that is dedicated to alignment target analysis. One positive thing that happens to be true, is that in scenarios where alignment target analysis is needed, there will probably be time to do such analysis. There is no particular reason to think that there will be enough time. But it seems very likely, that there will be a significant amount of time. Another positive thing, that happens to be true, is that one can prevent these types of dangers, without arriving at any actual answer (the dangers from a given bad alignment target, can for example be mitigated by noticing, that it is a bad alignment target. This can be done, without having any example of a good alignment target). Yet another positive thing, that also happens to be true, is that if the needed insights are in fact generated in time, then these insight will probably only have to be explained, to the types of people, that are capable of hitting an alignment target.
If one views the situation from a higher level of abstraction, then the lack of a dedicated research effort is even more strange. It obviously matters, what alignment target an AI project is aiming for. One way to phrase this, is that it simply cannot be the case, that the aimed for alignment target is both irrelevant to the outcome, and simultaneously supposed to reduce division, and unify people towards a common goal. (if an alignment target has been set in stone, then one might be able to reasonably argue that detailed descriptions of this alignment target would not be particularly valuable to a given project. But such arguments are simply not applicable, to suggestions that a proposed project should to aim for a different alignment target)
If an AI project is aiming at a bad alignment target, then this can not be fixed by adding some set of safety measures to this project. And a set of safety measures, is simply not relevant, when one tries to determine, whether or not the aimed for alignment target, is a bad alignment target. And since no safety measure is guaranteed to work, such measures can never make it ok, to launch an AI project, that is aiming at a bad alignment target. A very careful SRAI project, that is implementing a set of very well constructed safety measures, might obviously lead to SRAI. It is still putting everyone in unnecessary danger. One implication of this, is that if someone argues that an SRAI project is aiming for a bad alignment target, then this argument can not be countered, by pointing at safety measures. It is important to emphasise, that it is completely irrelevant, what this set of safety measures is. Such a counterargument can always be safely dismissed out of hand, without having any idea, of which specific type of do-over-button is being referred to. Such safety measures simply, can not, change the fact that SRAI is a bad alignment target. (if such a do-over-button is added to a SRAI project, and this do-over-button happens to work, then it can save everyone from getting killed by SRAI. So it is definitely not useless in general. It is just useless, for the specific task, of analysing alignment targets. In other words: it can stop everyone from getting killed. But it can not be used to build a counterargument, when someone points out that SRAI is a bad alignment target. In yet other words: it is only the counterargument that can be safely dismissed. Some specific safety measure, that such a counterargument is built around, might turn out to be more important than the entire field of alignment target analysis. So dismissing such a safety measure, based on the fact that it is being used as part of an invalid counterargument, is definitely not something that can be safely done)
So far, we have talked about safety measures, that might allow a designer to try again. Let’s now turn our attention to what might perhaps be roughly described as a ``goal softener″, that could be added to an alignment target. In other words: a type of add on that will modify the behaviour implied by an alignment target (not an add on, that will allow a designer to abandon an alignment target, if it implies bad behaviour). Let’s write sSRAI for a soft Suffering Reducing AI, that will determine the actual outcome, and that respects the underlying intentions, behind an effort to get it to (for example) ``avoid optimising too hard″. Specifically, sSRAI wants to reduce suffering, but sSRAI wants to do this, in a way that respects the intentions behind the designers attempts, to get it to do soft optimisation, of the SRAI alignment target.
Let’s use an undramatic example scenario, to make a few points, that have wider applicability. Consider a successfully implemented sSRAI. sSRAI acts in a way that is genuinely in line with the underlying ideas and intentions, behind an effort to get the AI to avoid optimising too hard. sSRAI wants to act in accordance with the spirit of the soft optimisation principles that the designers were trying to implement. In other words: this feature is not used as a way to buy time, or as a way to get a second chance in case of failure (less formally: It is not a speed bump. And it is not a do over button). Specifically, sSRAI will always act in the world, in a way that is consistent with being uncertain, regarding how precisely Suffering should be defined. And sSRAI will avoid any action, that the designers would find alarming or disconcerting in any way. And sSRAI will avoid any action that would be perceived as weird or dramatic. And sSRAI would like to avoid any rapid change, taking place in society or culture. And sSRAI will avoid any action that would be seen as deceptive or manipulative.
Now, let’s explore what sSRAI might do in the world, to see if this has turned the SRAI alignment target, into a good alignment target. sSRAI could, for example, offer a process of sSRAI guided self discovery, that causes people to settle on some new way of viewing life, that is far more fulfilling. There is a wide range of views that people arrive at, but almost all include the idea, that it is best to not have more than one child. Every year, asking sSRAI for such guidance becomes slightly more mainstream, and every year some fraction of the population asks for this. sSRAI also declines to create any set of circumstances, that would dramatically alter human society. For example, sSRAI declines to create any set of circumstances, that would make humans want to go on living indefinitely (as opposed to just wanting to live for a very, very long time). sSRAI also takes actions to increase comfort, happiness, and to reduce suffering. Basically, taking careful actions that lead to people living very long and very happy lives (this does imply gradual but significant changes. But those societal changes are seen as acceptable, as they are the least dramatic changes, that can achieve the objective of significantly reducing suffering in the short term).
At no point on this path, is sSRAI deviating from the spirit of the implied instructions. At no point is sSRAI deviating from what it is supposed to do: which is to reduce suffering, while adhering to the underlying intentions, of avoiding drastic / weird / etc actions, or rapid changes, or manipulations, etc, etc. At all points on this path, is it the case, that all of its plans, are entirely in line with genuinely wanting to respect the underlying intention, of soft optimisation. For example: the process of sSRAI guided self discovery, is not some trick, to get people to want weird things. It is genuinely a process, designed to lead to some viewpoint, that in turn leads to less suffering (and it never results in any viewpoint, that is a weird outlier, in the set of common, pre-AI, viewpoints). The process also takes the other aspirations of a given individual into account, and leads to viewpoints that additionally does things like increasing happiness, or improving artistic output (since taking peoples wishes into account when possible, is also implied by some of the underlying intentions, that lead the designers to try to get the AI to do soft optimisation. Specifically: intentions related to not having tunnel vision). And no one is ever manipulated into asking for it. It is simply offered to everyone. And then word spreads, that it is great. For example: sSRAI did not design it, with the design constraint, that it must become dominant (including such a constraint when designing the process, would have violated the designers underlying intentions. Specifically, they would see this as manipulation). But the process leads to outcomes, that people will eventually want, when it stops being seen as weird to ask for it. The process also does not cause any dramatic changes to peoples lives or personalities.
Additionally, and completely in line with the underlying intentions of the safety measures, sSRAI declines to enable any form of dramatic augmentation procedure (explaining, completely honestly, to anyone that wants dramatic augmentation, that this would both lead to dramatically increased suffering, and to rapid societal change). sSRAI will never decide to explain any concept or fact, that would result in transformative change or increased suffering (or allow any AI project, or any augmentation project, that would lead to such concepts or facts being discovered). In general, sSRAI will prevent any AI project, that would have dramatic impact on society or culture (again straightforwardly in line with the underlying intentions of the designers. Because they intended to prevent drastic AI actions in general, not just drastic sSRAI actions). This overall behaviour is a genuinely reasonable way of interpreting: ``exert a pushing force towards the outcome implied by the aimed for alignment target. But make sure that you always act in a way, that I would describe as pushing softly″.
Even conditioned on a successful implementation of sSRAI, the specific scenario outlined above would obviously never happen (for the same reason that the specific chess moves that I can come up with, are very unlikely to match the actual chess moves, of a good chess player, whose moves I am trying to predict). A more realistic scenario is that sSRAI would design some other path to a lifeless universe, that is a lot more clever, a lot faster, a lot less obvious, and even more in line with underlying intentions. If necessary, this can be some path, that simply can not be explained to any human, or any result of any of the augmentation procedures, that sSRAI will decide to enable. A path that simply can not be specified in any more detail than: ``I am trying to reduce suffering, for a wide range of definitions of this concept, while adhering to the underlying intentions, behind the design teams efforts, to prevent me from optimising too hard″. SRAI is simply a bad alignment target. And there does not exists any trick, that can change this fact, even in principle (even under an assumption of a successfully implemented, and genuinely clever, trick). If a last judge off switch happens to work, then the project mentioned above might be shut down by some extrapolated version of a last judge, with no harm done (in other words: there is no claim that a sSRAI project will lead to a lifeless universe. Just a claim that the project is a bad idea to try, and that it represents an unnecessary risk, because SRAI is a bad alignment target).
In other words: the core concept of reducing suffering, implies killing everyone. And this remains true, even if one does manage to get an AI to genuinely want to make sure that everyone stops living, using only slow, ethical, legal, non deceptive, etc, etc, methods. Or get the AI to use some incomprehensible strategy (that avoids sudden changes, or dramatic actions, or unpleasant aesthetics, or manipulation, etc, etc). SRAI is unsafe, because the implied outcome is unsafe. More generally: SRAI is unsafe because SRAI cares about what happens to you. This is dangerous, when it is combined with the fact, that you never had any meaningful influence, over the decision, of which you-referring-preferences, SRAI should adopt. Similar conclusions hold for other bad alignment targets. And not all of the implied outcomes are this undramatic, or this easy to analyse. For example: sSMAI does not have a known, easy to analyse, endpoint, that sSMAI is known to softly steer the future towards. But the danger implied by a soft push, towards a bad future, is not reduced by the fact that we are having trouble predicting the details. It makes it harder to describe the danger of a soft push in detail, but it does not reduce this danger.
We are now ready to illustrate the value of analysing alignment targets, by noting the difference between a world organised by sSRAI, and a world organised by sSMAI. Let’s consider an AI that genuinely wants to respect the designers underlying intentions, that motivated them to try to get the AI to avoid optimising too hard. We can refer to it as soft AI, or sAI. Let’s assume that sAI does not do anything that causes human culture to stray too far away from current value systems and behaviours. It does not take drastic actions, or manipulate, or deceive, etc, etc. It only acts, in ways that would be genuinely reasonable to view, as ``pushing softly″. It still seems clearly better, to have an AI that softly pushes to reduce suffering, than to have an AI that softly pushes to maximise suffering. In yet other words: analysing alignment targets will continue to matter, even if one is reasoning from some very, very, optimistic set of assumptions (the difference is presumably al lot less dramatic than the difference between SMAI and SRAI. But it still seems valuable to switch from sSMAI to sSRAI).
We can actually add even more optimistic assumptions. Let’s say that Gregg’s Method (GM), is completely guaranteed to turn SRAI into GMSRAI. GMSRAI will create a thriving and ever expanding human civilisation, with human individuals living as long lives as would be best for them. Adding GM to any other alignment target, is guaranteed to have equally dramatic effects, in a direction that is somehow guaranteed to be positive. It seems very likely, that even in the GMSRAI case, there will still be a remnant left, of the push towards a lifeless universe. So, the value of analysing alignment targets would remain, even if we had Greggs Method (because a remnant of the SRAI push, would still be preferable to a remnant of the SMAI push. So, finding a better alignment target, would remain valuable, even if we counterfactually had someone like Gregg to help us. In other words: while finding / modifying someone into / building / etc, a Gregg would be great, this would not actually remove the need to analyse alignment targets). (and again: the reason that analysing alignment targets requires a dedicated research effort, is that not all alignment targets are as easy to analyse and compare, as SRAI and SMAI)
Before leaving the sSRAI thought experiment, it makes sense to make three things explicit. One is that ideas along the lines of soft optimisation, have not been shown to be dead ends or bad ideas. Because nothing along these lines can stop a bad alignment target from being bad, even in principle. Basically: unless the AI project is secretly trying to implement an AI that does, whatever a single designer wants the AI to do (in which case the claimed alignment target is irrelevant. From the perspective of an individual without any special influence, such a project is equivalent to a ``random dictator AI″ (regardless of the words used to describe the project)), then it will continue to matter, that SRAI exerts a pushing force towards a lifeless universe. Either this push will impact the outcome, or the alignment target is an empty PR slogan (either it has an influence on the outcome. Or it does not have an influence on the outcome). So, nothing said here is incompatible with some concept along the lines of soft maximisation turning out to be a genuinely valuable and important concept (because we knew that it would fail to solve the task of turning a bad alignment target into a good alignment target, the second we knew what type of thing, was supposed to perform what type of task).
Secondly, the fact that no one in the sSRAI scenario recognises, that SRAI implies a push towards a lifeless universe, makes things look strange (the same thing makes the various SMAI examples look even stranger). This is unavoidable, when one constructs an example with an alignment target, that is universally understood to be a bad alignment target (if the audience understands this, then the people in the thought experiment should also understand this). In other words: no sSRAI project poses any actual threat, because SRAI is already known to be a bad alignment target. The goal of analysing alignment targets, is to neutralise threats from other alignment targets, that are not currently known to be bad.
And finally, it makes sense to make one particular distinction explicit: the sSRAI thought experiment is not discussing a safety measure of a type that is designed to give a designer a chance to try again, in case of a bad outcome (those are discussed in other parts of this comment). Thus, we have not shown anything at all, regarding this version of the concept. In more straightforwards, but a lot less exact, words: the sSRAI thought experiment is about the goal softener version of the concept. Not the do over button version. The issue with the goal softener is that it only dilutes the bad parts. And the issue with the do over button is that it might fail. So even using both don’t make it ok to start a project that aims at a bad alignment target. Goal softeners and do over buttons can not change a bad alignment target into a good alignment target. They can however interact with each other. For example: when implementing a last judge off switch, there is a strong need to limit the extrapolation distance. Because extrapolation is not intuitive for humans. This need does not mix well with goal softeners. Because goal softeners can make it harder to see the bad effects of a bad goal. Seeing the push towards a lifeless universe in the sSRAI case requires no extrapolation at all. Because this push is already a universally known feature. Because SRAI is already well understood. For example: everyone already knows exactly why SRAI is fundamentally different from Clippy. But when trying to see a less obvious problem, the benefit of modest extrapolation can be canceled out by the obscuring effect of a well designed goal softener. Goal softeners can also interact with definitional issues in weird ways. The actions of SRAI is not sensitive to the definition of Suffering. All reasonable definitions lead to the same behaviour. But the actions of sSRAI might be very sensitive to the specific definition of Suffering used. Nothing discussed in this paragraph counts as analysing alignment targets. Simply because no topic discussed in this paragraph can ever be relevant to any question along the lines of: is SRAI a bad alignment target? So everything in this paragraph is out of scope of the proposed research agenda.
Being able to discuss the suitability of an alignment target, as a separate issue, is important. Because, if an AI project succeeds, and hits the alignment target that the project is aiming for, then it matters a lot, what alignment target the project was aiming for. It is important to be able to discuss this, as a separate question, removed from questions of strategies for buying time, or safety measures that might give a project a chance to try again, or measures that might lead some AI to settle for an approximation of the outcome implied by the alignment target, etc, etc. This is why we need the concepts, that all of these examples, are being used to define (the words used to refer to these concepts can of course be changed. But the concepts themselves must be possible to refer to).
Let’s say that two well intentioned designers, named Dave and Bill, both share your values. Dave is proposing an AI project aiming for SRAI, and Bill is proposing an AI project aiming for SMAI. Both projects have a set of very clever safety measures. Both Dave and Bill say that if some specific attempt to describe SRAI / SMAI fails, and leads to a bad outcome, then their safety measures will keep everyone safe, and they will find a better description (and they are known to be honest. And they are known to share your definition of what counts as a bad outcome. But it is of course not certain, that their safety measures will actually hold). If you can influence a decision, regarding which of these two project gets funded, then it seems very important to direct the funding away from Bill. Because the idea that clever safety measures, makes the alignment target irrelevant, is just straightforwardly false. Even if something along the lines of soft optimisation, is somehow guaranteed to work in a way that fully respects underlying intentions, then it would still matter, which of these two alignment targets are hit (even under the assumption of a non drastic path, to a soft version of the outcome, sSRAI is still preferable to sSMAI).
A very careful sSRAI / sSMAI project can obviously lead to an implemented SRAI / SMAI, that does not respect the underlying intentions of soft maximisation (for example because the resulting AI does not care at all about these intentions due to some technical failure. Or because it only cares about something, that turns out to have no significant impact on the outcome). The claim that this should be seen as a very surprising outcome, of any real world project, would be an extraordinary claim. It would require an extraordinarily solid argument. Even if such a solid-seeming argument were to be provided. The most likely scenario would still be that this argument is wrong in some way. The idea that an actual real world project plan is genuinely solid, in a way that makes a non soft implementation, of the aimed for alignment target, genuinely unlikely, does not seem plausible (even with a solid seeming argument). It seems a lot more likely that someone has constructed a solid seeming argument, on top of an unexamined implicit assumption. I don’t think that anyone, who takes AI dangers seriously, will dispute this fact. Conditioned on a project aiming for a bad alignment target, it is simply not possible for a careful person to rule out the scenario, where the project leads to the outcome, that is implied by the aimed for alignment target. Simply assuming that some safety measure will actually work, is just straightforwardly incompatible, with taking AI risks seriously.
It is worth pointing out, that this constitutes a fully separate argument, in favour of preferring a genuinely careful sSRAI project, to a genuinely careful sSMAI project (an argument that is fully distinct, from arguments based on preferring an outcome determined by sSRAI, to an outcome determined by sSMAI). If one acknowledges the value of switching from a genuinely careful sSMAI project, to a genuinely careful sSRAI project, then it is difficult to deny the necessity, of being able to analyse alignment targets.
From a higher level of abstraction, we can consider the scenario where two novel alignment targets are proposed. Neither one of them has any obvious flaws. This scenario takes place in a the real world, and involve real people as decision makers. Thus, these people deciding to wait indefinitely, is obviously not something that we can safely assume. In this scenario, it would be valuable, if an alignment target analysis effort, has advanced to the point where these two proposals can be meaningfully analysed and / or compared to each other (for example concluding that one is better than the other. Or concluding that both must be discarded, since they both lack at least one feature, that was found to be necessary for safety, while analysing other alignment targets). This remains valuable, even if a separate research effort has designed multiple layers of genuinely valuable safety measures.
Conditioned on Bill’s SMAI project being inevitable, safety measures might be very useful (they might hold up until Bill finally discovers that he is wrong. Until Bill finally sees, that SMAI is a bad alignment target. Until Bill finally realise, that the problems are inherent in the core concept. That the problems are not related to the difficulty, of finding a good definition of Suffering). But, regardless of what safety measures exists, Bill should not initiate an AI project that aims for the SMAI alignment target. This conclusion is completely independent of the specifics of the set of safety measures involved. In other words, it is important to separate the issue of safety measures, from the question of whether or not, a project is aiming for a bad alignment target. When analysing alignment targets, it makes sense to assume that this target will be successfully hit (in other words, it makes sense to assume that there will be nothing along the lines of molecular Squiggles, nothing along the lines of tiny pictures of human faces, nothing along the lines of soft optimisation, nothing along the lines of a triggered do-over-button, nothing along the lines of someone reconsidering the wisdom of aiming at this alignment target, etc, etc, etc, etc). Because if a project is launched, aiming at a bad alignment target, then no person who takes AI risks seriously, can dismiss the scenario, where the outcome implied by this alignment target, ends up getting fully implemented.
(when designing safety measures, it might make sense to imagine the mirror of this. In other words, you could imagine that you are designing these measures for an unpreventable SMAI project. A project lead by a very careful designer named Bill, that share your values, and that is aiming for SMAI, due to a sincere misunderstanding. And it might make sense to assume, that Bill will be very slow to update, regarding the suitability of SMAI as an alignment target. That Bill has severe tunnel vision, and has used his very powerful mind to almost completely insulate himself from object level critique, and closed down essentially every avenue, that might force him to admit that he is wrong. And assume that when Bill sees a bad outcome, that is prevented by some safety measure that you design, then this will be explained away by Bill, as being a result of a failure to describe the SMAI alignment target. One can for example imagine that Bill is the result of an augmentation process, that dramatically increased technical ability and persuasion ability. But also lead to extreme tunnel vision and very entrenched views, on questions of alignment targets. For example because persuasion ability and ability to get things done, is the result of a total and unquestioning adherence to a set of assumptions, that are, on the whole, far superior to any set of assumptions used by any baseline human (but still not fully free from flaws, in the particular case of alignment target analysis). This line of reasoning is entirely about safety measure design principles. In other words: nothing in this parenthesis counts as analysing alignment targets. Thus, everything in this parenthesis is out of scope of the proposed research agenda)
What links here?

ThomasCederborg Feb 23, 2024, 5:32 PM
2 points
0
in reply to: VojtaKovarik’s comment on: Extinction Risks from AI: Invisible to Science?
What about the term uncaring AI? In other words, an AI that would keep humans alive, if offered resources to do so. This can be contrasted with a Suffering Reducing AI (SRAI), which would not keep humans alive in exchange for resources. SRAI is an example of successfully hitting a bad alignment target, which is an importantly different class of dangers, compared to the dangers of an aiming failure leading to an uncaring AI. While an uncaring AI would happily agree to leave earth alone in exchange for resources, this is not the case for SRAI, because killing humans is inherent in the core concept, of reducing suffering. Any reasonable set of definitions, simply leads to a version of SRAI that rejects all such offers (assuming that the AI project that was aiming for the SRAI alignment target, manages to successfully hit this alignment target).
The term Uncaring AI is not meant to imply that the AI does not care about anything. Just that it does not care about anything that humans care about. Such as human lives. Which means that the question of extinction (and everything else that humans care about) is entirely determined by strategic considerations. The dangers stemming from the case where an aiming failure leads to an uncaring AI by accident, is importantly different, from the dangers stemming from a design team that successfully hits a bad alignment target. How about including a footnote, saying that you use Extinction as a shorthand for an outcome where humans are completely powerless, and where the fate of every living human is fully determined by an AI, that does not care about anything, that any human cares about? (and perhaps capitalise Extinction in the rest of the text) (and perhaps mention, in that same footnote, that if a neighbouring AI will pay such an uncaring AI to keep humans alive, then it would happily do so)
If an AI project succeeds, then it matters a lot, what alignment target the project was aiming for. Different bad alignment targets imply different behaviours. Consider a Suffering Maximising AI (SMAI). If a SMAI project successfully hits the SMAI alignment target, the resulting AI would create a lot of people. This is again importantly different from the case where the project fails to provide a reasonable definition of Suffering, and the resulting AI goes on to create little molecular pictures of sad faces or something similar. This molecular-sad-face-AI is, from a human perspective, basically the same as an AI that creates a lot of molecular squiggles, due to some other type of failure, that is unrelated to the definition of Suffering. They are both uncaring, and will both treat humans in a fully strategic manner. And both of these uncaring AIs, lead to the same outcome, as a failed SRAI project would lead to (whether through definitional issues, or something that leads to a lot of squiggles). They all treat everything that humans care about, in a fully strategic manner (they will all lead to Extinction, as defined in the proposed footnote mentioned above). But a successful SMAI project, would be importantly different from a successful SRAI project (which would actually lead to extinction, in the standard usage of that word. But would not lead to Extinction, as defined above). In the case of SMAI, different sets of reasonable definitions, also lead to importantly different outcomes (if the definition of suffering is reasonable, then the outcome of a successful SMAI project, will be a bad outcome. But different definitions still lead to importantly different outcomes). It is important to separate these dangers, from the dangers stemming from an uncaring AI. Because doing so allows us to explain, why it matters, which alignment target, an AI project is aiming for.
Let’s say that two well intentioned designers, named Dave and Bill, both share your values. Dave is proposing an AI project aiming for SRAI, and Bill is proposing an AI project aiming for SMAI. Both projects have a set of very clever safety measures. Both Dave and Bill say that if some specific attempt to describe SRAI / SMAI fails, and leads to a bad outcome, then their safety measures will keep everyone safe, and they will find a better description (and they are known to be honest. And are known to share your definition of what counts as a bad outcome. But it is of course not certain, that their safety measures will actually hold). If you can influence the decision, of which project gets funded, then it seems very important to direct the funding away from Bill. Because the idea that clever safety measures, makes the alignment target irrelevant, is just straightforwardly false. An actual project, started in the actual real world, that are fully dedicated to the idea of carefully trying things out with soft maximisation, might of course result in non-soft maximisation, of the alignment target being aimed at (and it is better if this happens to Dave’s proposed project, than to Bill’s proposed project). Conditioned on a SMAI project being inevitable, then safety measures might be very useful (they might hold up until Bill finally discovers that he is wrong. Until Bill finally sees, that SMAI is a bad alignment target). Using your example safety measure, we can phrase this as: Even if Bill sincerely intends to use iterative goal specification, it is still of course possible, that his proposed SMAI project, will end in a successful implementation, of the aimed for alignment target: SMAI. It is possible that Bill will successfully use iterative goal specification to avoid catastrophe. But this is not guaranteed. Thus, Bill should still not aim for the SMAI alignment target. This conclusion is completely independent of the specifics of the set of safety measures involved. In other words, it is important to separate the issue of safety measures, from the question of whether or not, a project is aiming for a bad alignment target.
In yet other words: it seems important to be able to separate dangers related to successfully hitting bad alignment targets, from dangers related to aiming failures. Because in the case of a project that successfully hits the alignment target that it is aiming for, it matters a lot, which alignment target the project was aiming for (in the case of SMAI, the details of the definitions are also influencing the outcome in an important way). We need to maintain the ability to, for example, differentiate between the type of extinction outcome that SRAI implies, from the type of Extinction outcome that you are discussing.
For more on the dangers involved with successfully hitting the wrong alignment target, see:
A problem with the most recently published version of CEV
(in the context of the linked philpapers paper, this is a word choice comment about the word: extinction. It is not a comment on the set of conditions that the paper identify as necessary for the purpose of evaluating arguments about these things. If taken extremely literally, your current word choices would imply, that you should add at least one more condition to your set: specifically the condition that the model must be able to talk about the strategic incentives that an uncaring AI would face (for example from a neighbouring AI, that an uncaring AI is expecting to possibly run into, in the distant future). Specifically, the additional condition is necessary for evaluating A2 and / or A3 (depending on how the text is interpreted). A model without this additional condition is useless for evaluating extinction arguments, in the same sense as a model with a static moon, is useless for evaluating arguments about rockets hitting the moon (they both fail on the first step). But I think that the above footnote is probably more in line with what you are trying to do with your paper. In other words: since you presumably do not want to add a strategic-environment-condition to your set of conditions, you will presumably prefer to add a footnote, and switch from extinction to Extinction (since this new condition would be necessary for evaluating arguments about extinction, but would not be necessary for evaluating arguments about Extinction))

ThomasCederborg Feb 14, 2024, 5:08 PM
1 point
0
in reply to: Vladimir_Nesov’s comment on: Managing risks while trying to do good
If your favoured alignment target suffers from a critical flaw, that is inherent in the core concept, then surely it must be useful for for you to discover this. So I assume that you agree that, conditioned on me being right about CEV suffering from such a flaw, you want me to tell you about this flaw. In other words, I think that I have demonstrated, that CEV suffers from a flaw, that is not related to any detail, of any specific version, or any specific description, or any specific proxy, or any specific attempt to describe what CEV is, or anything else along those lines. Instead, this flaw is inherent in the core concept, of building an AI that is describable as ``doing what a Group wants″. The Suffering Reducing AI (SRAI) alignment target is known to suffer from this type of a core flaw. The SRAI flaw is not related to any specific detail, of any specific version, or proxy, or attempt to describe what SRAI is, etc. And the flaw is not connected to any specific definition of ``Suffering″. Instead, the tendency to kill everyone, is inherent in the core concept of SRAI. It must surely be possible for you to update the probability that CEV also suffers from a critical flaw of this type (a flaw inherent in the core concept). SRAI sounds good on the surface, but it it is known to suffer from such a core flaw. Thus, the fact that CEV sounds good on the surface, does not rule out the existence of such a core flaw in CEV.
I do not think, that it possible to justify making no update, when discovering that the version of CEV, that you linked to, implies an outcome that would be far, far worse that extinction. I think that the probability must go up, that CEV contains a critical flaw, inherent in the core concept. Outcomes massively worse than extinction, is not an inherent feature, of any conceivable detailed description, of any conceivable alignment target. To take a trivial example, such an outcome is not implied by any given specific description of SRAI. The only way that you can motivate not updating, is if you already take the position, that any conceivable AI, that is describable as ``implementing the Coherent Extrapolated Volition of Humanity″, will lead to an outcome that is far, far, worse than extinction. If this is your position, then you can justify not updating. But I do not think that this is your position (if this were your position, then I don’t think that CEV would be your favoured alignment target).
And this is not filtered evidence, where I constructed a version of CEV and then showed problems in that version. It is the version that you link to, that would be far, far, worse than extinction. So, from your perspective, this is not filtered. Other designs that I have mentioned elsewhere, like USCEV, or the ``non stochastic version of PCEV″, are versions that other people have viewed as reasonable attempts to describe what CEV is. The fact that you would like AI projects to implement safety measures, that would (if they work as intended) protect against these types of dangers, is great. I strongly support that. I would not be particularly surprised if a technical insight in this type of work turns out to be completely critical. But this does not allow you to justify not updating on unfiltered data. You simply can not block off all conceivable paths, leading to a situation, where you conclude that CEV suffers from the same type of core flaw, that SRAI is known to suffer from.
If one were to accept the line of argument, that all information of this type can be safely dismissed, then this would have very strange consequences. If Steve is running a SRAI project, then he could use this line of argument, to dismiss any finding, that a specific version of SRAI, leads to everyone dying. If Steve has a great set of safety measures, but simply does not update, when presented with the information, that a given version of SRAI would kill everyone, then Steve can never reach the point where he says: ``I was wrong. SRAI is not a good alignment target. The issue is not due to any details, of any specific version, or any specific definition or suffering, or anything else along those lines. The issue is inherent in the core concept of building an AI, that is describable as a SRAI. Regardless of how great some set of safety measures looks to the design team, no one should initiate a SRAI project″. Surely, you do not want to accept a line of argument, that would have allowed Steve, to indefinitely avoid making such a statement, in the face of any conceivable new information about the outcomes of different SRAI variants.
The alternative to debating specific versions, is to make arguments on the level, of what one should expect based on the known properties of a given proposed alignment target. I have tried to do this and I will try again. For example, I wonder how you would answer the question: ``why would an AI, that does what an arbitrarily defined abstract entity wants that AI to do, be good for a human individual?″. One can discover that the Coherent Extrapolated Volition of Steve, would lead to the death of all of Steve’s cells (according to any reasonable set of definitions). One can similarly discover that the Coherent Extrapolated Volition of ``a Group″, is bad for the individuals in that group (according to any reasonable set of definitions). Neither statement suffers from any logical tension. For humans, this should in fact be the expected conclusion for any ``Group AI″, given that, (i): many humans certainly sound as if they will ask the AI to hurt other humans as much as possible, (ii): a human individual is very vulnerable, to a powerful AI that is trying to hurt her as much as possible, and (iii): in a ``Group AI″ no human individual can have any meaningful influence, in the initial dynamic, regarding the adoption of those preferences that refer to her (if the group is large). If you doubt the accuracy of one of these three points, then I would be happy to elaborate, on whichever one you find doubtful. None of this, has any connection, to any specific version, or proxy, or attempt to describe what CEV is, or anything else along those lines. It is all inherent in the core concept of CEV (and any other AI proposal, that is describable as ``building an AI that does what a group wants it to do″). If you want, we can restrict all further discussion to this form of argument.
If one has already taken the full implications of (i), (ii), and (iii) into account, then one does not have to make a huge additional update, when observing an unfiltered massively-worse-than-extinction type outcome. But this is only because, when one has taken the full implications of (i), (ii), and (iii) into account, then one has presumably already concluded, that CEV suffers from a critical, core, flaw.
I don’t understand your sentence: ``Variants of CEV all having catastrophic flaws is some sort of argument that there is no optimization target that’s worth optimizing for.″. The statement ``CEV is not a good alignment target″ does not imply the non existence of good alignment targets. Right? In other words: it looks to me like you are saying, that a rejection of CEV as an alignment target, is equivalent to a rejection of all conceivable alignment targets. To me, this sounds like nonsense, so I assume that this is not what you are saying. To take a trivial example: I don’t think that SRAI is a good alignment target. But surely a rejection of CEV does not imply a rejection of SRAI. Right? Just to be clear: I am definitely not postulating the non existence of good alignment targets. Discovering that ``the Coherent Extrapolated Volition of Steve implies the death of all his cells″, does not imply the non existence of alignment targets, where Steve’s cells survive. Similarly, discovering that ``the Coherent Extrapolated Volition of Humanity is bad for human individuals″, does not imply the non existence of alignment targets, that are good for human individuals. (I don’t think that good alignment targets are easy to find, or easy to describe, or easy to evaluate, etc. But that is a different issue)
I think it’s best that I avoid building a whole argument, based on a guess, regarding what you mean here. But I do want to say, that if you are using ``CEV″ as a shorthand for ``the Coherent Extrapolated Volition of a single designer″, then you have to be explicit about this if you want me to understand you. And similarly: if ``CEV″ is simply a label, that you assign to any reasonable answer, to the ``what alignment target should be aimed at?″ question (provisional or otherwise), then you have to be explicit about this if you want me to understand you. If that is the case then I would have to phrase my claim as: ``Under no reasonable set of definitions does the Coherent Extrapolated Volition of Humanity deserve the label ``CEV‴’. This only sounds odd due to the chosen label. There is no more logical tension in that statement, than there is logical tension in the statement: ``Under no reasonable set of definitions, does the Coherent Extrapolated Volition of Steve, result in any of Steve’s cells surviving″ (discovering this about Steve should not be very surprising. And discovering this about Steve does not imply the non existence of alignment targets where Steve’s cells survive).

PS:
I am aware of the fact that you (and Yudkowsky, and Bostrom, and a bunch of other people), can not be reasonably described as having any form of reckless attitude along the lines of: ``Conditioned on knowing how to hit alignment targets, the thing to do is to just instantly hit some alignment target that sounds good″. I hope that it is obvious, that I am aware of this. But I wanted to be explicit about this, just in case it is not obvious to everyone, that I am aware of this. Given the fact that there is one of those green leaf thingies next to my username, it is probably best to be explicit about this sort of thing.

ThomasCederborg Feb 9, 2024, 6:21 PM
1 point
0
in reply to: Vladimir_Nesov’s comment on: Managing risks while trying to do good
The version of CEV, that is described on the page that your CEV link leads to, is PCEV. The acronym PCEV was introduced by me. So this acronym does not appear on that page. But that’s PCEV that you link to. (in other words: the proposed design, that would lead to the LP outcome, can not be dismissed as some obscure version of CEV. It is the version that your own CEV link leads to. I am aware of the fact, that you are viewing PCEV as: ``a proxy for something else″ / ``a provisional attempt to describe what CEV is″. But this fact still seemed noteworthy)
On terminology: If you are in fact using ``CEV″ as a shorthand, for ``an AI that implements the CEV of a single human designer″, then I think that you should be explicit about this. After thinking about this, I have decided that without explicit confirmation that this is in fact your intended usage, I will proceed as if you are using CEV as a shorthand, for ``an AI that implements the Coherent Extrapolated Volition of Humanity″ (but I would be perfectly happy to switch terminology, if I get such confirmation). (another reading of your text, is that: ``CEV″ (or: ``something like CEV″) is simply a label that you attach, to any good answer, to the correct phrasing of the ``what alignment target should be aimed at?″ question. That might actually be a sort of useful shorthand. In that case I would, somewhat oddly, have to phrase my claim as: under no reasonable set of definitions, does the Coherent Extrapolated Volition of Humanity, deserve the label ``CEV″ / ``something like CEV″. Due to the chosen label(s), the statement looks odd. But there is no more logical tension in the above statement, than there is logical tension in the following statement: ``under no reasonable set of definitions, does the Coherent Extrapolated Volition of Steve, result in the survival of any of Steve’s cells″ (which is presumably a true statement for at least some human individuals). Until I hear otherwise, I will however stay with the terminology, where ``CEV″ is shorthand for ``an AI that implements the Coherent Extrapolated Volition of Humanity″, or ``an AI that is helping humanity″, or something less precise, that is still hinting at something along those lines)
It probably makes sense to clarify my own terminology some more. I think this can be done by noting, that I think that CEV, sounds like a perfectly reasonable way of helping ``a Group″ (including the PCEV version that you link to, and that implies the LP outcome). I just don’t think that helping ``a Group″ (that is made up of human individuals) is good for the (human) individuals that make up that ``Group″ (in expectation). Pointing a specific version of CEV (including PCEV) at a set of individuals, might be great for some other type of individuals. Let’s consider a large number of ``insatiable, Clippy like maximisers″. Each of them cares exclusively about the creation of a different, specific, complex object. No instances of any of these very complex objects will ever exist, unless someone looks at the exact specification of a given individual, and uses this specification to create such objects. In this case PCEV might, from the perspective of each of those individuals, be the best thing that can happen (if special influence is off the table). It is also worth noting, that a given human individual might get what she wants, if some specific version of CEV is implemented. But CEV, or ``helping humanity″, is not good, for human individuals, in exception, compared to extinction. And why would it be? Groups and human individuals are completely different types of things. And a human individual is very vulnerable to a powerful AI, that wants to hurt her. And humanity certainly looks like it contains an awful lot of ``will to hurt″, specifically directed at existing human individuals.
If I zoom out a bit, I would describe the project of ``trying to describe what CEV is″ / ``trying to build an AI that helps humanity″ as: A project that searches for an AI design that helps an arbitrarily defined abstract entity. But this same project is, in practice, evaluating specific proposed AI designs, based on how they interact with a completely different type of thing: human individuals. You are for example presumably discarding PCEV, because the LP outcome implied by PCEV, contains a lot of suffering individuals (when PCEV is pointed at billions of humans). It is however not obvious to me why LP would be a bad way of helping an arbitrarily defined abstract entity (especially considering that the negotiation rules of PCEV simultaneously (i): implies LP, and is also (ii): an important part of the set of definitions, that is needed to differentiate the specific abstract entity that is to be helped, from the rest of the vast space of entities, that a mapping from billions-of-humans to the ``class-of-entities-that-can-be-said-to-want-things″, can point to). Thus, I suspect that PCEV is not actually being discarded, due to being bad at helping an abstract entity (and my guess it that PCEV is actually being discarded, because LP is bad for human individuals).
I think that one reasonable way of moving past this situation, is to switch perspective. Specifically: adopt the perspective of a single human individual, in a population of billions, and ask: ``without giving her any special treatment, compared to other existing humans, what type of AI, would want to help her″. And then try to answer this question, while making as few assumptions about her as possible (for example making sure that there is no implicit assumption, regarding whether she is ``selfish or selfless″, or anything along those lines. Both ``selfless and selfish″ human individuals, would strongly prefer to avoid being a Heretic in LP. Thus, discarding PCEV does not contain an implicit assumption related to the ``selfish or selfless″ issue. Discarding PCEV, does however, involve an assumption, that human individuals are not like the ``insatiable Clippy maximisers″ mentioned above. So, such maximisers might justifiably feel ignored, when we discard PCEV. But no one can justifiably feel ignored when we discard PCEV, on account of where she is on the ``selfish or selfless″ spectrum). When one adopts this perspective, it becomes obvious to suggest that, the initial dynamic, should grant this individual meaningful influence, regarding the adoption of those preferences, that refer to her. Making sure that such influence, is included as a core aspect of the initial dynamic, is made even more important, by the fact, that the designers will be unable to consider all implications of a given project, and will be forced to rely on, potentially flawed, safety measures (for example along the lines of a ``Last Judge″ off switch, which might fail to trigger. Combined with a learned DWIKIM layer, that might turn out to be very literal, when interpreting some specific class of statements). If such influence is included, in the initial dynamic, then the resulting AI is no longer describable as ``doing what a Group wants it to do″. Thus, the resulting AI can not be described as a version of CEV. (it might however be describable as ``something like CEV″. Sort of how one can describe an Orca as ``something like a shark″, despite the fact that an Orca is not a type of shark (or a type of a fish). I would guess, that you would say, that an AI that grants such influence, as part of the initial dynamic, is not ``something like CEV″. But I’m not sure about this)
(I should have added ``,in the initial dynamic,″ to the text in my earlier comments. It is explicit in the description of MPCEV, but I should have added this phrase to my comments here too. As a tangent, I agree that the intuition, that you were trying to counter, with your Boundaries / Membrane mention, is probably both common and importantly wrong. Countering this intuition makes sense, and I should have read this part of your comment more carefully. I would however like to note, that the description of the LP outcome, in the PCEV thought experiment, actually contains billions of (presumably very different) localities. Each locality is optimised according to very different criteria. Each place is designed to hurt a specific individual human Heretic. And each such location, is additionally bound by it’s own unique ``comprehension constraint″, that refers to the specific individual Heretic, being punished in that specific location)
Perhaps a more straightforward way to move this discussion along is to ask a direct question, regarding what you would do if you were in the position, that I believe, that I find myself in. In other words: a well intentioned designer called John, wants to use PCEV as the alignment target for his project (rejecting any other version of CEV out of hand, by saying: ``if that is indeed a good idea, then it will be the outcome of Parliamentary Negotiations″). When someone points out that PCEV is a bad alignment target, John responds by saying that PCEV cannot, by definition, be a bad alignment target. John claims that any thought experiment, where PCEV leads to a bad outcome, must be due to a bad extrapolation of human individuals. John says that any given ``PCEV with a specific extrapolation procedure″ is just an attempt, to describe what PCEV is. If aiming at a given ``PCEV with a specific extrapolation procedure″ is a bad idea, then it is a badly constructed PCEV. Aiming at PCEV is a good idea, by intention that defines PCEV. John further says that his project will include features that (if they are implemented successfully, and are not built on top of any problematic unexamined implicit assumption) will to let John try again, if a given attempt to ``say what PCEV is″, fails. Do you agree that this project, is a bad idea? (compared to achievable alternatives, that start with a different set of, findable, assumptions) If so, what would you say to John? (what you are proposing is different from what John is proposing. I predict that you will say that John is making a mistake. My point is that, to me, it looks like you are making a mistake, of the same type as John’s mistake. So, I wonder what you would say to John (your behaviour in this exchange, is not the same as John’s behaviour in this thought experiment. But it looks to me, like you are making the same class of mistake, as John. So, I’m not asking how you would ``act in a debate, as a response to Johns behaviour″. Instead, I’m curious about how you would explain to John, that he is making an object level mistake))
Or maybe a better approach, is to go less meta, and get into some technical details. So, let’s use the terminology in your CEV link, to explore some of the technical details in that post. What do you think would happen, if the learning algorithm that outputs the DWIKIM layer in John’s PCEV project, is built on top of an unexamined implicit assumption, that turns out to be wrong? Let’s say that the DWIKIM layer that pops out, interprets the request to build PCEV, as a request to implement the straightforward interpretation of PCEV. The DWIKIM layer happens to be very literal, when presented with the specific phrasing, used in the request. In other words: it interprets John as requesting, something along the lines of LP? I think this might result in an outcome, along the lines of LP (if the problems with the DWIKIM layer, stems form a problematic unexamined implicit assumption, related to extrapolation, then the exact same problematic assumption, might also render something along the lines of a ``Last Judge off switch add on″, ineffective). I think that it would be better, if John had aimed at something, that does not suffer from known, avoidable, s-risks. Something whose straightforward interpretation, is not known to imply an outcome, that would be far, far, worse than extinction. For the same reason, I make the further claim, that I do not think that it is a good idea, to subject everyone to the known, avoidable, s-risks associated with any AI, that is describable as ``doing what a Group wants″ (which includes all versions of CEV). Again, I’m certainly not against some feature that, might, let you try again, or that, might, re interpret an unsafe request, as a request for something completely different, that happens to be safe (such as, for example, a learned DWIKIM layer). I am aware of the fact, that you do not have absolute faith in the DWIKIM layer (if this layer is perfectly safe, in the sense of reliably re interpreting requests that straightforwardly imply LP, as something desirable to the designer. Then the full architecture would be functionally identical, to an AI, that simply does, whatever the designer wants the AI to do. In that case, you would not care what the request was. You might then, just as well have the designer ask the DWIKIM layer, for an AI, that maximises the number of bilberries. So, I am definitely not implying, that you are unaware, of the fact that the DWIKIM layer, is unable to provide reliable safety).
Zooming out a bit, it is worth noting that the details of the safety measure(s) is actually not very relevant to the points that I am trying to make here. Any conceivable, human implemented, safety measure, might fail. And, more importantly, these measures do not help much, when one is deciding what to aim at. For example: MPCEV, can also be built on top of a (potentially flawed) DWIKIM layer, in the exacts same way as you can build CEV on top of a DWIKIM layer (and you can stick a ``Last Judge off switch add on″ to MPCEV too. Etc, etc, etc). Or in yet other words: anything, along the lines of, a ``Last Judge off switch add on″ can be used by many different projects aiming at many different targets. Thus, the ``Last Judge″ idea, or any other idea along those lines (including a DWIKIM layer), provides very limited help, when one decides what to aim at. And even more generally: regardless of what safety measure is used, John is, still, subjecting everyone to an unnecessary, avoidable, s-risk. I hope we can agree that John should not do that with, any, version of ``PCEV with a specific extrapolation procedure″. The further claim, that I am making, is that no one should do that with, any, ``Group AI″, for similar reasons. Surely, discovering that this further claim is true, cannot be, by definition, impossible.
While re reading our exchange, I realised that I never actually clarified, that my primary reason for participating in this exchange (and my primary reason for publishing things on LW), is not actually to stop CEV projects. However, I think that a reasonable person might, based on my comments here, come to believe that my primary goal is to stop CEV projects (which is why the present clarification is needed). My focus is actually on trying to make progress on the ``what alignment target should be aimed at?″ question. In the present exchange, my target is the idea, that this question has already been given an answer (and, specifically, that the answer is CEV). The first step to progress, on the ``what alignment target should be aimed at?″ question, is to show that this question does not currently have an answer. This is importantly different, from saying that: ``CEV is the answer, but the details are unknown″ (I think that such a statement is importantly wrong. And I also think, that the fact that people still believe things along these lines, is standing in the way of getting a project off the ground, that is devoted to making progress on the ``what alignment target should be aimed at?″ question).
I think that it is very unlikely, that the relevant people will stay committed to CEV, until the technology arrives, that would make it possible for them to hit CEV as an alignment target (the reason I find this unlikely, is that, (i): I believe that I have outlined a sufficient argument, to show that CEV is a bad idea, and (ii): I think that such technology will take time to arrive, and (iii): it seems likely that this team of designers, who are by assumption capable of hitting CEV, will be both careful enough to read that argument before reaching the point of no return on their CEV launch, and also capable enough to understand it. Thus, since the argument against CEV already exists, in my estimate, it would not make sense to focus on s-risks, related to a successfully implemented CEV). If that unlikely day ever does arrive, then I might switch focus, to trying to prevent direct CEV related s-risk, by arguing against this imminent CEV project. But I don’t expect to ever see this happening.
The set of paths that I am actually focused on reducing the probability of, can be hinted at by outlining the following specific scenario. Imagine a well intentioned designer that we can call Dave, who is aiming for Currently Unknown Alignment Target X (CUATX). Due to an unexamined implicit assumption, that CUATX is built on top of, turning out to be wrong in a critical way, CUATX implies an outcome, along the lines of LP. But the issue that CUATX suffers from, is far more subtle than the issue that CEV suffers from. And progress on the ``what alignment target should be aimed at?″ question, has not yet progressed to the point, where this problematic unexamined implicit assumption can be seen. CUATX has all the features, that are known at launch time, to be necessary for safety (such as the necessary, but very much not sufficient, feature that any safe AI must give each individual, meaningful influence, regarding the adoption of those preferences, that refer to her). Thus, the CUATX idea leads to a CUATX project, which in turn leads to an, avoidable, outcome along the lines of LP (after some set of human implemented safety measures fail). That is the type of scenario that I am trying to avoid (by trying to make sufficient progress on the ``what alignment target should be aimed at?″ question, in time). My real ``opponent in this debate″ is an implemented CUATX, not the idea of CEV (and very definitely not you. Or anyone else that has contributed, or is likely to contribute, valuable insights related to the ``what alignment target should be aimed at?″ question). It just happens to be the case, that the effort to prevent CUATX, that I am trying to get off the ground, starts by showing that CEV, is not an answer, to the ``what alignment target should be aimed at?″ question. And you just happen to be the only person, that is pushing back against this in public (and again: I really appreciate the fact that you chose to engage on this topic).
(I should also note explicitly, that I am most definitely not against exploring safety measures. They might stop CUATX. In some plausible scenarios, they might be the only realistic thing, that can stop CUATX. And I am not against treaties. And I am open to hearing more about the various human augmentation proposals that have been going around for many years. I am simply noting, that a safety measure, regardless of how clever it sounds, simply cannot fill the function of a substitute, for progress on the ``what alignment target should be aimed at?″ question. An attempt to get people to agree to a treaty might fail. Or a successfully implemented treaty might fail to actually prevent a race dynamic for long enough. And similarly, augmented humans might systematically tend towards being: (i): superior at alignment, (ii): superior at persuasion, (iii): well intentioned, and (iv): not better at dealing with the ``what alignment target should be aimed at?″ question, than the best baseline humans (but still, presumably, capable of understanding an insight on this question, at least if that insight is well explained). Regardless of augmentation technique, selection for ``technical ability and persuasion ability″ seems like a far more likely, de facto, outcome to me, due to being far easier to measure. I expect it to be far more difficult to measure the ability to deal with the ``what alignment target should be aimed at?″ question (and it is not obvious that the abilities needed to deal with the ``what alignment target should be aimed at?″ question, will be strongly correlated with the thing that I think will, de facto, have driven the trial and error augmentation process, of the augments that eventually hits an alignment target: ``technical-ability-and-persuasion-ability-and-ability-to-get-things-done″). Maybe the first augment will be great at making progress on the ``what alignment target should be aimed at?″ question, and will quickly render all previous work on this question irrelevant (and in that case, the persuasion ability is probably good for safety). But assuming that this will happen, seems like a very unsafe bet to make. Even more generally: I simply do not think that it is possible to come up with any type of clever sounding trick, that makes it safe to skip the ``what alignment target should be aimed at?″ question (to me, the ``revolution-analogy-argument″, in the 2004 CEV text, looks like a sufficient argument for the conclusion, that it is important to make progress on the ``what alignment target should be aimed at?″ question. But it seems like many people do not consider this, to be a sufficient argument for this conclusion. It is unclear to me, why this conclusion, seems to require such extensive further argument)).
If my overall strategic goal was not clear, then this was probably my fault (in addition to not making this goal explicit, I also seem to have a tendency to lose focus on this larger strategic picture, during back and fourth technical exchanges).
Two out of of my three LW posts are in fact entirely devoted to arguing, that making progress on the ``what alignment target should be aimed at?″ question, is urgent (in our present discussion, we have only talked about the one post, that is not exclusively focused on this). See:
Making progress on the ``what alignment target should be aimed at?″ question, is urgent
The proposal to add a ``Last Judge″ to an AI, does not remove the urgency, of making progress on the ``what alignment target should be aimed at?″ question.
(I am still very confused about this entire conversation. But I don’t think that re reading everything, yet again, will help much. I have been continually paying, at least some, attention to SL4, OB, and LW since around 2002-2003. I can’t remember exactly who said what when, or where. However, I have developed a strong intuition, that can be very roughly translated as: ``if something sounds strange, then it is very definitely not safe, to explain away this strangeness, by conveniently assuming that Nesov is confused on the object-level″. I am nowhere near the point where I would consider going against this intuition. So, I expect that I will remain very confused about this exchange, until there is some more information available. I don’t expect to be able to just think my way out of this one (wild speculation, regarding what it might be, that I was missing, by anyone that happens to stumble on this comment, at any point in the future, are very welcome. For example in a LW comment, or in a LW DM, or in an email))

ThomasCederborg Feb 3, 2024, 8:23 PM
1 point
0
in reply to: the gears to ascension’s comment on: Managing risks while trying to do good
I was clearly wrong regarding how you feel about your cells. But surely the question of whether or not an AI that is implementing the CEV of Steve, would result in any surviving cells, is an empirical question? (which must settled by referring to facts about Steve. And trying to figure out what these facts mean in terms of how the CEV of Steve would treat his cells). It cannot possibly be the case that it is impossible, by definition, to discover that any reasonable way of extrapolating Steve would result in all his cells dying?
Thank you for engaging on this. Reading your description of how you view your own cells was a very informative window, into how a human mind can work. (I find it entirely possible, that I am very wrong regarding how most people view their cells. Or about how they would view their cells upon reflection. I will probably not try to introspect, regarding how I feel about my own cells, while this exchange is still fresh)
Zooming out a bit, and looking at this entire conversation, I notice that I am very confused. I will try to take a step back from LW and gain some perspective, before I return to this debate.

ThomasCederborg

The case for more Align­ment Tar­get Anal­y­sis (ATA)

A nec­es­sary Mem­brane for­mal­ism feature

Cor­rigi­bil­ity could make things worse

The case for more Alignment Target Analysis (ATA)

A necessary Membrane formalism feature

Corrigibility could make things worse