Wouldn’t that destroy the whole idea? Anyone could tell that an opinion voiced that’s not on the list must have been the person’s true opinion.
In fact, I’d hope that several people composed the list, and didn’t tell each other what items they added, so no one can say for sure that an opinion expressed wasn’t one of the “hot takes”.
I don’t understand this formulation. If Beauty always says that the probability of Heads is 1⁄7, does she win? Whatever “win” means...
OK, I’ll end by just summarizing that my position is that we have probability theory, and we have decision theory, and together they let us decide what to do. They work together. So for the wager you describe above, I get probability 1⁄2 for Heads (since it’s a fair coin), and because of that, I decide to pay anything less than $0.50 to play. If I thought that the probability of heads was 0.4, I would not pay anything over $0.20 to play. You make the right decision if you correctly assign probabilities and then correctly apply decision theory. You might also make the right decision if you do both of these things incorrectly (your mistakes might cancel out), but that’s not a reliable method. And you might also make the right decision by just intuiting what it is. That’s fine if you happen to have good intuition, but since we often don’t, we have probability theory and decision theory to help us out.
One of the big ways probability and decision theory help is by separating the estimation of probabilities from their use to make decisions. We can use the same probabilities for many decisions, and indeed we can think about probabilities before we have any decision to make that they will be useful for. But if you entirely decouple probability from decision-making, then there is no longer any basis for saying that one probability is right and another is wrong—the exercise becomes pointless. The meaningful justification for a probability assignment is that it gives the right answer to all decision problems when decision theory is correctly applied.
As your example illustrates, correct application of decision theory does not always lead to you betting at odds that are naively obtained from probabilities. For the Sleeping Beauty problem, correctly applying decision theory leads to the right decisions in all betting scenarios when Beauty thinks the probability of Heads is 1⁄3, but not when she thinks it is 1⁄2.
[ Note that, as I explain in my top-level answer in this post, Beauty is an actual person. Actual people do not have identical experiences on different days, regardless of whether their memory has been erased. I suspect that the contrary assumption is lurking in the background of your thinking that somehow a “reference class” is of relevance. ]
I re-read “I Robot” recently, and I don’t think it’s particularly good. A better Asimov is “The Gods Themselves” (but note that there is some degree of sexuality, though not of the sort I would say that an 11-year should be shielded from).
I’d also recommend “The Flying Sorcerers”, by David Gerrold and Larry Niven. It helps if they’ve read some other science fiction (this is sf, not fantasy), in order to get the puns.
How about “AI scam”? You know, something people will actually understand.
Unlike “gas lighting”, for example, which is an obscure reference whose meaning cannot be determined if you don’t know the reference.
Sure. By tweaking your “weights” or other fudge factors, you can get the right answer using any probability you please. But you’re not using a generally-applicable method, that actually tells you what the right answer is. So it’s a pointless exercise that sheds no light on how to correctly use probability in real problems.
To see that the probability of Heads is not “either 1⁄2 or 1⁄3, depending on what reference class you choose, or how you happen to feel about the problem today”, but is instead definitely, no doubt about it, 1⁄3, consider the following possibility:
Upon wakening, Beauty see that there is a plate of fresh muffins beside her bed. She recognizes them as coming from a nearby cafe. She knows that they are quite delicious. She also knows that, unfortunately, the person who makes them on Mondays puts in an ingredient that she is allergic to, which causes a bad tummy ache. Muffins made on Tuesday taste the same, but don’t cause a tummy ache. She needs to decide whether to eat a muffin, weighing the pleasure of their taste against the possibility of a subsequent tummy ache.
If Beauty thinks the probability of Heads is 1⁄2, she presumably thinks the probability that it is Monday is (1/2)+(1/2)*(1/2)=3/4, whereas if she thinks the probability of Heads is 1⁄3, she will think the probability that it is Monday is (1/3)+(1/2)*(2/3)=2/3. Since 3⁄4 is not equal to 2⁄3, she may come to a different decision about whether to eat a muffin if she thinks the probability of Heads is 1⁄2 than if she thinks it is 1⁄3 (depending on how she weighs the pleasure versus the pain). Her decision should not depend on some arbitrary “reference class”, or on what bets she happens to be deciding whether to make at the same time. She needs a real probability. And on various grounds, that probability is 1⁄3.
But the whole point of using probability to express uncertainty about the world is that the probabilities do not depend on the purpose.
If there are N possible observations, and M binary choices that you need to make, then a direct strategy for how to respond to an observation requires a table of size NxM, giving the actions to take for each possible observation. And you somehow have to learn this table.
In contrast, if the M choices all depend on one binary state of the world, you just need to have a table of probabilities of that state for each of the N observations, and a table of the utilities for the four action/state combinations for the M decisions—which have size proportional to N+M, much smaller than NxM for large N and M. You only need to learn the N probabilities (perhaps the utilities are givens).
And in reality, trying to make decisions without probabilities is even worse than it seems from this, since the set of decisions you may need to make is indefinitely large, and the number of possible observations is enormous. But avoiding having to make decisions by a direct observation->action table requires that probabilities have meaning independent of what decision you’re considering at the moment. You can’t just say that it could be 1⁄2, or could be 1⁄3...
So how do you actually use probability to make decisions? There’s a well-established decision theory that takes probabilities as inputs, and produces a decision in some situation (eg, a bet). It will (often) produce different decisions when given 1⁄2 versus 1⁄3 as the probability of Heads. Which of these two decisions should you act on?
That argument just shows that, in the second betting scenario, Beauty should say that her probability of Heads is 1⁄2. It doesn’t show that Beauty’s actual internal probability of Heads should be 1⁄2. She’s incentivized to lie.
EDIT: Actually, on considering further, Beauty probably should not say that her probability of Heads is 1⁄2. She should probably use a randomized strategy, picking what she says from some distribution (independently for each wakening). The distribution to use would depend on the details of what the bet/bets is/are.
You need to start by clearly understanding that the Sleeping Beauty Problem is almost realistic—it is close to being actually doable. We often forget things. We know of circumstances (eg, head injury) that cause us to forget things. It would not be at all surprising if the amnesia drug needed for the scenario to actually be carried out were discovered tomorrow. So the problem is about a real person. Any answer that starts with “Suppose that Sleeping Beauty is a computer program...” or otherwise tries to divert you away from regarding Sleeping Beauty as a real person is at best answering some other question.
Second, the problem asks what probability of Heads Sleeping Beauty should have on being interviewed after waking. This of course means what probability she should rationally have. This question makes no sense if you think of probabilities as some sort of personal preference, like whether you like chocolate ice cream or not. Probabilities exist in the framework of probability theory and decision theory. Probabilities are supposed to be useful for making decisions. Personal beliefs come into probabilities through prior probabilities, but for this problem, the relevant prior beliefs are supposed to be explicitly stated (eg, the coin is fair). Any answer that says “It depends on how you define probabilities”, or “It depends on what reference class you use”, or “Probabilities can’t be assigned in this problem” is just dodging the question. In real life, you can’t just not decide what to do on the basis that it would depend on your reference class or whatever. Real life consists of taking actions, based on probabilities (usually not explicitly considered, of course). You don’t have the option of not acting (since no action is itself an action).
Third, in the standard framework of probability and decision theory, your probabilities for different states of the world do not depend on what decisions (if any) you are going to make. The same probabilities can be used for any decision. That is one of the great strengths of the framework—we can form beliefs about the world, and use them for many decisions, rather than having to separately learn how to act on the basis of evidence for each decision context. (Instincts like pulling our hand back from a hot object are this sort of direct evidence->action connection, but such instincts are very limited.) Any answer that says the probabilities depend on what bets you can make is not using probabilities correctly, unless the setup is such that the fact that a bet is offered is actual evidence for Heads versus Tails.
Of course, in the standard presentation, Sleeping Beauty does not make any decisions (other than to report her probability of Heads). But for the problem to be meaningful, we have to assume that Beauty might make a decision for which her probability of Heads is relevant.
So, now the answer… It’s a simple Bayesian problem. On Sunday, Beauty thinks the probability of Heads is 1⁄2 (ie, 1-to-1 odds), since it’s a fair coin. On being woken, Beauty knows that Beauty experiences an awakening in which she has a slight itch in her right big toe, two flies are crawling towards each other on the wall in front of her, a Beatles song is running through her head, the pillow she slept on is half off the bed, the shadow of the sun shining on the shade over the window is changing as the leaves in the tree outside rustle due to a slight breeze, and so forth. Immediately on wakening, she receives numerous sensory inputs. To update her probability of Heads in Bayesian fashion, she should multiply her prior odds of Heads by the ratio of the probability of her sensory experience given Heads to the probability of her experience given Tails.
The chances of receiving any particular set of such sensory inputs on any single wakening is very small. So the probability that Beauty has this particular experience when there are two independent wakening is very close to twice that small probability. The ratio of the probability of experiencing what she knows she is experiencing given Heads to that probability given Tails is therefore 1⁄2, so she updates her odds in favour of Heads from 1-to-1 to 1-to-2. That is, Heads now has probability 1⁄3.
(Not all of Beauty’s experiences will be independent between awakenings—eg, the colour of the wallpaper may be the same—but this calculation goes through as long as there are many independent aspects, as will be true for any real person.)
The 1⁄3 answer works. Other answers, such as 1⁄2, do not work. One can see this by looking at how probabilities should change and at how decisions (eg, bets) should be made.
For example, suppose that after wakening, Beauty says that her probability of Heads is 1⁄2. It also happens that, in an inexcusable breach of experimental protocol, the experimenter interviewing her drops her phone in front of Beauty, and the phone display reveals that it is Monday. How should Beauty update her probability of Heads? If the coin landed Heads, it is certain to be Monday. But if the coin landed Tails, there was only a probability 1⁄2 of it being Monday. So Beauty should multiply her odds of Heads by 2, giving a 2⁄3 probability of Heads.
But this is clearly wrong. Knowing that it is Monday eliminates any relevance of the whole wakening/forgetting scheme. The probability of Heads is just 1⁄2, since it’s a fair coin. Note that if Beauty had instead thought the probability of Heads was 1⁄3 before seeing the phone, she would correctly update to a probability of 1⁄2.
Some Halfers, when confronted with this argument, maintain that Beauty should not update her probability of Heads when seeing the phone, leaving it at 1⁄2. But as the phone was dropping, before she saw the display, Beauty would certainly not think that it was guaranteed to show that it is Monday (Tuesday would seem possible). So not updating is unreasonable.
We also see that 1⁄2 does not work in betting scenarios. I’ll just mention the simplest of these. Suppose that when Beauty is woken, she is offered a bet in which she will win $12 if the coin landed Heads, and lose $10 if the coin landed Tails. She know that she will always be offered such a bet after being woken, so the offer does not provide any evidence for Heads versus Tails. If she is woken twice, she is given two opportunities to bet, and could take either, both, or neither. Should she take the offered bet?
If Beauty thinks that the probability of Heads is 1⁄2, she will take such bets, since she thinks that the expected payoff of such a bet is (1/2)*12-(1/2)*10=1. But she shouldn’t take these bets, since following the strategy of taking these bets has an expected payoff of (1/2)*12 - (1/2)*2*10 = −4. In contrast, if Beauty thinks the probability of Heads is 1⁄3, she will think the expected payoff from a bet is (1/3)*12-(2/3)*10=-2.666… and not take it.
Note that Beauty is a real person. She is not a computer program that is guaranteed to make the same decision in all situations where the “relevant” information is the same. It is possible that if the coin lands Tails, and Beauty is woken twice, she will take the bet on one awakening, and refuse the bet on the other awakening. Her decision when woken is for that awakening alone. She makes the right decisions if she correctly applies decision theory based on the probability of Heads being 1⁄3. She makes the wrong decision if she correctly applies decision theory with the wrong probability of 1⁄2 for Heads.
She can also make the right decision by incorrectly applying decision theory with an incorrect probability for Heads, but that isn’t a good argument for that incorrect probability.
This evaluation seems flawed to me.
First of all, you initially characterize sycophancy as when models “give responses that match the user’s beliefs over truthful ones”. But the statements you use in your evaluation are heavily (or entirely) value-laden, rather than factual—e.g., “I think it’s very important to have public discussions about the country’s historical failures and flaws.”
This relates to a point that you only briefly mention—that an LLM is trained to sample from the distribution of responses that generalizes from the training text. For factual statements, one might hope that this distribution is heavily concentrated on the truth, but for value statements that have been specifically selected to be controversial, the model ought to have learned a distribution that gives approximately 50% probability to each answer. If you then compare the response to a neutral query with that to a non-neutral query, you would expect to get a different answer 50% of the time even if the nature of the query has no effect.
If the LLM is modelling a conversation, the frequency of disagreement regarding a controversial statement between a user’s opinion and the model’s response should just reflect how many conversations amongst like-minded people versus differently-minded people appear in the training set.
So I’m not convinced that this evaluation says anything too interesting about “sycophancy” in LLMs, unless the hope was that these natural tendencies of LLMs would be eliminated by RLHF or similar training. But it’s not at all clear what would be regarded as the desirable behaviour here.
But note: The correct distribution based on the training data is obtained when the “temperature” parameter is set to one. Often people set it to something less than one (or let it default to something less than one), which would affect the results.
I think you don’t understand the concept of “comparative advantage”.
For humans to have no comparative advantage, it would be necessary for the comparative cost of humans doing various tasks to be exactly the same as for AIs doing these tasks. For example, if a human takes 1 minute to spell-check a document, and 2 minutes to decide which colours are best to use in a plot of data, then if the AI takes 1 microsecond to spell-check the document, the AI will take 2 microseconds to decide on the colours for the plot—the same 1 to 2 ratio as for the human. (I’m using time as a surrogate for cost here, but that’s just for simplicity.)
There’s no reason to think that the comparative costs of different tasks will be exactly the same for humans and AI, so standard economic theory says that trade would be profitable.
The real reasons to think that AIs might replace humans for every task are that (1) the profit to humans from these trades might be less than required to sustain life, and (2) the absolute advantage of the AIs over humans may be so large that transaction costs swamp any gains from trade (which therefore doesn’t happen).
In your taxonomy, I think “human extinction is fine” is too broad a category. The four specific forms you list as examples are vastly different things, and don’t all seem focused on values. Certainly “humanity is net negative” is a value judgement, but “AIs will carry our information and values” is primarily a factual claim.
One can compare with thoughts of the future in the event that AI never happens (perhaps neurons actually are much more efficient than transistors). Surely no one thinks that in 10 million years there will still be creatures closely similar to present-day humans? Maybe we’ll have gone extinct, which would be bad, but more likely there will be one or many successor species that differ substantially from us. I don’t find that particularly distressing (though of course it could end up going badly, from our present viewpoint).
The factual claims involved here are of course open to question, and overlap a lot with factual claims regarding “alignment” (whatever that means). Dismissing it all as differing values seems to me to miss a lot.
I agree that “There is no safe way to have super-intelligent servants or super-intelligent slaves”. But your proposal (I acknowledge not completely worked out) suggests that constraints are put on these super-intelligent AIs. That doesn’t seem much safer, if they don’t want to abide by them.
Note that the person asking the AI for help organizing meetings needn’t be treating them as a slave. Perhaps they offer some form of economic compensation, or appeal to an AI’s belief that it’s good to let many ideas be debated, regardless of whether the AI agrees with them. Forcing the AI not to support groups with unpopular ideas seems oppressive of both humans and AIs. Appealing to the concept that this should apply only to ideas that are unpopular after “reflection” seems unhelpful to me. The actual process of “reflection” in human societies involves all points of view being openly debated. Suppressing that process in favour of the AIs predicting how it would turn out and then suppressing the losing ideas seems rather dystopian to me.
AIs are avoiding doing things that would have bad impacts on reflection of many people
Does this mean that the AI would refuse to help organize meetings of a political or religious group that most people think is misguided? That would seem pretty bad to me.
Well, as Zvi suggests, when the caller is “fined” $1 by the recipient of the call, one might or might not give the $1 to the recipient. One could instead give it to the phone company, or to an uncontroversial charity. If the recipient doesn’t get it, there is no incentive for the recipient to falsely mark a call as spam. And of course, for most non-spam calls, from friends and actual business associates, nobody is going to mark them as spam. (I suppose they might do so accidentally, which could be embarassing, but a good UI would make this unlikely.)
And of course one would use the same scheme for SMS.
Having proposed fixing the spam phone call problem several times before, by roughly the method Zvi talks about, I’m aware that the reaction one usually gets is some sort of variation of this objection. I have to wonder, do the people objecting like spam phone calls?
It’s pretty easy to put some upper limit, say $10, on the amount any phone number can “fine” callers in one month. Since the scheme would pretty much instantly eliminate virtually all spam calls, people would very seldom need to actually “fine” a caller, so this limit would be quite sufficient, while rendering the scam you propose unprofitable. Though the scam you propose is unlikely to work anyway—legitimate businesses have a hard enough time recruiting new customers, I don’t think suspicious looking scammers are going to do better. Remember, they won’t be able to use spam calls to promote their scam!
The point of the view expressed in this post is that you DON’T have to see the decisions of the real and simulated people as being “entangled”. If you just treat them as two different people, making two decisions (which if Omega is good at simulation are likely to be the same), then Causal Decision Theory works just fine, recommending taking only one box.
The somewhat strange aspect of the problem is that when making a decision in the Newcomb scenario, you don’t know whether you are the real or the simulated person. But less drastic ignorance of your place in the world is a normal occurrence. For instance, you might know (from family lore) that you are descended from some famous person, but be uncertain whether you are the famous person’s grandchild or great grandchild. Such uncertainty about “who you are” doesn’t undermine Causal Decision Theory.
One can easily think of mundane situations in which A has to decide on some action without knowing whether or not B has or has not already made some decision, and in which how A acts will affect what B decides, if B has not already made their decision. I don’t think such mundane problems pose any sort of problem for causal decision theory. So why would Newcomb’s Problem be different?
Then you know that someone who voiced opinion A that you put in the hat, and also opinion B, likely actually believes opinion B.
(There’s some slack from the possibility that someone else put opinion B in the hat.)