There’s a minor error in the formula giving the cross entropy: you need a minus sign on the RHS so that it reads E[- log P[X|M_2] | M_2]

The preceding text is “Of course, we could be wrong about the distribution—we could use a code optimized for a model M2 which is different from the “true” model M1. In this case, the average number of bits used will be”

It’s hard for me to respect a Safety org so obviously wrong about the most important factors of their chosen topic.

I won’t judge a random celebrity for expecting e.g. very long timelines but an org like this? I’m sure they are very cool people but come on.

As in ultimately more people are likely to like their condition and agree (comparably more) with the AI’s decisions while having roughly equal rights.

If you can’t provide a few unambiguous examples of the dilemma in the post that actually happened in the real world, I’m less likely to take your post seriously.

Might be worth thinking more and then coming up with examples.

I consider you to be basically agreeing with me for 90% of what I intended and your disagreements for the other 10% to be the best written of any so far, and basically valid in all the places I’m not replying to it. I still have a few objections:

What if my highest value is getting a pretty girl with a country-sized dowry, while having not betrayed the Truth? … In short, no, Rationality absolutely can be about both Winning and about The Truth.

I agree the utility function isn’t up for grabs and that that is a coherent set of values to have, but I have this criticism that I want to make that I feel I don’t have the right language to make. Maybe you can help me. I want to call that utility function perverse. The kind of utilityfunction that an entity is probably mistaken to imagine itself as having.

For any particular situation you might find yourself in, for any particular sequence of actions you might do in that situation, there is a possible utilityfunction you could be said to have such that the sequence of actions is the rational behaviour of a perfect omniscient utility maximiser. If nothing else, pick the exact sequence of events that will result, declare that your utility function is +100 for that sequence of events and 0 for anything else, and then declare yourself a supremely efficient rationalist.

Actually doing that would be a mistake. It wouldn’t be making you better. This is not a way to succeed at your goals, this is a way to observe what you’re inclined to do anyway and paint the target around it. Your utility function (fake or otherwise) is supposed to describe stuff you actually want. Why would you want specifically that in particular?

I think the stronger version of Rationality is the version that phrases it as about getting the things you want, whatever those things might be. In that sense, if The Truth is merely a value, you should carefully segment it in your brain out from your practice of rationality: Your rationality is about mirroring the mathematical structure best suited for obtaining goals, and then to whatever degree you value The Truth above its normal instrumental value is something you buy where it’s cheapest like all your other values. Mixing the two makes both worse, you pollute your concept of rational behaviour with a love of the truth (and therefore, for example, are biased towards imagining that other people who display rationality are probably honest, or other people who display honesty are probably rational) and you damage your ability to pursue the truth by not putting in the values category where it belongs where it will lead you to try to cheaply buy more of it.

Of course maybe you’re just the kind of guy who really loves mixing his value for The Truth in with his rationality into a weird soup. That’d explain your actiosn without making you a walking violation of any kind of mathematical law, it’d just be a really weird thing for you to innately want.

I am still trying to find a better way to phrase this argument such that someone might find it persuasive of something, because I don’t expect this phrasing to work.

I say and write things^[3] because I consider those things to be true, relevant, and at least somewhat important. That by itself is very often (possibly usually) sufficient for a thing to be useful in a general sense (i.e., I think that the world is better for me having said it, which necessarily involves the world being better for the people in it). Whether the specific person to whom the thing is nominally or factually addressed will be better off as a result of what I said or wrote is not my concern in any way other than that.

I think I meant something subtly different that what you’ve taken that part to mean. I think you understand that, f other people noticed a pattern that everything you said was false, irrelevant, or unimportant, they would eventually stop bothering to listen when you talk, and this would mean you’d lose the ability to get other people to know things, which is a useful ability to have. This is basically my position! Whether the specific person you address is better off in each specific case isn’t materal because you aren’t trying to always make them better off, you’re just trying to avoid being seen as someone who predictibly doesn’t make them better off. I agree that calculating the full expected consequences to every person of every thing you say isn’t necessary for this purpose.

No, this is a terrible idea. Do not do this. Act consequentialism does not work. … Look, this is going to sound fatuous, but there really isn’t any better general rule than this: you should only lie when doing so is the right thing to do.

I agree that Act Consequentialism doesn’t really work. I was trying to be a Rule consequentialist instead wben I wrote the above rule. I agree that that sounds fatuous, but I think the immediate feeling is pointing at a valid retort: You haven’t operationalized this position into a decision process that a person can actually do (or even pretend to do).

I took great effort to try to right down my policy as something explicit in terms a person could try to do (even though I am willing to admit it is not really correct mostly because finite agent problems), because a person can’t be a real Rule Consequentialist without actually having a Rule. What is the rule for “Only lie when doing so is the right thing to do”? It sounds like an instruction to pass the act to my rightness calculator, but if I program that rule into my rightness calculator, and then give it any input, it gets into an infinite loop. I have an Act Consequentialist rightness calculator as a backup, but if I pass the rule “only lie when doing so is the right thing to do” into that as a backup I’m just right back at doing act consequentialism.

If you can write down a better rule for when to lie the than what I’ve put above (that is also better than the “never” or “only by coming up with galaxy-brained ways it technically isn’t lying” or Eliezer’s meta-honesty idea that I’ve read before) I’d consider you to have (possibly) won this issue, but that’s the real price of entry. It’s not enough to point out the flaws where all my rules don’t work, you have to produce rules that work better.

re 2: Now that you mention it, I realized sharpening can be easily outsourced. My mistake.

re 1: I don’t see it, buying pre-chopped onions is simply not equivalent to having a freshly chopped onion and some vegetables cannot be bought pre-cut. While cutting isn’t a bottleneck for most people I had this chain in mind: (no cutting skills) → (cooking takes more time and is less pleasant) → (Less willingness to try new or complex recipes).

(Also, if you don’t have proper technique, you’re at a higher risk of cutting yourself. In that respect, it’s like free climbing /​ using safety ropes)

re 3: I had self-experiments in general in mind (people run self-experiments, without knowing statistics, or even gathering data), but it did not occur to me that not all self-experiments are QS (probably most aren’t). As written you are, of course, correct.

There are a number of ways that the US seems to have better values than the CCP, by my lights, but it seems incredibly strange to claim the US values being egalitarian, and social equality or harmony more.

Rule of law, fostering diversity, encouraging human excellence? Sure, there you would have an argument. But egalitarian?

China has no specific animal welfare laws. There are also some Chinese that regard animal welfare as a Western import. Maybe the claim that they have no concept at all is too strong, but it’s certainly minimized by previous regimes.
ie

Mao regarded the love for pets and the sympathy for the downtrodden as bourgeoise

And China’s average meat consumption being lower could just be a reflection of their gdp per capita being lower. I don’t know where you got the 14% vegetarian number. I can find 5% online. About the same as US numbers.

Disagree with

• Cooking /​ cutting vegetables (also other things)

• Cutting vegetables /​ sharpening knives

• QS experiments /​ knowing statistics

The first two is pretty much like sketch /​ making pencils and paper, and the third one is absolutely essential and not a skill than you can not have

What would it mean for an AGI to be aligned with “Democracy,” or “Confucianism,” or “Marxism with Chinese characteristics,” or “the American Constitution”? Contingent on a world where such an entity exists and is compatible with my existence, what would my life be like in a weird transhuman future as a non-citizen in each system?

None of these philosophies or ideologies was created with an interplanetary transhuman order in mind, so to some extent a superintelligent AI guided by them, will find itself “out of distribution” when deciding what to do. And how that turns out, should depend on underlying features of the AGI’s thought—how it reasons and how it deals with ontological crisis. We could in fact do some experiments along these lines—tell an existing frontier AI to suppose that it is guided by historic human systems like these, and ask how it might reinterpret the central concepts, in order to deal with being in a situation of relative omnipotence.

Supposing that the human culture of America and China is also a clue to the world that their AIs would build when unleashed, one could look to their science fiction for paradigms of life under cosmic circumstances. The West has lots of science fiction, but the one we keep returning to in the context of AI, is the Culture universe of Iain Banks. As for China, we know about Liu Cixin (“Three-Body Problem” series), and I also dwell on the xianxia novels of Er Gen, which are fantasy but do depict a kind of politics of omnipotence.

This is the kind of comment that becomes harder to take at face value (from you) after reading your dissent on honesty.

I used to really like Matthew Barnett’s posts as providing contrarian but interesting takes.

However, over the last few years, I’ve started to few more negatively about them. I guess I feel that his posts tend to be framed in a really strange way such that, even though there’s often some really good research there, it’s more likely to confuse the average reader than anything else and even if you can untangle the frames, I usually don’t find worth it the time.

I should mention though that as I’ve started to feel more negative about them, I’ve started to read less of them and to engage less deeply with the ones I do look it, so there’s a chance my view would be different if I read more.

I’d probably feel more positive about any posts he writes that are closer to presenting data and further away from interpretation.

That said, Epoch overall has produced some really high-quality content and I’d definitely like to see more independent scenes.

I would expect that player 2 would be able to win almost all of the time for most normal hash functions, as they could just play randomly for the first 39 turns, and then choose one of the 2^8 available moves. It is very unlikely that all of those hashes are zero. (For commonly used hashes, player 2 could just play randomly the whole game and likely win, since the hash of any value is almost never 0.)

Agreed that the current situation is weird and confusing.

The AI Alignment Forum is marketed as the actual forum for AI alignment discussion and research sharing. However, it seems that the majority of discussion shifted to LessWrong itself, in part due to most people not being allowed to post on the Alignment Forum, and most AI Safety related content not being actual AI Alignment research.

I basically agree with Reviewing LessWrong: Screwtape’s Basic Answer. It would be much better if AI Safety related content had its own domain name and home page, with some amount of curated posts flowing to LessWrong and the EA Forum to allow communities to stay aware of each other.

Makes sense. Those were real questions, to be clear.

I am unsure how free the average Chinese person is, nor how to weigh freedom of speech with certain economic freedoms and competent local government, low crime, the tendency of modern democracies to rent seek from the young in favour of the old, zoning laws, restriction on industrial development, a student loan system that seems to be a weird form of indenture. I do come from a country with rather strict hate speech laws. And we do not, in fact, have freedom of speech by any strict definition. And this is a policy American elites in and out of government strongly approve of.

I ask out of relative ignorance of what life in China is like for the average Chinese person, but with a slight suspicion that we might be defining our western notion of ‘freedom’ in such a way that ignores the many ways we are restricted and extracted from, and ways in which the average Chinese may be more free.

It’s very clear the CCP has committed far larger crimes against its people in living memory. But it is also a very different organization than it was at its worst.

I think the question is still worth asking. And the argument worth justifying.

I do think it’d be useful for the rest of us if you put them in a comment. :)

(FWIW I resonated with your motivation, but also think your suggestions fail on the practical grounds jenn mentioned, and would hence on net harm the people you intend to help.)

I’m guessing you live in a country with a US military base? Are you more free than the average Chinese citizen?

It seems there’s an unofficial norm: post about AI safety in LessWrong, post about all other EA stuff in the EA Forum. You can cross-post your AI stuff to the EA Forum if you want, but most people don’t.

I feel like this is pretty confusing. There was a time that I didn’t read LessWrong because I considered myself an AI-safety-focused EA but not a rationalist, until I heard somebody mention this norm. If we encouraged more cross-posting of AI stuff (or at least made the current norm more explicit), maybe the communities on LessWrong and the EA Forum would be more aware of each other, and we wouldn’t get near-duplicate posts like these two.

(Adapted from this comment.)

Side note—it seems there’s an unofficial norm: post about AI safety in LessWrong, post about all other EA stuff in the EA Forum. You can cross-post your AI stuff to the EA Forum if you want, but most people don’t.

I feel like this is pretty confusing. There was a time that I didn’t read LessWrong because I considered myself an AI-safety-focused EA but not a rationalist, until I heard somebody mention this norm. If we encouraged more cross-posting of AI stuff (or at least made the current norm more explicit), maybe we wouldn’t get near-duplicate posts like these two.