I’m writing a book about epistemology. It’s about The Problem of the Criterion, why it’s important, and what it has to tell us about how we approach knowing the truth.
I’ve also written a lot about AI safety. Some of the more interesting stuff can be found at the site of my currently-dormant AI safety org, PAISRI.
How do you get something to take virtuous action without optimizing for taking virtuous actions, and how is this different from optimizing for virtue?
Yeah I guess I should be clear that I generally like the idea of building virtuous AI and maybe somehow this solves some of the problems we have with other designs, the trick is building something that actually implements whatever we think it means to be virtuous, which means getting precise enough about what it means to be virtuous that we can be sure we don’t simply collapse back into the default thing all negative feedback systems do: optimize for their targets as hard as they can (with “can” doing a lot of work here!).
No matter what the goal, power seeking is of general utility. Even if an AI is optimizing for virtue instead of some other goal, more power would, in general, give them more ability to behave virtuously. Even if the virtue is something like “be an equal partner with other beings”, an AI could ensure equality by gaining lots of power and enforcing equality on everyone.
More reasons to shower: smelling nice is good.
And lo, the people, though they were virtuous, were not virtuous enough to receive the blessing of the 12th virtue.
Yes, though note that this is still concerning.
Normally the way this works in a startup is that spend exceeding revenue should be in service of bootstrapping the company. That means that money is usually spent in a few ways:
R&D
functions that have relatively high floors (relative to revenue) to function well but then grow sublinearly (marketing, sales, etc.)
to gain a first mover advantage that no one will be able to catch up to
OpenAIs spend is concerning for the same reason, say, Uber and Netflix have/had concerning spend: they have to actually win their market to have a chance of reaping rewards, and if they don’t they’ll simply be forced to raise prices and cut quality/R&D.
I think there’s something to be said for this straightforward counterargument here: conceptual rounding off is performing the valuable service of lumping together ideas that are not that different and only seem importantly different to their authors/supporters rather than actually meaningfully different ideas.
An idea should either be precisely defined enough that it’s clear why it can’t be rounded off (once the precise definition is known), or it’s a vague idea and it either needs to become more precise to avoid being rounded or it is inherently vague and being vague there can’t be much harm from rounding because it already wasn’t clear where its boundaries were in concept space.
For what it’s worth I do something pretty similar to this using Claude Code. I can just point it at the tests and say “hey, make the code make this pass” and it will come up with something, although the something isn’t always what I want if the tests are underspecified!
My understanding is that that cloud method wastes perfume because less of it actually gets on you. Better to spray directly on the skin, usually either on the chest or on the back of the neck. If you want some on your hands, rub the back of your hands on the sprayed area immediately after spraying.
3 sprays with the cloud method is probably fine.
Haha, yes, hopefully that is obvious to most readers, but one never knows!
So while your point is mostly true, I want to highlight there are some situations where simply asking people to respect your food norms is a problem, and they mostly arise in a specific sort of culture that is especially communal with regard to food and sees you as part of the ingroup.
For example, it’s a traditional upper-class Anglo norm that it’s rude to put your hosts out by asking them to make you something special to accommodate your diet. You’re expected to get along and eat what everyone else eats. You will be accommodated if you ask, but you will also be substantial downgraded in how willing to get along you are, and you’ll be a less desired dinner guest, and thus get fewer invites and be less in.
I’ve heard of similar issues in some East Asian cultures where going vegan is seen as an affront to the family. “What do you mean you won’t eat my cooking?!? Do you think you’re better than your mother???!”
The problem is that food is tied with group membership, and you’re expected to eat the same food as the rest of the ingroup. If you’re not a rare outsider guest, you’ll be seen as defecting on group cohesion.
But most Westerners are not part of cultures like these. Western culture is highly atomized, and everyone is seen as a unique individual, so it’s not unusual that individuals might have unique food needs, and it becomes polite and a sign of a good host to accommodate everybody. But this is historically an unusual norm to have within the ingroup.
Okay, you just doubled down, so clearly this discussion isn’t going anywhere. It’s also off topic anyway since it’s not directly addressing the claims of this post.
Please kindly refrain from continuing this line for discussion on this post. I’d welcome additional comments if you wanted to address the claims of the post directly, though.
Your claims overgeneralize and it makes them false. To assure I’m not just biased because I am religious myself, here’s Claude’s take on your comment when I asked it to fact check it.
Prompt: can you fact check this comment that was posted to an online forum (not mine but i suspect it might not be correct but don’t trust myself not to be biased so looking for your take) [your comment copied in]
Response:
I’ll analyze this forum comment by examining its claims about religion, compassion, and philosophical history.
The comment contains several sweeping generalizations and historical inaccuracies:
1. **Religious texts and violence**: While Abrahamic texts do contain violent passages, characterizing the “overwhelming majority” as “justifications for genocide and ethnic supremacy” is factually incorrect. These texts contain diverse content including ethical teachings, poetry, historical narratives, and legal codes. The violent passages represent a minority of the content.
2. **”2,000 years of the worst violence in history”**: This statement ignores that violence has existed in all human societies regardless of religion. It also overlooks that many historical atrocities were driven by non-religious ideologies (e.g., 20th century totalitarian regimes).
3. **Religious monopoly on compassion**: While some religious groups do claim exclusive moral authority, many traditions explicitly teach universal compassion that extends beyond group boundaries. The comment oversimplifies complex theological positions across diverse traditions.
4. **Platonic origins claim**: The assertion that Abrahamic religions derived their concepts of compassion and empathy primarily from Plato is historically questionable. While Hellenistic philosophy influenced later Jewish and Christian thought, these traditions also drew from their own cultural and textual sources that pre-dated significant Greek influence.
5. **”Universal religion”**: This term is never clearly defined, making many of the claims difficult to evaluate precisely.
The comment does raise legitimate concerns about religious exclusivism and historical misuse of religion to justify violence, but its broad generalizations undermine its credibility as an objective analysis of religion’s relationship to compassion and empathy.
Point 5 is obviously an artifact of me failing to give Claude context on what universal religion means, and I didn’t define it in the article, but I think it’s clear what I mean: religions that see it as their purpose to apply to all people, not just to a single ethnic group or location.
Ranked in order of how interesting they were to me when I got interested in them, which means in approximately chronological order because the more ideas I knew the less surprising new ideas were (since they were in part predicted by earlier ideas that had been very interesting).
Cybernetics
Evolution
Game Theory
Developmental psychology
Dependent Origination
The Problem of the Criterion
While history suggests we should be skeptical, current AI models produce real results of economic value, not just interesting demos. This suggests that we should be willing to take more seriously the possibility that they will be produce TAI since they are more clearly on that path and already having significant transformative effects on the world.
I think it’s a mistake in many cases to let philosophy override what you care about. That’s letting S2 do S1′s job.
I’m not saying no one should ever be able to be convinced to care about something, only that the convincing, even if a logical argument is part of it, should not be all of it.
I don’t think a philosophy of mind is necessary for this, no, although I can see why it might seem like it is if you’ve already assumed that philosophy is necessary to understand the world.
It’s enough to just be able to model other minds in the world to know how to show them compassion, and even without modeling compassion can be enacted, even if it’s not known to be compassionate behavior. This modeling need not rise to the level of philosophy to get the job done.
Depends on whose sense of smell you’re optimizing for.
My cats like to sniff each other’s butts.
Many dogs love smelling stinky garbage.
I’m not sure I would trust my cats’ senses of smell to tell me if I would smell good to other humans.