tailcalled

Karma: 7,865

tailcalled Apr 15, 2025, 9:57 PM
6 points
0
in reply to: Steven Byrnes’s comment on: johnswentworth’s Shortform
(IMO this is kinda unrelated to the OP, but I want to continue this thread.)
I think it’s quite related to the OP. If a field is founded on a wrong assumption, then people only end up working in the field if they have some sort of blind spot, and that blind spot leads to their work being fake.
Have you elaborated on this anywhere?
Not hugely. One tricky bit is that it basically ends up boiling down to “the original arguments don’t hold up if you think about them”, but the exact way they don’t hold up depends on what the argument is, so it’s kind of hard to respond to in general.
Perhaps you missed it, but some guy in 2022 wrote this great post which claimed that “Consequentialism, broadly defined, is a general and useful way to develop capabilities.” ;-)
Haha! I think I mostly still stand by the post. In particular, “Consequentialism, broadly defined, is a general and useful way to develop capabilities.” remains true; it’s just that intelligence relies on patterns and thus works much better on common things (which must be small, because they are fragments of a finite world), than on rare things (which can be big, though don’t have to). This means that consequentialism isn’t very good at developing powerful capabilities unless it works in an environment that has already been highly filtered to be highly homogenous, because an inhomogenous environment is going to BTFO the intelligence.
(I’m not sure I stand 101% by my post; there’s some funky business about how to count evolution that I still haven’t settled on yet. And I was too quick to go from “imitation learning isn’t going to lead to far-superhuman abilities” to “consequentialism is the road to far-superhuman abilities”. But yeah I’m actually surprised at how well I stand by my old view despite my massive recent updates.)
I’m actually just in the course of writing something about why “consequentialism provides an extremely powerful but difficult-to-align method of converting intelligence into agency” … maybe I can send you the draft for criticism when it’s ready?
Sounds good!

tailcalled Apr 15, 2025, 8:46 AM
13 points
4
on: A Dissent on Honesty
Actually, even if your personality is good enough, you should probably still pretend to be Flynn Rider, because his personality is better. It was, after all, carefully designed by a crack team of imagineers. Was yours? Didn’t think so.
Personalities don’t just fall into a linear ranking from worse to better.
Imagineers’ job isn’t to design a good personality for a friendless nerd, it’s to come up with children’s stories that inspire and entertain parents and which they proudly want their children to consume.
The parents think they should try to balance the demands of society with the needs of their children by teaching their children to scam the surrounding society but being honest about the situation with their loved ones. Disney is assisting the parents with producing propaganda/instructions for it.
https://benjaminrosshoffman.com/guilt-shame-and-depravity/
Basing your life on scamming society is a bad idea but you shouldn’t solve it by also trying to scam your loved ones. If you are honest, you can more easily collaborate with others to figure out what is needed and how you can contribute and what you want.

tailcalled Apr 15, 2025, 7:18 AM
2 points
0
in reply to: Kajus’s comment on: johnswentworth’s Shortform
I’m not trying to present johnswentworth’s position, I’m trying to present my position.

tailcalled Apr 14, 2025, 7:48 PM
1 point
−27
in reply to: niplav’s comment on: johnswentworth’s Shortform
The entire field is based on fears that consequentialism provides an extremely powerful but difficult-to-align method of converting intelligence into agency. This is basically wrong. Yes, people attempt to justify it with coherence theorems, but obviously you can be approximately-coherent/approximately-consequentialist and yet still completely un-agentic, so this justification falls flat. Since the field is based on a wrong assumption with bogus justification, it’s all fake.
What links here?
- tailcalled's comment on Alexander Gietelink Oldenziel’s Shortform by Alexander Gietelink Oldenziel (Apr 16, 2025, 10:13 AM; 2 points)

tailcalled Apr 14, 2025, 2:19 PM
3 points
7
on: Slopworld 2035: The dangers of mediocre AI
The big picture is plausible but one major error you make is assuming “academics” will be a solid bastion of opposition. My understanding is that academics are often some of the first ones to fall (like when teachers struggle with students who use ChatGPT to cheat on homework), and many of the academic complaints about AI are just as slop-y as what the AI produces.

tailcalled Apr 14, 2025, 11:00 AM
3 points
0
in reply to: Knight Lee’s comment on: Eli’s shortform feed
Maybe someone who believes in following the will of the majority even if he/she disagrees (and could easily become a dictator)?
Do you mean “resigns from a presidential position/declines a dictatorial position because they disagree with the will of the people” or “makes policy they know will be bad because the people demand it”?
Maybe a good parent who listens to his/her child’s dreams?
Can you expand on this?

tailcalled Apr 14, 2025, 10:51 AM
3 points
0
in reply to: Knight Lee’s comment on: Eli’s shortform feed
Can you give 1 example of a person choosing to be corrigible to someone they are not dependent upon for resources/information and who they have much more expertise than?

tailcalled Apr 14, 2025, 10:44 AM
2 points
0
in reply to: habryka’s comment on: ricraz’s Shortform
I feel like “evil” and “corruption” mean something different.
Corruption is about selfish people exchanging their power within a system for favors (often outside the system) when they’re not supposed to according to the rules of the system. For example a policeman taking bribes. It’s something the creators/owners of the system should try to eliminate, but if the system itself is bad (e.g. Nazi Germany during the Holocaust), corruption might be something you sometimes ought to seek out instead of to avoid, like with Schindler saving his Jews.
“Evil” I’ve in the past tended to take to take to refer to a sort of generic expression of badness (like you might call a sadistic sexual murderer evil, and you might call Hitler evil, and you might call plantation owners evil, but this has nothing to do with each other), but that was partly due to me naively believing that everyone is “trying to be good” in some sense. Like if I had to define evil, I would have defined it as “doing bad stuff for badness’s sake, the inversion of good, though of course nobody actually is like that so it’s only really used hyperbolically or for fictional characters as hyperstimuli”.
But after learning more about morality, there seem to be multiple things that can be called “evil”:
- Antinormativity (which admittedly is pretty adjacent to corruption, like if people are trying to stop corruption, then the corruption can use antinormativity to survive)
- Coolness, i.e. countersignalling against goodness-hyperstimuli wielded by authorities, i.e. demonstrating an ability and desire to break the rules
- People who hate great people cherry-picking unfortunate side-effects of great people’s activities to make good people think that the great people are conspiring against good people and that they must fight the great people
- Leaders who commit to stopping the above by selecting for people who do bad stuff to prove their loyalty to those leaders (think e.g. the Trump administration)
I think “evil” is sufficiently much used in the generic sense that it doesn’t make sense to insist that any of the above are strictly correct. However if it’s just trying to describe someone who might unpredictably do something bad then I think I’d use words like “dangerous” or “creepy”, and if it’s just trying to describe someone who carries memes that would unpredictably do something bad then I think I’d use words like “brainworms” (rather than evil).

tailcalled Apr 14, 2025, 8:51 AM
3 points
0
in reply to: Knight Lee’s comment on: Eli’s shortform feed
If the AI can’t do much without coordinating with a logistics and intelligence network and collaborating with a number of other agents, and its contact to this network routes through a commanding agent that is as capable if not more capable than the AI itself, then sure, it may be relatively feasible to make the AI corrigible to said commanding agent, if that is what you want it to be.
(This is meant to be analogous to the soldier-commander example.)
But was that the AI regime you expect to find yourself working with? In particular I’d expect you expect that the commanding agent would be another AI, in which case being corrigible to them is not sufficient.

tailcalled Apr 14, 2025, 8:29 AM
3 points
0
in reply to: Knight Lee’s comment on: Eli’s shortform feed
Discriminating on the basis of the creators vs a random guy on the street helps with many of the easiest cases, but in an adversarial context, it’s not enough to have something that works for all the easiest cases, you need something that can’t predictably made to fail by a highly motivated adversary.
Like you could easily do some sort of data augmentation to add attempts at invoking the corrigibility system from random guys on the street, and then train it not to respond to that. But there’ll still be lots of other vulnerabilities.

tailcalled Apr 14, 2025, 7:07 AM
3 points
0
in reply to: Eli Tyre’s comment on: Eli’s shortform feed
Let’s say you are using the AI for some highly sensitive matter where it’s important that it resists prompt-hacking—e.g. driving a car (prompt injections could trigger car crashes), something where it makes financial transactions on the basis of public information (online websites might scam it), or military drones (the enemy might be able to convince the AI to attack the country that sent it).
A general method for ensuring corrigibility is to be eager to follow anything instruction-like that you see. However, this interferes with being good at resisting prompt-hacking.

tailcalled Apr 12, 2025, 6:36 PM
5 points
1
on: What is autism?
My current best guess is that:
- Like for most other concepts, we don’t have rigorous statistics and measurements showing that there is a natural clustering of autism symptoms, (there are some non-rigorous ones though)
- When various schools of psychotherapy, psychiatry and pediatrics sorted children with behavioral issues together, they often ended up with an autistic group,
- Each school has their own diagnosis on what exactly is wrong in the case of autism, and presumably they aren’t all correct about all autistic people, so to know the True Reason autism is “a thing”, you’d first have to figure out which school is correct in its analysis of autism,
- “Autism” as a concept exists because the different schools mostly agreed that the kids in question had a similar pathology, even if they disagreed on what the pathology is.

tailcalled Apr 11, 2025, 10:24 AM
3 points
0
on: Why are neuro-symbolic systems not considered when it comes to AI Safety?
https://www.lesswrong.com/posts/gebzzEwn2TaA6rGkc/deep-learning-systems-are-not-less-interpretable-than-logic

tailcalled Apr 4, 2025, 11:20 AM
2 points
0
in reply to: Davidmanheim’s comment on: Is instrumental convergence a thing for virtue-driven agents?
The assumption of virtue ethics isn’t that virtue is unknown and must be discovered—it’s that it’s known and must be pursued.
If it is known, then why do you not ever answer my queries about providing an explicit algorithm for converting intelligence into virtuous agency, instead running in circles about how There Must Be A Utility Function!?
If the virtuous action, as you posit, is to consume ice cream, intelligence would allow an agent to acquire more ice cream, eat more over time by not making themselves sick, etc.
I’m not disagreeing with this, I’m saying that if you apply the arguments which show that you can fit a utility function to any policy to the policies that turn down some ice cream, then as you increase intelligence and that increases the pursuit of ice cream, the resulting policies will score lower on the utility function which values turning down ice cream.
But any such decision algorithm, for a virtue ethicist, is routing through continued re-evaluation of whether the acts are virtuous, in the current context, not embracing some farcical LDT version of needing to pursue ice cream at all costs. Your assumption, which is evidently that the entire thing turns into a compressed and decontextualized utility function (“algorithm”) is ignoring the entire hypothetical.
You’re the one who said that virtue ethics implies a utility function! I didn’t say anything about it being compressed and decontextualized, except as a hypothetical example of what virtue ethics is because you refused to provide an implementation of virtue ethics and instead require abstracting over it.
I’m not interested in continuing this conversation until you stop strawmanning me.

tailcalled 4 Apr 2025 7:37 UTC
2 points
0
in reply to: Davidmanheim’s comment on: Is instrumental convergence a thing for virtue-driven agents?
No, that’s not my argument.
Let’s imagine that True Virtue is seeking and eating ice cream, but that you don’t know what true virtue is for some reason.
Now let’s imagine that we have some algorithm for turning intelligence into virtuous agency. (This is not an assumption that I’m willing to grant (since you haven’t given something like argmax for virtue), and really that’s the biggest issue with my proposal, but let’s entertain it to see my point.)
If the algorithm is run on the basis of some implementation of intelligence that is not good enough, then the resulting agent might turn down some opportunities to get ice cream, by mistake, and instead do something else, such as pursue money (but less money than you could get the ice cream for). As a result of this, you would conclude that pursuing ice cream is not virtuous, or at least, not as virtuous as pursuing money.
If you then turn up the level of intelligence, the resulting agent would pursue ice cream in this situation where it previously pursued virtue. However, this would make it score worse on your inferred utility function where pursuing money is more virtuous than pursuing intelligence.
Now of course you could say that your conclusion that pursuing ice cream is less virtuous than pursuing money is wrong. But then you can only say that if you grant that you cannot infer a virtue-ethical utility function from a virtue-ethical policy, as this utility function was inferred from the policy.

tailcalled 4 Apr 2025 5:49 UTC
2 points
0
in reply to: Davidmanheim’s comment on: Is instrumental convergence a thing for virtue-driven agents?
I didn’t say you need to understand what an argument is, I said you need to understand your own argument.
It is true that if the utility functions cover a sufficiently broad set of possibilities, any “reasonable” policy (for a controversial definition of “reasonable”) maximizes a utility function, and if the utility functions cover an even broader set of possibilities, literally any policy maximizes a utility function.
But, if you want to reference these facts, you should know why they are true. For instance, here’s a rough sketch of a method for finding a utility function for the first statement:
If you ask a reasonable policy to pick between two options, it shouldn’t have circular preferences, so you should be able to offer it different options and follow the preferred one until you find the absolute best scenario according to the policy. Similarly, you should be able to follow the dispreferred one until you find the absolute worst scenario according to the policy. Then you can define the utility of any outcome based on the probability mixture of the best and worst scenario where the policy switches between preferring the outcome vs preferring the probability mixture.
Now let’s say there’s an option where e.g. you’re not smart enough to realize that option gives you ice cream. Then you won’t be counting the ice cream when you decide at what threshold you prefer that option to the mixture. But then that means the induced utility function won’t include the preference for ice cream.

tailcalled 3 Apr 2025 22:54 UTC
−2 points
0
in reply to: Davidmanheim’s comment on: Is instrumental convergence a thing for virtue-driven agents?
I’m showing that the assumptions necessary for your argument don’t hold, so you need to better understand your own argument.

tailcalled 3 Apr 2025 5:56 UTC
2 points
0
in reply to: Davidmanheim’s comment on: Is instrumental convergence a thing for virtue-driven agents?
The methods for converting policies to utility functions assume no systematic errors, which doesn’t seem compatible with varying the intelligence levels.

tailcalled 3 Apr 2025 5:18 UTC
2 points
0
in reply to: mattmacdermott’s comment on: Is instrumental convergence a thing for virtue-driven agents?
This.
In particular imagine if the state space of the MDP factors into three variables x, y and z, and the agent has a bunch of actions with complicated influence on x, y and z but also just some actions that override y directly with a given value.
In some such MDPs, you might want a policy that does nothing other than copy a specific function of x to y. This policy could easily be seen as a virtue, e.g. if x is some type of event and y is some logging or broadcasting input, then it would be a sort of information-sharing virtue.
While there are certain circumstances where consequentialism can specify this virtue, it’s quite difficult to do in general. (E.g. you can’t just minimize the difference between f(x) and y because then it might manipulate x instead of y.)

tailcalled 2 Apr 2025 12:34 UTC
6 points
2
in reply to: Davidmanheim’s comment on: Is instrumental convergence a thing for virtue-driven agents?
I didn’t claim virtue ethics says not to predict consequences of actions. I said that a virtue is more like a procedure than it is like a utility function. A procedure can include a subroutine predicting the consequences of actions and it doesn’t become any more of a utility function by that.
The notion that “intelligence is channeled differently” under virtue ethics requires some sort of rule, like the consequentialist argmax or Bayes, for converting intelligence into ways of choosing.