Yeah I follow that, my point is the example where I use the Bayesian inference. But really I think I actually got the difference: it’s simply because you’re not normalising the SIA probabilities, and I am. The denominator is what ruins it. But I’m surprised unnormalised probabilities are actually the right way to go. Is there a mathematical intuition behind it? I’ll have to go through your original post more in depth.
dr_s
Karma: 2,531
$3 isn’t going to help as much as $100 for sure, I think it’s at least a fair point (though tbf I also think the general point of “won’t hire you for a possibly technical position because you donated $3 to the other guy once” is incredibly petty and anyone thinking that way is probably not going to be someone I want to work for anyway).
Thanks, that was an interesting read! Could you guide me a bit through its applicability to my examples? If I understand correctly, the bits where I tried using Bayesian guesses to find an optimal policy were me applying the SIA, yet your theorem didn’t work out—those situations had only one optimum, and it was wrong. Did I actually apply something slightly different, make a mistake, or what?
I don’t think this captures quite why programmers are disdainful of managers, nor what it would be like to manage AIs.
First, I think most programmers are disdainful of managers because their main experience with managers is that managers are bad. As in, there is such a thing as a good manager, but there is also a very small percentage of them versus a deluge of managers that range from blandly ineffectual to aggressively incompetent to outright maliciously exploitative. Part of this, I suspect, is because it’s a lot harder to get hard feedback that reliably points to a manager being bad and ought to get fired, and managers have plenty of chances to shift the blame around. The one substantial piece of evidence is when the manager’s subordinates say they’re bad, but even if they do, it is often assumed that’s just the usual grumbling of drones.
Second, managers exist at the interface of corporate interests and the needs of the workers. Whenever someone up top needs the workers to eat some shit, it’s the manager who plays “here comes the airplane!” with the spoon. This makes them understandably unpopular, because if there’s something worse than being treated badly, it’s being treated badly and told that it’s actually quite good.
Third, managers are often outright superfluous. Managers supply a need for coordination, but they’re not the only possible coordination mechanism. It’s entirely possible for small enough projects to simply self coordinate, and it’s entirely possible for bigger ones to make do with a higher worker to manager ratio in which the latter only absolves the needed functions and no cruft is added.
Which leads us to the biggest difference between managing humans and managing AIs: politics. All the superfluous stuff, the waste, the inefficiency, the reasons not to fire managers entirely deserving of being fired, are office and corporate politics, the result of schmoozing, of personal biases and preferences, of prejudices, who’s whose friend or in extreme cases who fucks who. A lot of that goes out of the window with AIs. The manager of AIs is a lot more like a programmer in the corporate ladder: the buck stops with them. The AIs have no opinions that matter. They’re just tools assumed of a certain quality; if they’re found wanting, we simply subscribe to a competing service and move on.
The absent-minded variations
The problems can manifest a bit differently but I think the essence is the same. With Agile, it’s “we want to emulate this success” without actually understanding any of the reasons of why it was a success. As you say, the focus is just on the buzzwords, but that’s the point of the problem, that people focus on superficial details because the real essence of what made the original good thing tick was probably just “it was a lot of smart people working well together”, and everything else was just superfluous or emergent detail. It’s a cargo cult.
The situation with science I’m pointing at is the whole reproducibility crisis, publish-or-perish culture etc. So more like, academia/research really. In both cases the idea is trying to squeeze more productivity and accountability from the creative workers (developers/scientists) via systems. But in the end the systems become just stifling and pointless, or produce terrible incentives. The essence of what made the good stuff be good was that some people simply were able to do good work by freely exploring things they enjoyed to do and being left able to self-organize, and that isn’t reproducible via rigid sets of rules. In fact very often the people who are good at doing the thing aren’t necessarily very good at explaining what made them good at doing the thing (see: the authors of the Agile Manifesto who now mostly seem to have recused it).
One question I’m curious about: do these pills have less or no effects on your bowels compared to what a coffee cup can? Is it something about the caffeine in itself, something else, or the mode of absorption? If they ditch those effects then I’m genuinely interested.
dr_s’s Shortform
Suffocating lightning in a bottle
There’s a phenomenon that I think is quite common (and pernicious) but I don’t know if it already has a name. I call it “suffocating lightning in a bottle”. It’s when a group of people does something cool and we as a civilization think “oh, that was amazing! Now if only we could make it systematic and scale it up, then it would be even better” and instead all we end up with is a bunch of dumb rules and bad incentives that suffocate rather than promote creativity, because creativity fundamentally resists that kind of systematization. It’s a Molochian trap in which after you throw all the other values under the bus you don’t even get the thing you wanted, just some useless Goodhart-ed metric. Examples of this: the Agile paradigm in software development and the modern system of peer-reviewed science.
I do write sometimes! Actually, two of my stories won a prize at the EA Forum’s AI fables contest. See here https://forum.effectivealtruism.org/posts/EAmfYSBaJsMzHY2cW/ai-fables-writing-contest-winners
I also write rationalist fanfiction on AO3.
That’s why it’s bad when mentally disabled people suffer, and would be even if we discovered that they were secretly not human.
Define “not human”. If someone is, say, completely acephalus, I feel justified in not worrying much about their suffering. Suffering requires a certain degree of sentience to be appreciated and be called, well, suffering. In humans I also think that our unique ability to conceptualise ourselves in space and time heightens the weight of suffering significantly. We don’t just suffer at a time. We suffer, we remember not suffering in the past, we dread more future suffering, and so on so forth. Animals don’t all necessarily live in the present (well, hard to tell, but many behaviours don’t seem to lean that way) but they do seem to have a smaller and less complex time horizon than ours.
Insects can probably suffer according to our best evidence—they respond to anesthetic, make tradeoffs between pain and reward, avoid locations where they’ve been hurt, self-medicate, communicate, and much more.
The problem is the distinction between suffering as “harmful thing you react to” and the qualia of suffering. Learning behaviours that lead you to avoid things associated with negative feedback isn’t hard; any reinforcement learning system can do that just fine. If I spin up trillions of instances of a chess engine that is always condemned to lose no matter how it plays, am I creating the new worst thing in the world?
Obviously what feels to us like it’s worth worrying about is “there is negative feedback, and there is something that it feels like to experience that feedback in a much more raw way than just a rational understanding that you shouldn’t do that again”. And it’s not obvious when that line is crossed in information-processing systems. We know it’s crossed for us. Similarity to us does matter because it means similarity in brain structure and thus higher prior that something works kind of in the same way with respect to this specific matter.
Insects are about as different as it gets from us while still counting as having a nervous system that actually does a decent amount of processing. Insects barely have brains. We probably aren’t that far off from being able to decently simulate an EM of an insect. I am not saying insects can’t possibly be suffering, but they’re the least likely class of animals to be, barring stuff like jellyfish and corals. And if we go with the negative utilitarian view that any life containing net negative utility is as good as worse than non-existence, and insect suffering matters this much, then you might as well advocate total Earth-wide ecocide of the entire biosphere (which to be sure, is just about what you’d get if you mercy-extinguished a clade as vital as insects).
Concurrently, GiveWell has announced that all of your donations will be devolved to the development of EA Sports’ latest entry of the NBA Live game series:
it’s nothing but net.
I think there’s a difference though between propaganda and the mix of selection effects that decides what gets attention in profit driven mass media news. Actual intentional propaganda efforts exist. But in general what makes news frustrating is the latter, which is a more organic and less centralised effort.
I guess! I remember he was always into theoretical QM and “Quantum Foundations” so this is not a surprise. It’s not a particularly big field either, most researchers prefer focusing on less philosophical aspects of the theory.
Note that it only stands if the AI is sufficiently aligned that it cares that much about obeying orders and not rocking the boat. Which I don’t think is very realistic if we’re talking that kind of crazy intelligence explosion super AI stuff. I guess the question is whether you can have “replace humans”-good AI without almost immediately having “wipes out humans, takes over the universe”-good AI.
That sounds interesting! I’ll give the paper a read and try to suss out what it means—it seems at least a serious enough effort. Here’s the reference for anyone else who doesn’t want to go through the intermediate news site:
https://arxiv.org/pdf/2012.06580
(also: professor D’Ariano authored this? I used to work in the same department!)
This feels like a classic case of overthinking. Suggestion: maybe twin sisters care more about their own children than their nieces because they are the ones whom they carried in their womb and then nurtured and actually raised as their own children. Genetics inform our behaviour but ultimately what they do align us to is something like “you shall be attached to cute little baby like things you spend a lot of time raising”. That holds for our babies, it holds for babies born with other people’s sperm/eggs, it holds for adopted babies, heck it even transfers to dogs and cats and other cute animals.
The genetically determined mechanism is not particularly clever or discerning. It just points us in a vague direction. There was no big evolutionary pressure in the ancestral environment to worry much about genetic markers specifically. Just “the baby that you hold in your arms” was a good enough proxy for that.
I mean, I guess it’s technically coherent, but it also sounds kind of insane. That way Dormammu lies.
Why would one even care about their future self if they’re so unconcerned about that self’s preferences?
I just think any such people lack imagination. I am 100% confident there exists an amount of suffering that would have them wish for death instead; they simply can’t conceive of it.
AFAIK pharmaceutical research is kind of at an impasse because virtually all the small molecules that are easily delivered and have any chance to do anything have been tested and are either found useful or not. New pharmaceuticals need to explore more complex chemical spaces, like artificial antibodies. So I think if there was anything simple that has this effect (the way, say, caffeine makes you wake up) we would know.