Congratulations!
Also you should remember that LW has a fairly wide knowledge base. If you’re looking for a place to get started on a complex topic, I’ll bet that this site would be a good place to ask a few initial questions and establish a broad research outline.
We here are largely aware of Robertson’s comments not because they have particular merit as a thought experiment, but because they occupy the sweet spot of maximizing controversy. That is, it is easy to present as objectionable within Blue Tribe, and easy to present as defensible in Red Tribe, and so in the end it’s a fairly textbook toxoplasma. This isn’t to say that the general question isn’t interesting; it’s just more important than usual that interested parties treat the thought experiment like a finger pointing to an interesting argument.
Personally, I find it fairly interesting that Robertson (et. al.) is concerned with assigning moral legitimacy to his outrage at suffering these various horrible events. Going to the extreme case has the advantages that Scott articulated, but it also seems to blunt the perceived need for a complex ethic. The things are ‘bad’ in the sense that any sane person would be deeply unhappy if they occurred; what is the point of invoking a whole metaphysics to justify that near-universal impulse? Wouldn’t it be more interesting to focus on the places where an objective moral system would produce different results?
I would happily take advantage of such a system. (Also I would be a little worried about regular injections of political tarbabies.)
I forget who said this originally, but much of rationalism is internalizing the fact that you, yes you, are prone to all manner of biases and mental tics that lead to error. If ‘arrogance’ as it’s being used here is something that interferes with recognizing the errors that you’re making in any given moment, then arrogance is certainly antithetical to rationalism.
On the other hand, I’m fairly used to people thinking of me as arrogant in person-to-person communication, and when they give me that label it never has much to do with my willingness to admit error. Usually it has to do with my vocabulary, or other patterns of behavior that they interpret as a desire to be seen as intellectually superior. If that’s what ‘arrogance’ means, then it’s more orthogonal to rationalism. Heck, arrogance as an affect might even be rational in certain circumstances, depending on how you want to be seen during particular social situations.
If you are actually concerned quite a lot with being better than other people, and your challenges are not directly competitive or collaborative in nature (if you’re trying to invent something in your garage, or write well, or do well in classes that aren’t graded on a curve), then that form of arrogance is probably a failure mode. It implies that you’re performing a social role, not trying to succeed, and so you’ll tend to optimize for best appearances and not best results.
My housemate has this exact problem- right down to the issues with jewelry in particular. If she has to shake hands with somebody who’s wearing a metal ring, she has to sort of ritualistically wipe off her hands afterwards. Metal in general seems to trigger the reaction much more strongly, so she’ll have problems with loose coins but not stickers.
It’s been persistent throughout her life, I understand, but exposure therapy has reduced its severity.
Pratchett himself stated an intention to commit suicide before his disease progressed past a certain point. “To jump before I am pushed”, I believe was the phrase he used at one point.
http://www.theguardian.com/society/2010/feb/02/terry-pratchett-assisted-suicide-tribunal
The BBC claims that he didn’t take his own life, and given his advocacy I think that his family would have been honest about his suicide if it were one, but it’s a reason to look more closely at least.
One of the things that we can’t necessarily know is whether Terry Pratchett was likely to become Terry Pratchett, given his beginning. We never do hear about the failures. It’s not just that mediocre students often fail to make something of themselves- how often do mediocre students with a mind like Pratchett’s accomplish world changing things like Discworld? If you knew a priori that you had such a mind, and wanted to maximize your contributions, would it be a good idea to be an aimless student?
That said, it may be that you wouldn’t be likely to get a Pratchett without a certain baseline of failed attempts; and that a youth spent in such a way may be bad for the individual on average and good for society in aggregate. It would be an interesting dilemma.
Broadly speaking, I’m suspicious of social solutions to problems that will persist for geological periods of time. If we’re playing the civilization game for the long haul, then the threat of overpopulation could simply wait out any particular legal regime or government.
That argument goes hinky in the event of FAI singleton, of course.
I am fairly confident that I understand your intentions here. A quick summary, just to test myself:
HAL cares only about world states in which an extremely unlikely thermodynamic even occurs- namely, the world in which one hundred random bits are generated spontaneously during a specific time interval. HAL is perfectly aware that these are unlikely events, but cannot act in such a way as to make the event more likely. HAL will therefore increase total utility over all possible worlds where the unlikely even occurs, and otherwise ignore the consequences of its choices.
This time interval corresponds by design with an actual signal being sent. HAL expects the signal to be sent, with a very small chance that it will be overwritten by spontaneously generated bits and thus be one of the words where it wants to maximize utility. Within the domain of world states that the machine cares about, the string of bits is random. There is a string among all these worlds states that corresponds to the signal, but it is the world where that signal is generated randomly by the spontaneously generated bits. Thus, within the domain of interest to HAL, the signal is extremely unlikely, whereas within all domains known to HAL, the signal is extremely likely to occur by means of not being overwritten in the first place. Therefore, the machine’s behavior will treat the actual signal in a counterfactual way despite HAL’s object-level knowledge that the signal will occur with high probability.
If that’s correct, then it seems like a very interesting proposal!
I do see at least one difference between this setup, and a legitimate counterfactual belief. In particular, you’ve got to worry about behavior in which (1-epsilon)% of all possible worlds have a constant utility. It may not be strictly equivalent to the simple counterfactual belief. Suppose, in a preposterous example, that there exists some device which marginally increases your ability to detect thermodynamic miracles (or otherwise increases your utility during such a miracle); unfortunately, if no thermodynamic miracle is detected, it explodes and destroys the Earth. If you simply believe in the usual way that a thermodynamic miracle is very likely to occur, you might not want to use the device, since it’s got catastrophic consequences for the world where your expectation is false. But if the non-miraculous world states are simply irrelevant, then you’d happily use the device.
As I think about it, I think maybe the real weirdness comes from the fact that your AI doesn’t have to worry about the possibility of it being wrong about there having been a thermodynamic miracle. If it responds to the false belief that a thermodynamic miracle has occurred, there can be no negative consequences.
It can account for the ‘minimal’ probability that the signal itself occurs, of course- that’s included in the ‘epsilon’ domain of worlds that it cares about. But when the signal went through, the AI would not necessarily be acting in a reasonable way on the probability that this was a non-miraculous event.
A simiar one by Vonnegut:
It was a movie about American bombers in the Second World War and the gallant men who flew them. Seen backwards by Billy, the story went like this: American planes, full of holes and wounded men and corpses took off backwards from an airfield in England. Over France a few German fighter plans flew at them backwards, sucked bullets and shell fragments from some of the planes and crewmen. They did the same for wrecked American bombers on the ground, and those planes flew up backwards to join the formation. The formation flew backwards over a German city that was in flames. The bombers opened their bomb bay doors, exerted a miraculous magnetism which shrunk the fires, gathered them into cylindrical steel containers, and lifted the containers into the bellies of the planes. The containers were stored neatly in racks. The Germans below had miraculous devices of their own, which were long steel tubes. They used them to suck more fragments from the crewmen and planes. But there were still a few wounded Americans, though, and some of the bombers were in bad repair. Over France, though, German fighters came up again, made everything and everybody good as new. When the bombers got back to their base, the steel cylinders were taken from the racks and shipped back to the United States of America, where factories were operating night and day, dismantling the cylinders, separating the dangerous contents into minerals. Touchingly, it was mainly women who did this work. The minerals were then shipped to specialists in remote areas. It was their business to put them into the ground, to hide them cleverly so they would never hurt anybody ever again. The American fliers turned in their uniforms, became high school kids. And Hitler turned into a baby, Billy Pilgrim supposed. That wasn’t in the movie. Billy was extrapolating. Everybody turned into a baby, and all humanity, without exception, conspired biologically to produce two perfect people named Adam and Eve, he supposed.
Capitalists (not the only elites, but a good fraction of them) serve the economy as speculators and coordinators, not as direct providers of a service. It’s a facilitator class of being, and as such things like ‘people skills’ are central to the practice.
In the same way, they benefit from a very wide-angle view of the business landscape- Tim Cook is notorious for managing supply chains for Apple products through the most elaborate contortions. An engineer who is an expert in the design of iPhone speaker systems is unlikely to have a really informed view of how much his speakers’ wiring will cost in three years.
A good car salesman is one who convinces as many people as possible to buy a car, but a banker who gave out as many loans as possible would be a terrible banker. The banker’s job is to carefully consider which businesses and people are most likely to provide a return on an investment, given both the characteristics of the individual asking for money and the likely trends of the economy as a whole. That looks a whole lot like being a generalist with good people skills.
You’re comparing two different measures, military casualties versus total casualties. For a reasonably good apples-to-apples measure of wartime violence over time, Better Angels of our Nature is one that I found readable and informative.
IIRC, overall per capita violent death during the world wars was roughly comparable to living in a tribal society during a typical time- of course, for us, those were the exception and not the rule.
In my experience, virtuosity is often roughly measured by the answer to questions like “what fraction of the population could have achieved this goal?” or “how many hours of practice were required to gain the necessary skills for this?”, depending on the circumstances in which the word is used. I suppose that’s fairly objective, although not precise. If painter A could paint both X and Y, and many painters B, C, D… could paint X but not Y, that is some evidence that painting Y is more ‘excellent’ than X in some way that goes beyond preference.
It can also be used as a self-compliment on the part of an audience member; in this usage, it is implied that one must have a great deal of experience with the medium in order to appreciate the work.
In the more frequently considered case of a non-stable utility function, my understanding is that the agent will not try to identify the terminal attractor and then act according to that- it doesn’t care about what ‘it’ will value in the future, except instrumentally. Rather, it will attempt to maximize its current utility function, given a future agent/self acting according to a different function. Metaphorically, it gets one move in a chess game against its future selves.
I don’t see any reason for a temporarily uncertain agent to act any differently. If there is no function that is, right now, motivating it to maximize paperclips, why should it care that it will be so motivated in the future? That would seem to require a kind of recursive utility function, one in which it gains utility from maximizing its utility function in the abstract.
Well, let’s further say that you assign p(+u)=0.51 and p(-u)=0.49, slightly favoring the production of paperclips over their destruction. And just to keep it a toy problem, you’ve got a paperclip-making button and a paperclip-destroying button you can push, and no other means of interacting with reality.
A plain old ‘confident’ paperclip maximizer in this situation will happily just push the former button all day, receiving one Point every time it does so. But an uncertain agent will have the exact same behavior; the only difference is that it only gets .02 Points every time it pushes the button, and thus a lower overall score in the same period of time. But the number of paperclips produced is identical. The agent would not (for example) push the ‘destroy’ button 49 times and the ‘create’ button 51 times. In practical effect, this is as inconsequential as telling the confident agent that it gets two Points for every paperclip.
So in this toy problem, at least, uncertainty isn’t a moderating force. On the other hand, I would intuitively expect different behavior in a less ‘toy’ problem- for example, an uncertain maximizer might build every paperclip with a secret self-destruct command so that the number of paperclips could be quickly reduced to zero. So there’s a line somewhere where behavior changes. Maybe a good way to phrase my question would be- what are the special circumstances under which an uncertain utility function produces a change in behavior?
Made me think of Rawl’s veil of ignorance, somewhat. I wonder- is there a whole family of techniques along the lines of “design intelligence B, given some ambiguity about your own values”, with different forms or degrees of uncertainty?
It seems like it should avoid extreme or weirdly specialized results (i.e. paper-clipping), since hedging your bets is an immediate consequence. But it’s still highly dependent on the language you’re using to model those values in the first place.
I’m a little unclear on the behavioral consequences of ‘utility function uncertainty’ as opposed to the more usual empirical uncertainty. Technically, it is an empirical question, but what does it mean to act without having perfect confidence in your own utility function?
It is important to be rational in such cases. But ‘situations where rationality is important’ isn’t the same set as ‘situations that are good didactic tools for rationality’. I mean, this is basically the central point of Politics is the Mind Killer: “What on Earth was the point of choosing this as an example? To rouse the political emotions of the readers and distract them from the main question?”
Charlie Hebdo wasn’t brought up as the topic, it was brought up as an example of a system that could have been demonstrated with a lot less baggage.
I downvoted because if you’re going to try to practice rationality or hone your understanding of your own biases, emotionally charged current events are a very bad domain to play around in. It’s like trying to learn the basics of Newtonian physics by studying Theo Jansen’s sculptures. There’s a giant tangle of stupid-inducing factors in this case, and a well chosen toy problem would be able to address the same subject without being likely to inadvertently flip those switches.
It’s rarely updated these days, but Quiet Babylon has some very rewarding material. I particularly suggest the ‘sequence’ on architects and cyborgs.
Who’s to say that evil isn’t a substance? Or at least, couldn’t be? It seems perfectly reasonable to write a story in which that map and the territory are not wholly distinct (and of course, even in the real world, maps are ultimately made of atoms...)
The real problem with much of the modern Extruded Fantasy Product is that it doesn’t deal creatively with the implications of its own claims and genre tropes. They allow evil to be a substance, but then use that to justify certain patterns of storytelling rather than actually treating evil like a substance. If you are dissatisfied with that, then you might enjoy fantasy roleplaying games like D&D, where you can construct your own narratives around the assumptions of a fantasy setting.
For example, you might have a village that casts ‘detect evil’ spells on every newborn infant, and kills all evil babies through exposure- thus creating a harmonious society of only good people. Or a whole metropolis of extremely weak gods that exist by enforcing a quota of five believers per god, and explore the interpersonal relationships between the weak gods and their private family of believers. Perhaps a whodunit, in which the players must track down a murderer who is spreading atheism.