If Ilya was willing to cooperate, the board could fire Altman, with the Thanksgiving break available to aid the transition, and hope for the best.
Alternatively, the board could choose once again not to fire Altman, watch as Altman finished taking control of OpenAI and turned it into a personal empire, and hope this turns out well for the world.
Could they not have also gone with option 3: fill the vacant board seats with sympathetic new members, thus thwarting Altman’s power play internally?
Alternative framing: The board went after Altman with no public evidence of any wrongdoing. This appears to have backfired. If they had proof of significant malfeasance, and presented it to their employees, the story may have gone a lot differently.
Applying this to the AGI analogy would be be a statement that you can’t shut down an AGI without proof that it is faulty or malevolent in some way. I don’t fully agree though: I think if a similar AGI design had previously done a mass murder, people would be more willing to hit the off switch early.
Civilization involves both nice and mean actions. It involves people being both nice and mean to each other.
From this perspective, if you care about Civilization, optimizing solely for niceness is as meaningless and ineffective as optimizing for meanness.
Who said anything about optimizing solely for niceness? Everyone has many different values that sometimes conflict with each other, that doesn’t mean that “niceness” shouldn’t be one of them. I value “not killing people”, but I don’t optimize solely for that: I would still kill Mega-Hitler if I had the chance.
Would you rather live in a society that valued “niceness, community and civilization”, or one that valued “meanness, community and civilization”? I don’t think it’s a tough choice.
I think that being mean is sometimes necessary in order to preserve other, more important values, but that doesn’t mean that you shouldn’t be nice, all else being equal.
Partially, but it is still true that Eliezer was critical of NN’s at the time, see the comment on the post:
I’m no fan of neurons; this may be clearer from other posts.
“position” is nearly right. The more correct answer would be “position of one photon”.
If you had two electrons, say, you would have to consider their joint configuration. For example, one possible wavefunction would look like the following, where the blobs represent high amplitude areas:
This is still only one dimensional: the two electrons are at different points along a line. I’ve entangled them, so if electron 1 is at position P, electron 2 can’t be.
Now, try and point me to where electron 1 is on the graph above.
You see, I’m not graphing electrons here, and neither were you. I’m graphing the wavefunction. This is where your phrasing seems a little weird: you say the electron is the collection of amplitudes you circled: but those amplitudes are attached to configurations saying “the electron is at position x1” or “the electron is at position x2″. It seems circular to me. Why not describe that lump as “a collection of worlds where the electron is in a similar place”?
If you have N electrons in a 3d space, the wavefunction is not a vector in 3d space (god I wish, it would make my job a lot easier). It’s a vector in 3N+1 dimensions, like the following:
where r1, r2, etc are pointing to the location of electron 1, 2, 3, etc, and each possible configuration of electron 1 here, electron 2 there, etc, has an amplitude attached, with configurations that are more often encountered experimentally empirically having higher amplitudes.
Nice graph!
But as a test, may I ask what you think the x-axis of the graph you drew is? Ie: what are the amplitudes attached to?
I’m not claiming the conceptual boundaries I’ve drawn or terminology I’ve used in the diagram above are standard or objective or the most natural or anything like that. But I still think introducing probabilities and using terminology like “if you now put a detector in path A , it will find a photon with probability 0.5” is blurring these concepts together somewhat, in part by placing too much emphasis on the Born probabilities as fundamental / central.
I think you’ve already agreed (or at least not objected to) saying that the detector “found the photon” is fine within the context of world A. I assume you don’t object to me saying that I will find the detector flashing with probability 0.5. And I assume you don’t think me and the detector should be treated differently. So I don’t think there’s any actual objection left here, you just seem vaguely annoyed that I mentioned the empirical fact that amplitudes can be linked to probabilities of outcomes. I’m not gonna apologise for that.
Okay, let me break in down in terms of actual states, and this time, let’s add in the actual detection mechanism, say an electron in a potential well. Say the detector is in the ground state energy, E=0, and the absorption of a photon will bump it up to the next highest state, E=1. We will place this detector in path A, but no detector in path B.
At time t = 0, our toy wavefunction is:
1/sqrt2 |photon in path A, detector E=0> + 1/sqrt2 |photon in path B, detector E=0>
If the photon in A collides with the detector at time t =1, then at time t=2, our evolved wavefunction is:
1/sqrt2 |no free photon, detector E=1> + 1/sqrt2 |photon in path B, detector E=0>
Within the context of world A, a photon was found by the detector. This is a completely normal way to think and talk about this.
I think it’s straight up wrong to say “the photon is in the detector and in path B”. Nature doesn’t label photons, and it doesn’t distinguish between them. And what is actually in world A is an electron in a higher energy state: it would be weird to say it “contains” a photon inside of it.
Quantum mechanics does not keep track of individual objects, it keeps track of configurations of possible worlds, and assigns amplitudes to each possible way of arranging everything.
What part of “finding a photon” implies that the photon is a billiard ball? Wave-particle duality aside, a photon is a quanta of energy: the detector either finds that packet or it doesn’t (or in many worlds, one branched detector finds it and the other branched detector doesn’t).
I’m interested to hear more about how you interpret the “realness” of different branches. Say there is an electron in one of my pinky fingers that is in a superposition of spin up and spin down. Are there correspondingly two me’s, one with with pinky electron up and one with pinky electron down? Or is there a single me, described by the superposition of pinky electrons?
I am assuming you are referring to the many worlds interpretation of quantum mechanics, where superpositions extend up to the human level, and the alternative configurations correspond to real, physical worlds with different versions of you that see different results on the detector.
Which is puzzling, because then why would you object to “the detector finding a photon”? The whole point of the theory is that detectors and humans are treated the same way. In one world, the detector finds the photon, and then spits out a result, and then one You sees the result, and in a different world, the detector finds the photon, spits out the other result, and a different result is seen. There is no difference between “you” and “it” here.
As for the photon “being” the complex amplitudes… That doesn’t sound right to me. Would you say that “you” are the complex amplitudes assigned to world 1 and world 2? It seems more accurate to say that there are two yous, in two different worlds (or many more).
Assuming you are a many worlder, may I ask which solution to the Born probabilities you favour?
I’m a little confused by what your objection is. I’m not trying to stake out an interpretation here, I’m describing the calculation process that allows you to make predictions about quantum systems. The ontology of the wavefunction is a matter of heated debate, I am undecided on it myself.
Would you object to the following modification:
If you now put a detector in path A , it will find a photon with probability ( ), and same for path B. If you repeated this experiment a very large number of times, the results would converge to finding it 50% of the time in the configuration |photon in path A only>, and 50% of the time in the configuration |photon in path B only>. The arrow direction still has no effect on the probability.
Apologies for the late reply, but thank you for your detailed response.
Responding to your objection to my passage, I disagree, but I may edit it slightly to be clearer.
I was simply trying to point out the empirical fact that if you put a detector in path A and a detector in path B, and repeat the experiment a bunch of times, you will find the photon in detector A 50% of the time, and the photon in detector B 50% of the time. If the amplitudes had different values, you would empirically find them in different proportions, as given by the squared amplitudes.
I don’t find these probabilities to be an “afterthought”. This is the whole point of the theory, and the reason we consider quantum physics to be “true”. We never see these amplitudes directly, we infer them from the fact that they give correct probabilities via the Born rule. Or more specifically, this is the formula that works. That this formula works is an empirical fact, all the interpretations and debate are a question of why this formula works.
Regarding the defense of the original sequence, I’m sorry, but incorrect math is incorrect math. The people who figured out the mistake in the comments figured it out from other sources. If anything, it is even more damning that people pointed the mistake out 10 years ago, and it still hasn’t been fixed. For every person who figured out the problem or sifted through hundreds of comments to figure out the issue, there are dozens more who accepted the incorrect framework, or decided they were too dumb to understand the math when it was the author who was wrong.
My problem is that the post is misinforming people. I will make no apology for being harsh about that.
I will restrain my opinion on Eliezers other quantum posts for a future post when I tackle the overstated case for many worlds theories.
Bayes can judge you now: your analysis is half-arsed, which is not a good look when discussing a matter as serious as this.
All you’ve done is provide one misleading statistic. The base rate of experiencing psychosis may be 1-3%, but the base rate of psychotic disorders is much lower, at 0.25% or so.
But the most important factor is one that is very hard to estimate, which is what percentage of people with psychosis manifest that psychosis as false memories of being groped by a sibling. If the psychosis had involved seeing space aliens, we would be having a different discussion.
We would then have to compare this with the rate of teenagers groping their toddler siblings. This is also very difficult. A few studies claim that somewhere around 20% of women are sexually abused as children, but I don’t have a breakdown of that by source of abuse and age, etc. Obviously the figure for our particular subset of assault cases will be significantly lower, but I don’t know by how much.
I thinks it’s highly likely that the number of women groped as a toddler by a sibling is much higher than the number of women who falsely claim to be groped as a toddler by a sibling as a result of psychosis or other mental illness, although again there is high uncertainty. This certainly makes intuitive sense: absent any other information, the most likely explanation for someone being accused of a specific crime is that they committed the crime.
All the further evidence seems at least consistent with either scenario
The sudden onset of the memory could be due to onset of psychosis… or it could be a repressed memory, which can also trigger at any time.
Suppose the claim about shadowbanning is false. That could be due to psychosis, or it could be a combination of misunderstanding technology and being fearful of an abusive sibling in the tech sector. I don’t think it’s strong evidence of psychosis in particular.
Is moving 20 times evidence of psychosis? The claimed reason is that she’s broke and had to rely on sex work for money. This seems orthogonal to psychosis.
Is using Zoloft evidence for being psychotic? Only weakly, since she was taking Zoloft for non-psychotic reasons since her teen years.
The claim about her dads money being withheld is only evidence for psychosis if it is false. This would be easy for sam altman to prove, and he hasn’t yet.
I don’t think there’s enough information to be truly certain of either side, but there is more than enough information to be concerned, and to want further investigation and evidence.
At no point did I ever claim that this was a conclusive debunking of AI risk as a whole, only an investigation into one specific method proposed by Yudkowksy as an AI death dealer.
In my post I have explained what DMS is, why it was proposed as a technology, how far along the research went, the technical challenges faced in it’s construction, some observations of how nanotech research works, the current state of nanotech research, what near-term speedups can be expected from machine learning, and given my own best guess on whether an AGI could pull off inventing MNT in a short timeframe, based on what was learned.
This is only “broadly non-sequiter” if you think that none of that information is relevant for assessing the feasibility of diamondoid bacteria AI weapons, which strikes me as somewhat ridiculous.
Note that the nearer side feeling colder than the farther side is be completely possible.
The key is that they didn’t check the temperatures of each side with a thermometer, but with their hands. And your hands don’t feel temperature directly, they feel heat conduction. If you have a cake and a baking tin that are the same temperature, the metal will feel hotter because it is more conductive.
If I wanted to achieve the effect described here without flipping trickery, I might make the side near the radiator out of a very nonconductive plastic (painted to look like metal), and the side further away out of a very thermally conductive metal. It seems entirely plausible to me that once everything is in thermal equilibrium, the plastic would feel cooler than the metal, despite having a higher temperature. Of course, this would depend on the actual temperatures, conductivities, etc involved. But in principle it seems completely possible.
Suddenly the physics students don’t look so foolish. In fact, in this scenario, both the heat conduction guy and the “strange metals in the plate” guy are on the right track! (even the convective guy could have a point, if that’s the primary method of heating the far side). They are acknowledging their confusion, while trying to suggest avenues of exploration that might reveal an aspect of the problem they didn’t think of. In this scenario, the Rationalist who dismisses these answers is the real fool, acting as the curiosity stopper, just because they didn’t think of how hands feel heat. (of course, the ideal reasoner would think of both “different metal” and “flipped plate” as hypotheses).
I think the analysis for “bomb” is missing something.
This is a scenario where the predictor is doing their best not to kill you: if they think you’ll pick left they pick right, if they think you’ll pick right they’ll pick left.
The CDT strategy is to pick whatever box doesn’t have a bomb in it. So if the player is a perfect CDTer, the predictor is 100% guaranteed to be correct in their pick. The predictor actually gets to pick whether the player loses 100 bucks or not. If the predictor is nice, the CDTer gets to walk away without paying anything and a 0% chance of death.
Eliezers response is not comprehensive. He responds to two points (a reasonable choice), but he responds badly, first with a strawman, second with an argument that is probably wrong.
The first point he argues is about brain efficiency, and is not even a point made by the OP. The OP was simply citing someone else, to show that “Eliezer is overconfident about my area of expertise” is an extremely common opinion. It feels very weird to attack the OP over citing somebody else’s opinion.
Regardless, Eliezer handles this badly anyway. Eliezer gives a one paragraph explanation of why brain efficiency is not close to tha Landauer limit. Except that If we look at the actual claim that is quoted, Jacob is not saying that it is at the limit, only that it’s not six orders of magnitude away from the limit, which was Eliezer’s original claim. So essentially he debunks a strawman position and declares victory. (I do not put any trust in Eliezers opinions on neuroscience)
When it comes to the zombies, I’ll admit to finding his argument fairly hard to follow. The accusation levelled against him, both by the OP and Chalmers, is that he falsely equates debunking epiphenomenalism with debunking the zombie argument as a whole.
Eliezer unambiguously does equate the two things, as proven by the following quote highlighted by the OP:
It seems to me that there is a direct, two-way logical entailment between “consciousness is epiphenomenal” and “zombies are logically possible”
The following sentence, from the comment, seems (to me) to be a contradiction of his earlier claim.
It’s not that I think philosophers openly claim that p-zombies demonstrate epiphenomenalism
The most likely explanation, to me, is that Eliezer made a mistake, the OP and Chalmers pointed it out, and then he tried to pretend it didn’t happen. I’m not certain this is what happened (as the zombies stuff is highly confusing), but it’s entirely in line with Eliezer’s behavior over the years.
I think Eliezer has a habit of barging into other peoples domains, making mistakes, and then refusing to be corrected by people that actually know what they are talking about, acting rude and uncharitable in the process.
Imagine someone came up to you on the street and claimed to know better than the experts in quantum physics, and nanoscience, and AI research, and ethics, and philosophy of mind, and decision theory, and economic theory, and nutrition, and animal consciousness, and statistics and philosophy of science, and epistemology and virology and cryonics.
What odds would you place on such a person being overconfident about their own abilities?
I would be interested in your actual defense of the first two sections. It seems the OP went to great lengths to explain exactly where Eliezer went wrong, and contrasted Eliezer’s beliefs with citations to actual, respected domain level experts.
I also do not understand your objection to the term “gross overconfidence”. I think the evidence provided by the OP is completely sufficient to substantiate this claim. In all three cases (and many more I can think of that are not mentioned here), Eliezer has stated things that are probably incorrect, and then dismissively attacked, in an incredibly uncharitable manner, people who believe the opposite claims. “Eliezer is often grossly overconfident” is, in my opinion, a true claim that has been supported with evidence. I do not think charitability requires one to self-censor such a statement.
Okay, I’m gonna take my skeptical shot at the argument, I hope you don’t mind!
It’s not true that whatever the AI tried to do would happen. What if an AI wanted to travel faster than the speed of light, or prove that 2+2=5, or destroy the sun within 1 second of being turned on?
You can’t just say “arbitrary goals”, you have to actually explain what goals there are that would be realistically achievable by an realistic AI that could be actually built in the near future. If those abilities fall short of “destroy all of humanity”, then there is no x-risk.
This is fictional evidence. Genies don’t exist, and if they did, it probably wouldn’t be that hard to add enough caveats to your wish to prevent global genocide. A counterexample might be the use of laws: sure, there are loopholes, but not big enough that the law would let you off on a broad daylight killing spree.
Well, there is laws of physics and maths that put limits on available computational power, which in turn puts a limit on what an AI can actually achieve. For example, a perfect Bayesian reasoner is forbidden by the laws of mathematics.