The reason why we want editing for IQ is because we want something unusual like “+1SD above von Neumann”, I’m not sure we want something beyond statistical range of human personality traits. Why do not select outliers from population using personality testing and give them high intelligence?
quetzal_rainbow
Karma: 1,748
I think the simplest objection to your practical CF part is that (here and further, phenomenal) consciousness is physiologically and evolutionarily robust: an infinite number of factors can’t be required to have consciousness because the probability of having an infinite number of factors right is zero.
On the one hand, we have evolutionary robustness: it seems very unlikely that any single mutation could cause Homo sapiens to become otherwise intellectually capable zombies.
You can consider two extreme possibilities. Let’s suppose that Homo sapiens is conscious and Homo erectus isn’t. Therefore, there must be a very small number of structural changes in the brain that cause consciousness among a very large range of organisms (different humans), and ATP is not included here, as both Homo sapiens and Homo erectus have ATP.
Consider the opposite situation: all organisms with a neural system are conscious. In that case, there must be a simple property (otherwise, not all organisms in the range would be conscious) common among neural systems causing consciousness. Since neural systems of organisms are highly diverse, this property must be something with a very short description.
For everything in between: if you think that hydras don’t have consciousness but proconsuls do, there must be a finite change in the genome, mRNAs, proteins, etc., between a hydra egg and a proconsul egg that causes consciousness to appear. Moreover, this change is smaller than the overall distance between hydras and proconsuls because humans (descendants of proconsuls) have consciousness too.
From a physiological point of view, there is also extreme robustness. You need to be hit in the head really hard to lose consciousness, and you preserve consciousness under relatively large ranges of pH. Hemispherectomy often doesn’t even lead to cognitive decline. Autism, depression, and schizophrenia are associated with significant changes in the brain, yet phenomenal consciousness still appears to be here.
EDIT: in other words, imagine that we have certain structure in Homo sapiens brain absent in Homo erectus brain which makes us conscious. Take all possible statements distinguishing this structure from all structures in Homo erectus brain. If we exclude all statements, logically equivalent to “this structure implements such-n-such computation”, we are left… exactly with what? We are probably left with something like “this structure is a bunch of lipid bubbles pumping sodium ions in certain geometric configuration” and I don’t see any reasons for ion pumping in lipid bubbles to be relevant to phenomenal consciousness, even if it happens in a fancy geometric configuration.
I meant by “authority” not brainwashing but, like, the fact that AGIs are smarter and know better: “If you believe you can survive without taking over the world, please consult the graph”. If China hawks believe themselves to be correct, they expect AGIs to actually prove that.
I agree this scenario is terrible.
I think that you miss possibility of leveraging AGI authority here. I.e., if your military/natsec-aligned AI will scream at your superiors “ATTACK NOW YOU FOOLS”, maybe your superiors will be more inclined to listen to plans to melt all GPUs.
Read your comment, donated 10$.
For me, “enjoying life” means “enjoying good things that happen in life”, while everything in-between is neutral at best. Many people seem to put positive value on “in-between” space.
If I try to point out, some people enjoy just the fact of being embodied.
After yet another news about decentralized training of LLM, I suggest to declare assumption “AGI won’t be able to find hardware to function autonomously” outdated.
My example is when people say “I enjoy life” they mean actually enjoying life-as-whole and not like “I’m glad my life is net-positive” or whatever.
Okay, I kinda understood where I am wrong spiritually-intuitively, but now I don’t understand where I’m wrong formally. Like which inference in chain
not Consistent(ZFC) → some subsets of ZFC don’t have a model → some subsets of ZFC + not Consistent(ZFC) don’t have a model → not Consistent(ZFC + not Consistent(ZFC))
is actually invalid?
Completeness theorem states that consistent countable FO theory has a model. Compactness theorem states that FO theory has a model iff every finite subset of FO theory has a model. Both theorems are provable in ZFC.
Therefore:
Consistent(ZFC) <-> all finite subsets of ZFC have a model ->
not Consistent(ZFC) <-> some finite subsets of ZFC don’t have a model ->
some finite subsets of ZFC + not Consistent(ZFC) don’t have a model <->
not Consistent(ZFC + not Consistent(ZFC)),
proven in ZFC + not Consistent(ZFC)
You are making an error here:
ZFC + not Consistent(ZFC)!=ZFC.Assuming
ZFC + not Consistent(ZFC)we can proveConsistent(ZFC), because inconsistent systems can prove everything andZFC + not Consistent(ZFC) + Consistent(ZFC)is, in fact, inconsistent. But it doesn’t say anything about consistency of ZFC itself, because you can freely assume any sufficiently powerful system instead of ZFC. If you assume inconsistent system, thensystem + not Consistent(system)is still inconsistent, if you assume consistent system, thensystem + not Consistent(system)is inconsistent for reasoning above, so it can’t prove whether assumed system is consistent or not.
There are no properties of brain which define that brain is “you”, except for the program that it runs.
I agree with your technical points, but I don’t think that we could particularly expect the other path. Safety properties of LLMs seem to be desirable from extremely safety-pilled point of view, not from perspective of average capabilities researcher and RL seems to be The Answer to many learning problems.
I agree that lab leaders are not in much better position, I just think that lab leaders causally screen off influence of subordinates, while incentives in the system causally screens off lab leaders.
It’s just no free lunch theorem? For every computable decision procedure you can construct environment which predicts exact output for this decision procedure and reacts in way of maximum damage, making decision procedure to perform worse than random action selection.
The nice thing about being a coward is that once you notice you can just stop.
- Eliezer Yudkowsky, lintamande, Planecrash, the woman of irori
I’m not sure median researcher is particularly important here, relatively to, say, median lab leader.
Median voter theorem works explicitly because votes of everyone are equal, but if you have lab/research group leader who disincentivizes bad research practices, then you theoretically should get lab with good research practices.
In practice, lab leaders are often people who Goodhart incentives, which results in current situation.
LessWrong has chance to be better exactly because it is outside of current system of perverse incentives. Although, it has its own bad incentives.
In effect, Omega makes you kill people by sending message.
Imagine two populations of agents, Not-Pull and Pull. 100% members of Not-Pull receive the message, don’t pull and kill one person. In Pull population 99% members do not get the message, pull and get zero people killed, 1% receive message, pull and in effect kill 5 people. Being member of Pull population has 0.05 expected casualties and being member of Not-Pull population has 1 expected casualty. Therefore, you should pull.
Wow, XOR-trolley really made me think
In fact, speed and accuracy in humans are at least somewhat mechanistically different