OK I take your point. In your opinion would this be an improvement “Humans have never completed at large scale engineering task without at least one mistake on the first attempt”?
For the argument with AI, will the process that is used to make current AI scale to AGI level? From what I understand that is not the case. Is that predicted to change?
Thank you for giving feedback.
But isn’t that also using a reward function? The AI is trying to maximise the reward it receives from the Reward Model. The Reward Model that was trained using Human Feedback.
Hi Edward, I can estimate you personally care about censorship, and outside the field of advanced AI that seems like a valid opinion. You are right that humans keep each other aligned by mass consensus. When you read more about AI you will be able to see that this technique no longer works for AI. Humans and AI are different.
Having AI alignment is a strongly supported opinion in this community and is also supported by many people outside this community as well. This is link is an open letter where a range of noteworthy people talk about the dangers of AI and how alignment may help. I recommend you give it a read. Pause Giant AI Experiments: An Open Letter—Future of Life Institute
AI risk is an emotionally challenging topic, but I believe that you can find the way to understand it more.
Yea, I have to say many of my opinions on AI are from Eliezer. I have read much of his work and compared it to the other expects I have read about and talked with, and I have to say, he seems to understand the problem very well.
I agree, aligned AGI seems like a very small island in the sea of possibly. If we have multiple tries at getting it right (for example with AGI in a perfectly secure simulation), I think we have a chance. But with only effectively one try, the probability of success seems astronomically low.
Which x-risks do you think AI will reduce? I have heard arguments that it would improve our ability to respond to potential asteroid impacts. However this reduction in x-risk seems very small in comparison to the x-risk that unaligned AGI poses. What makes you estimate that AI may reduce x-risk?
How would an AI be directed without using a reward function? Are there some examples I can read?
Okay, how much risk is worth the benefit? Would you advocate for a comparison of expected gains and expected losses?
For example if humanity wanted a 50% chance of surviving another 10 years in the presence of a meta-stably-aligned ASI, the ASI would need a daily non-failure rate of x^(365.25*10)=0.5, x=0.99981 or better.
Deep Learning is notorious for continuing to work while there are mistakes in it; you can accidentally leave all kinds of things out and it still works just fine. There are of course arguments that value is fragile and that if we get it 0.1% wrong then we lose 99.9% of all value in the universe, just as there are arguments that the aforementioned arguments are quite wrong. But “one mistake” = “failure” is not a firm principle in other areas of engineering, so it’s unlikely to be the case.
Ok the “An AGI that has at least one mistake in its alignment model will be unaligned” premise seems like the weakest one. Is there any agreement in the AI community about how much alignment is “enough”? I suppose it depends on the AI capabilities and how long you want to have it running for. Are there any estimates?
It’s a feature of human cognition that we can make what feel like good arguments for anything.
I would tend to agree, the confirmation bias is profound in humans.
you instead asked “hey, what would AI doom predict about the world, and are those predictions coming true?”
Are suggesting a process such as: assume a future → predict what would be required for this → compare prediction with reality.
Rather than: observe reality → draw conclusions → predict the future
Thank you for your reply 1a3orn, I will have a read over some of the links you posted.
Thank you David Mears for writing this linkpost. Thank you to Philosophy Bear for writing the original post.
I agree that this survey is urgently needed.
I have had a similar issue. The model outputs the comment:
Incorrect API key provided: sk-5LW5C***************************************HGYd. You can find your API key at https://platform.openai.com/account/api-keys.
Disagree vote this post if you disagree that the topic of predicting AI development is challenging.
Disagree vote this post if you disagree with liking the simplicity of the original post.
The above reply has two disagreement votes. I am trying to discern which reasons they are for. Disagree vote this post if you disagree that Mitchell_Porters suggestion was helpful.
For those who are downvoting this post: A short one sentence comment will help the original poster make better articles in the future.
I just read the abstract. Storing information in momentum makes a lot of sense as we know it is a conserved quantity. Practically challenging. But yes, this does move the theoretical limit even further away from all humans combined.