I think some of your questions here are answered in Greg Egan’s story Crystal Nights and in Jake’s simbox post. We can have programming without mention of computers. It can be based on things like a fictional system of magic.
Thank you! It was valuable to read Crystal Nights and the simbox post gave me new insights and I have made a lot of updates thanks to these reading tips. I would think it to be a lot safer to not go for a fictional system of magic that lets it program. I estimate it would greatly increase the chance it thinks it is inside a computer and gives a lot of clues about perhaps being inside a simulation to test it, which we want to prevent. I would say, first see if it passes the non-programming simbox. If it does not, great, we found an alignment technique that does not work. Then after that, then you can think of doing a run with programming. I do realize these runs can cost hundreds of millions of dollars, but not going extinct is worth the extra caution, I would say. What do you think?
I think some of your questions here are answered in Greg Egan’s story Crystal Nights and in Jake’s simbox post. We can have programming without mention of computers. It can be based on things like a fictional system of magic.
Thanks!
These are very useful references.
Thank you! It was valuable to read Crystal Nights and the simbox post gave me new insights and I have made a lot of updates thanks to these reading tips. I would think it to be a lot safer to not go for a fictional system of magic that lets it program. I estimate it would greatly increase the chance it thinks it is inside a computer and gives a lot of clues about perhaps being inside a simulation to test it, which we want to prevent. I would say, first see if it passes the non-programming simbox. If it does not, great, we found an alignment technique that does not work. Then after that, then you can think of doing a run with programming. I do realize these runs can cost hundreds of millions of dollars, but not going extinct is worth the extra caution, I would say. What do you think?
I agree, but I do see the high cost as a weakness of the plan. For my latest ideas on this, see here: https://ai-plans.com/post/2e2202d0dc87