offering people problems in mathematical logic is a good way to get people to work on Friendly AI problems
Mathematical logic problems are FAI problems. How are we going to build something self-improving that can reason correctly without having a theory of what “reasoning correctly” (ie logic) even looks like?
I think that the mathematical logic work is pretty far removed from the sort of work that will be needed for friendliness.
Based on what?
I’ll admit I don’t know that I would settle on mathematical logic as an important area of work, but EY being quite smart, working on this for ~10 years, and being able to convince quite a few people who are in a position to judge on this is good confirmation of the plausible idea that work in reflectivity of formal systems is a good place to be.
If you do have some domain knowledge that I don’t have that makes stable reflectivity seem less important and puts you in a position to disagree with an expert (EY), please share.
I believe that people who are interested in AI safety will not forget about AI safety entirely, independently of whether they have good problems to work on now.
People can get caught in other things. Maybe without something to work on now, they get deep into something else and build their skills in that and then the switching costs are too high to justify it. Mind you there is a steady stream of smart people, but opportunity costs.
Also, MIRI may be burning reputation capital by postponing actual work such that there may be less interested folks in the future. This could go either way, but it’s a risk that should be accounted for.
(I for one (as a donor and wannabe contributor) appreciate that MIRI is getting these (important-looking) problems available to the rest of us now)
I believe that people outside of MIRI will organically begin to work on AI safety without MIRI’s advocacy when AI is temporally closer.
How will they tell? What if it happens too fast? What if the AI designs that are furthest along are incompatible with stable reflection? Hence MIRI working on stategic questions like “how close are we, how much warning can we expect” (Intelligence Explosion Microecon), and “What fundamental architectures are even compatible with friendliness” (this Lob stuff).
Mathematical logic problems are FAI problems. How are we going to build something self-improving that can reason correctly without having a theory of what “reasoning correctly” (ie logic) even looks like?
See my responses to paper-machine on this thread for (some reasons) why I’m questioning the relevance of mathematical logic.
I’ll admit I don’t know that I would settle on mathematical logic as an important area of work, but EY being quite smart, working on this for ~10 years, and being able to convince quite a few people who are in a position to judge on this is good confirmation
I don’t see this as any more relevant than Penrose’s views on consciousness, which I recently discussed. Yes, there are multiple people who are convinced, but their may be spurious correlations which are collectively driving their interests. Some that come to mind are
Subject-level impressiveness of Eliezer.
Working on these problems offering people a sense of community.
Being interested in existential risk reduction and not seeing any other good options on the table for reducing existential risk.
Intellectual interestingness of the problems.
Also, I find Penrose more impressive than all of the involved people combined. (This is not intended as a slight – rather, the situation is that Penrose’s accomplishments are amazing.)
of the plausible idea that work in reflectivity of formal systems is a good place to be.
The idea isn’t plausible to me, again, for reasons that I give in my responses to paper-machine (among others).
If you do have some domain knowledge that I don’t have that makes stable reflectivity seem less important and puts you in a position to disagree with an expert (EY), please share.
No, my reasons are at a meta-level rather than an object level, just as most members of the Less Wrong community (rightly) believe that Penrose’s views on consciousness are very likely wrong without having read his arguments in detail.
People can get caught in other things. Maybe without something to work on now, they get deep into something else and build their skills in that and then the switching costs are too high to justify it. Mind you there is a steady stream of smart people, but opportunity costs.
This is possible, but I don’t think that it’s a major concern.
(I for one (as a donor and wannabe contributor) appreciate that MIRI is getting these (important-looking) problems available to the rest of us now)
Note this as a potential source of status quo bias.
How will they tell? What if it happens too fast?
Place yourself in the shoes of the creators of the early search engines, online book store, and social network websites. If you were in their positions, would you feel justified in concluding “if we don’t do it then no one else will”? If not, why do you think that AI safety will be any different?
I agree that it’s conceivable that it could happen to fast, but I believe that there’s strong evidence that it won’t happen within the next 20 years, and 20 years is a long time for people to become interested in AI safety.
Mathematical logic problems are FAI problems. How are we going to build something self-improving that can reason correctly without having a theory of what “reasoning correctly” (ie logic) even looks like?
Based on what?
I’ll admit I don’t know that I would settle on mathematical logic as an important area of work, but EY being quite smart, working on this for ~10 years, and being able to convince quite a few people who are in a position to judge on this is good confirmation of the plausible idea that work in reflectivity of formal systems is a good place to be.
If you do have some domain knowledge that I don’t have that makes stable reflectivity seem less important and puts you in a position to disagree with an expert (EY), please share.
People can get caught in other things. Maybe without something to work on now, they get deep into something else and build their skills in that and then the switching costs are too high to justify it. Mind you there is a steady stream of smart people, but opportunity costs.
Also, MIRI may be burning reputation capital by postponing actual work such that there may be less interested folks in the future. This could go either way, but it’s a risk that should be accounted for.
(I for one (as a donor and wannabe contributor) appreciate that MIRI is getting these (important-looking) problems available to the rest of us now)
How will they tell? What if it happens too fast? What if the AI designs that are furthest along are incompatible with stable reflection? Hence MIRI working on stategic questions like “how close are we, how much warning can we expect” (Intelligence Explosion Microecon), and “What fundamental architectures are even compatible with friendliness” (this Lob stuff).
See my responses to paper-machine on this thread for (some reasons) why I’m questioning the relevance of mathematical logic.
I don’t see this as any more relevant than Penrose’s views on consciousness, which I recently discussed. Yes, there are multiple people who are convinced, but their may be spurious correlations which are collectively driving their interests. Some that come to mind are
Subject-level impressiveness of Eliezer.
Working on these problems offering people a sense of community.
Being interested in existential risk reduction and not seeing any other good options on the table for reducing existential risk.
Intellectual interestingness of the problems.
Also, I find Penrose more impressive than all of the involved people combined. (This is not intended as a slight – rather, the situation is that Penrose’s accomplishments are amazing.)
The idea isn’t plausible to me, again, for reasons that I give in my responses to paper-machine (among others).
No, my reasons are at a meta-level rather than an object level, just as most members of the Less Wrong community (rightly) believe that Penrose’s views on consciousness are very likely wrong without having read his arguments in detail.
This is possible, but I don’t think that it’s a major concern.
Note this as a potential source of status quo bias.
Place yourself in the shoes of the creators of the early search engines, online book store, and social network websites. If you were in their positions, would you feel justified in concluding “if we don’t do it then no one else will”? If not, why do you think that AI safety will be any different?
I agree that it’s conceivable that it could happen to fast, but I believe that there’s strong evidence that it won’t happen within the next 20 years, and 20 years is a long time for people to become interested in AI safety.