There’s a gap between the general applicability of utility functions in theory, and their general inapplicability in practice. Indeed, there’s a general gap between theory and practice.
I would argue that this gap is a reason to do FAI research in a practical way—writing code, building devices, performing experiments. Dismissing gritty practicality as “too risky” or “not relevant yet” (which is what I hear SIAI doing) seems to lead to becoming a group without experience and skill at executing practical tasks.
Disclaimer: I’m aware that many FAI enthusiasts fall into the “Striving with all my hacker strength to build a self-improving friendly AI is FAI research, right?” error. That’s NOT what I’m advocating.
MBlume’s article “Put It To The Test” is pretty much what I have in mind.
If you think you understand a decision theory, can you write a test suite for an implementation of it? Can your test suite pass a standard implementation, and fail mutations of that standard implementation? Can you implement it? Is the performance of your implementation within a factor of ten-thousand of the standard implementation? Is it competitive? Can you improve the state of the art?
If you believe that the safe way to write code is to spend a long time in front of whiteboards, getting the design right, and then only a very short time developing (using a few high-IQ programmers) - How many times have you built projects according to this development process? What is your safety record? How does it compare to other development processes?
If you believe that writing machine-checkable proofs about code is important—Can you download and install one of the many tools (e.g. Coq) for writing proofs about code? Can you prove anything correct? What projects have you proved correct? What is their safety record?
What opportunities have you given reality to throw wrenches into your ideas—how carefully have you looked for those wrenches?
Any such “experiments” that allow for effective outbound communication from a proto-AI seem unacceptably risky. I’m curious what you think of the “oh crap, what if it’s right?” scenario I commented on over on the AI box post.
I didn’t SAY try to build a self-improving AI! That’s what the disclaimer was for!
Also, your claim of “unacceptably risky” needs actual arguments and reasoning to support it. As I see it, the only choice that is clearly unacceptably risky is inaction. Carefully confining your existential risk reduction activity to raising awareness about potential AI risks isn’t in any sense safe- for example, it could easily cause more new uFAI projects than it prevents.
Raising awareness about the problem isn’t just about getting would-be uFAI’ers to mend their sinful ways, you know. It’s absolutely necessary if you’re convinced you need help with it. As you said, inaction is untenable. If you’re certain that a goal of this magnitude is basically impossible given the status quo, taking some initial risks is a trivial decision. It doesn’t follow that additional risks share the same justification.
I’m also not convinced we understand the boundaries between “intelligent” and “self-improving” well enough to assume we can experiment with one and not the other. What sort of “practical tasks” do you have in mind that don’t involve potentially intelligent information-processing systems, and why do you think they’ll be at all relevant to the “real” work ahead?
There’s a gap between the general applicability of utility functions in theory, and their general inapplicability in practice. Indeed, there’s a general gap between theory and practice.
I would argue that this gap is a reason to do FAI research in a practical way—writing code, building devices, performing experiments. Dismissing gritty practicality as “too risky” or “not relevant yet” (which is what I hear SIAI doing) seems to lead to becoming a group without experience and skill at executing practical tasks.
Disclaimer: I’m aware that many FAI enthusiasts fall into the “Striving with all my hacker strength to build a self-improving friendly AI is FAI research, right?” error. That’s NOT what I’m advocating.
What sort of code, devices, experiments do you have in mind?
MBlume’s article “Put It To The Test” is pretty much what I have in mind.
If you think you understand a decision theory, can you write a test suite for an implementation of it? Can your test suite pass a standard implementation, and fail mutations of that standard implementation? Can you implement it? Is the performance of your implementation within a factor of ten-thousand of the standard implementation? Is it competitive? Can you improve the state of the art?
If you believe that the safe way to write code is to spend a long time in front of whiteboards, getting the design right, and then only a very short time developing (using a few high-IQ programmers) - How many times have you built projects according to this development process? What is your safety record? How does it compare to other development processes?
If you believe that writing machine-checkable proofs about code is important—Can you download and install one of the many tools (e.g. Coq) for writing proofs about code? Can you prove anything correct? What projects have you proved correct? What is their safety record?
What opportunities have you given reality to throw wrenches into your ideas—how carefully have you looked for those wrenches?
Any such “experiments” that allow for effective outbound communication from a proto-AI seem unacceptably risky. I’m curious what you think of the “oh crap, what if it’s right?” scenario I commented on over on the AI box post.
I didn’t SAY try to build a self-improving AI! That’s what the disclaimer was for!
Also, your claim of “unacceptably risky” needs actual arguments and reasoning to support it. As I see it, the only choice that is clearly unacceptably risky is inaction. Carefully confining your existential risk reduction activity to raising awareness about potential AI risks isn’t in any sense safe- for example, it could easily cause more new uFAI projects than it prevents.
Raising awareness about the problem isn’t just about getting would-be uFAI’ers to mend their sinful ways, you know. It’s absolutely necessary if you’re convinced you need help with it. As you said, inaction is untenable. If you’re certain that a goal of this magnitude is basically impossible given the status quo, taking some initial risks is a trivial decision. It doesn’t follow that additional risks share the same justification.
I’m also not convinced we understand the boundaries between “intelligent” and “self-improving” well enough to assume we can experiment with one and not the other. What sort of “practical tasks” do you have in mind that don’t involve potentially intelligent information-processing systems, and why do you think they’ll be at all relevant to the “real” work ahead?