What code (short of a full-functioning AGI) would be at all useful here?
Possible experiments could include:
Simulate Prisoner’s Dilemma agents that can run each others’ code. Add features to the competition (e.g. group identification, resource gathering, paying a cost to improve intelligence) to better model a mix of humans and AIs in a society. Try to simulate what happens when some agents gain much more processing power than others, and what conditions make this a winning strategy. If possible, match results to real-world examples (e.g. competition between people with different education backgrounds). Based on these results, make a prediction of the returns to increasing intelligence for AIs.
Create an algorithm for a person to follow recommendations from information systems—in other words, write a flowchart that would guide a person’s daily life, including steps for looking up new information on the Internet and adding to the flowchart. Try using it. Compare the effectiveness of this approach with a similar approach using information systems from 10 years ago, and from 100 years ago (e.g. books). Based on these results, make a prediction for how quickly machine intelligence will become more powerful over time.
Identify currently-used measures of machine intelligence, including tests normally used to measure humans. Use Moore’s Law and other data to predict the rate of intelligence increase using these measures. Make a prediction for how machine intelligence changes with time.
Write an expert system for making philosophical statements about itself.
In general, when presenting a new method or applied theory, it is good practice to provide the most convincing data possible—ideally experimental data or at least simulation data of a simple application.
having trouble predicting should be a reason to if anything be more worried rather than less.
You’re right—I am worried about the future, and I want to make accurate predictions, but it’s a hard problem, which is no excuse. I hope you succeed in predicting the future. I assume your goal is to make a general prediction theory to accurately assign probabilities to future events, e.g. an totalitarian AI appearing. I’m trying to say that your theory will need to accurately model past false predictions as well as past true predictions.
The concern is that the first true AGI will self-modify itself to become far smarter and more capable of controlling the >environment around it than anything else.
I agree that is a possible outcome. I expect multiple AIs with comparable strength to appear at the same time, because I imagine the power of an AI depends primarily on its technology level and its access to resources. I expect multiple AIs (or a mix of AIs and humans) will cooperate to prevent one agent from obtaining a monopoly and destroying all others, as human societies have often done (especially recently, but not always). I also expect AIs will stay at the same technology level because it’s much easier to steal a technology than to initially discover it.
That sounds exciting too. I don’t know enough about this field to get into a debate about whether to save the metaphorical whales or the metaphorical pandas first. Both approaches are complicated. I am glad the MIRI exists, and I wish the researchers good luck.
My main point re: “steel-manning” the MIRI mission is that you need to make testable predictions and then test them or else you’re just doing philosophy and/or politics.
Make it scientific articles instead. Thus MIRI will get more publications. :D
I suspect that either would be of sufficient interest that if well done it could get published. Also, there’s a danger in going down research avenues simply because they are more publishable.
You can also make different expect systems compete with each other by trying to get most publications and citations.
So instead o f paper clip maximizers we end up with a world turned into researchpapertronium?
(This last bit is a joke- I think your basic idea is sound.)
Thanks for the thoughtful reply!
Possible experiments could include:
Simulate Prisoner’s Dilemma agents that can run each others’ code. Add features to the competition (e.g. group identification, resource gathering, paying a cost to improve intelligence) to better model a mix of humans and AIs in a society. Try to simulate what happens when some agents gain much more processing power than others, and what conditions make this a winning strategy. If possible, match results to real-world examples (e.g. competition between people with different education backgrounds). Based on these results, make a prediction of the returns to increasing intelligence for AIs.
Create an algorithm for a person to follow recommendations from information systems—in other words, write a flowchart that would guide a person’s daily life, including steps for looking up new information on the Internet and adding to the flowchart. Try using it. Compare the effectiveness of this approach with a similar approach using information systems from 10 years ago, and from 100 years ago (e.g. books). Based on these results, make a prediction for how quickly machine intelligence will become more powerful over time.
Identify currently-used measures of machine intelligence, including tests normally used to measure humans. Use Moore’s Law and other data to predict the rate of intelligence increase using these measures. Make a prediction for how machine intelligence changes with time.
Write an expert system for making philosophical statements about itself.
In general, when presenting a new method or applied theory, it is good practice to provide the most convincing data possible—ideally experimental data or at least simulation data of a simple application.
You’re right—I am worried about the future, and I want to make accurate predictions, but it’s a hard problem, which is no excuse. I hope you succeed in predicting the future. I assume your goal is to make a general prediction theory to accurately assign probabilities to future events, e.g. an totalitarian AI appearing. I’m trying to say that your theory will need to accurately model past false predictions as well as past true predictions.
I agree that is a possible outcome. I expect multiple AIs with comparable strength to appear at the same time, because I imagine the power of an AI depends primarily on its technology level and its access to resources. I expect multiple AIs (or a mix of AIs and humans) will cooperate to prevent one agent from obtaining a monopoly and destroying all others, as human societies have often done (especially recently, but not always). I also expect AIs will stay at the same technology level because it’s much easier to steal a technology than to initially discover it.
Make it scientific articles instead. Thus MIRI will get more publications. :D
You can also make different expect systems compete with each other by trying to get most publications and citations.
That sounds exciting too. I don’t know enough about this field to get into a debate about whether to save the metaphorical whales or the metaphorical pandas first. Both approaches are complicated. I am glad the MIRI exists, and I wish the researchers good luck.
My main point re: “steel-manning” the MIRI mission is that you need to make testable predictions and then test them or else you’re just doing philosophy and/or politics.
I suspect that either would be of sufficient interest that if well done it could get published. Also, there’s a danger in going down research avenues simply because they are more publishable.
So instead o f paper clip maximizers we end up with a world turned into researchpapertronium?
(This last bit is a joke- I think your basic idea is sound.)