If I am reading you correctly, you are trying to build an incentive structure that will accelerate the development of AGI. Many alignment researchers (I am one) will tell you that this is not a good idea, instead you want to build an incentive structure that will accelerate the development of safety systems and alignment methods for AI and AGI.
There is a lot of open source production in the AI world, but you are right in speculating that a lot of AI code and know-how is never open sourced. Take a look at the self-driving car R&D landscape if you want to see this in action.
As already mentioned by Zac, for-profit companies release useful open source all the time for many self-interested reasons.
One reason not yet mentioned by Zac is that an open source release may be a direct attack to suck the oxygen our of the business model of one or more competitors, an attack which aims to commoditize the secret sauce (the software functions and know-how) that the competitor relies on to maintain profitability.
This motivation explains why Facebook started to release big data handling software and open source AI frameworks: they were attacking Google’s stated long-term business strategy, which relied on Google being better at big data and AI than anybody else. To make this more complicated, Google’s market power never relied as much on big data and advanced AI as it wanted its late-stage investors to believe, so the whole move has been somewhat of an investor story telling shadow war.
Personally, I am not a big fan of the idea that one might try to leverage crypto-based markets as a way to improve on this resource allocation mess.
I guess I got that impression from the ‘public good producers significantly accelerate the development of AGI’ in the title, and then looking at the impactcerts website.
I somehow overlooked the bit where you state that you are also wondering if that would be a good idea.
To be clear: my sense of the current AI open source space is that it definitely under-produces certain software components, software components that could be relevant for improving AI/AGI safety.
What are some of those [under-produced software] components? We can put them on a list.
Good question. I don’t have a list, just a general sense of the situation. Making a list would be a research project in itself. Also, different people here would give you different answers. That being said,
I occasionally see comments from alignment research orgs who do actual software experiments that they spend a lot of time on just building and maintaining the infrastructure to run large scale experiments. You’d have to talk to actual orgs to ask them what they would need most. I’m currently a more theoretical alignment researcher, so I cannot offer up-to-date actionable insights here.
As a theoretical researcher, I do reflect on what useful roads are not being taken, by industry and academia. One observation here is that there is an under-investment in public high-quality datasets for testing and training, and in the (publicly available) tools needed for dataset preparation and quality assurance. I am not the only one making that observation, see for example https://research.google/pubs/pub49953/ . Another observation is that everybody is working on open source ML algorithms, but almost nobody is working on open source reward functions that try to capture the actual complex details of human needs, laws, or morality. Also, where is the open source aligned content recommender?
On a more practical note, AI benchmarks have turned out to be a good mechanism for drawing attention to certain problems. Many feel that this benchmarks are having a bad influence on the field of AI, I have a lot of sympathy for that view, but you might also go with the flow. A (crypto) market that rewards progress on selected alignment benchmarks may be a thing that has value. You can think here of benchmarks that reward cooperative behaviour, truthfulness and morality in answers given by natural language querying systems, playing games ethically ( https://arxiv.org/pdf/2110.13136.pdf ), etc. My preference would be to reward benchmark contributions that win by building strong priors into the AI to guide and channel machine learning; many ML researchers would consider this to be cheating, but these are supposed to be alignment benchmarks, not machine-learning-from-blank-slate benchmarks. I have some doubts about the benchmarks for fairness in ML which are becoming popular, if I look at the latest NeurIPS: the ones I have seen offer tests which look a bit too easy, if the objective is to reward progress on techniques that have the promise of scaling up to more complex notions of fairness and morality you would like to have at the AGI level, or even for something like a simple content recommendation AI. Some cooperative behaviour benchmarks also strike me as being too simple, in their problem statements and mechanics, to reward the type of research that I would like to see. Generally, you would want to retire a benchmark from the rewards-generating market when the improvements on the score level out.
If I am reading you correctly, you are trying to build an incentive structure that will accelerate the development of AGI. Many alignment researchers (I am one) will tell you that this is not a good idea, instead you want to build an incentive structure that will accelerate the development of safety systems and alignment methods for AI and AGI.
There is a lot of open source production in the AI world, but you are right in speculating that a lot of AI code and know-how is never open sourced. Take a look at the self-driving car R&D landscape if you want to see this in action.
As already mentioned by Zac, for-profit companies release useful open source all the time for many self-interested reasons.
One reason not yet mentioned by Zac is that an open source release may be a direct attack to suck the oxygen our of the business model of one or more competitors, an attack which aims to commoditize the secret sauce (the software functions and know-how) that the competitor relies on to maintain profitability.
This motivation explains why Facebook started to release big data handling software and open source AI frameworks: they were attacking Google’s stated long-term business strategy, which relied on Google being better at big data and AI than anybody else. To make this more complicated, Google’s market power never relied as much on big data and advanced AI as it wanted its late-stage investors to believe, so the whole move has been somewhat of an investor story telling shadow war.
Personally, I am not a big fan of the idea that one might try to leverage crypto-based markets as a way to improve on this resource allocation mess.
No, I’m not sure how you got that impression (was it “failing to coordinate”?), I’m asking for the opposite reason.
I guess I got that impression from the ‘public good producers significantly accelerate the development of AGI’ in the title, and then looking at the impactcerts website.
I somehow overlooked the bit where you state that you are also wondering if that would be a good idea.
To be clear: my sense of the current AI open source space is that it definitely under-produces certain software components, software components that could be relevant for improving AI/AGI safety.
What are some of those components? We can put them on a list.
By the way, “myopic” means “pathologically short-term”.
Good question. I don’t have a list, just a general sense of the situation. Making a list would be a research project in itself. Also, different people here would give you different answers. That being said,
I occasionally see comments from alignment research orgs who do actual software experiments that they spend a lot of time on just building and maintaining the infrastructure to run large scale experiments. You’d have to talk to actual orgs to ask them what they would need most. I’m currently a more theoretical alignment researcher, so I cannot offer up-to-date actionable insights here.
As a theoretical researcher, I do reflect on what useful roads are not being taken, by industry and academia. One observation here is that there is an under-investment in public high-quality datasets for testing and training, and in the (publicly available) tools needed for dataset preparation and quality assurance. I am not the only one making that observation, see for example https://research.google/pubs/pub49953/ . Another observation is that everybody is working on open source ML algorithms, but almost nobody is working on open source reward functions that try to capture the actual complex details of human needs, laws, or morality. Also, where is the open source aligned content recommender?
On a more practical note, AI benchmarks have turned out to be a good mechanism for drawing attention to certain problems. Many feel that this benchmarks are having a bad influence on the field of AI, I have a lot of sympathy for that view, but you might also go with the flow. A (crypto) market that rewards progress on selected alignment benchmarks may be a thing that has value. You can think here of benchmarks that reward cooperative behaviour, truthfulness and morality in answers given by natural language querying systems, playing games ethically ( https://arxiv.org/pdf/2110.13136.pdf ), etc. My preference would be to reward benchmark contributions that win by building strong priors into the AI to guide and channel machine learning; many ML researchers would consider this to be cheating, but these are supposed to be alignment benchmarks, not machine-learning-from-blank-slate benchmarks. I have some doubts about the benchmarks for fairness in ML which are becoming popular, if I look at the latest NeurIPS: the ones I have seen offer tests which look a bit too easy, if the objective is to reward progress on techniques that have the promise of scaling up to more complex notions of fairness and morality you would like to have at the AGI level, or even for something like a simple content recommendation AI. Some cooperative behaviour benchmarks also strike me as being too simple, in their problem statements and mechanics, to reward the type of research that I would like to see. Generally, you would want to retire a benchmark from the rewards-generating market when the improvements on the score level out.