It seems plausible that transformative agents will be trained exclusively on real-world data (without using simulated environments) [EDIT: in “data” I mean to include the observation/reward signal from the real-world environment in an online RL setup]; including social media feed-creation algorithms, and algo-trading algorithms. In such cases, the researchers don’t choose how to implement the “other agents” (the other agents are just part of the real-world environment that the researchers don’t control).
Focusing on agents that are trained on simulated environments that involve multiple agents: For a lab to use copies of other labs’ agents, the labs will probably need to cooperate (or some other process that involves additional actors may need to exist). In any case, using copies of the agent that is being trained (i.e. self-play) seems to me very plausible. (Like, I think both AlphaZero and OpenAI Five were trained via self-play and that self-play is generally considered to be a very prominent technique for RL-in-simulated-environments-that-involve-multiple-agents).
It seems plausible that transformative agents will be trained exclusively on real-world data (without using simulated environments); including social media feed-creation algorithms, and algo-trading algorithms. In such cases, the researchers don’t choose how to implement the “other agents” (the other agents are just part of the real-world environment that the researchers don’t control).
I have quite a different intuition on this, and I’m curious if you have a particular justification for expecting non-simulated training for multi-agent problems. Some reasons I expect otherwise:
At the very least, in the early days, you simply won’t have much (accurate, up-to-date) data on the behavior of AI systems made by other labs because they’re new. So you’ll probably need to use some simulations and/or direct coordination with those other labs.
Even as the systems become more mature, if there’s rapid improvement over time as seems likely, again the relevant data that would let you accurately predict the behavior of the other systems will be sparse.
Even if data are abundant, presumably for some strategic purposes developers won’t want to design their AIs to behave so predictably that learning from such data will be sufficient. You’ll probably need to use some combination of game theoretically informed models and augmentation of the data to account for distributional shifts, if you want to put your AI to use in any task that involves interacting with AIs you didn’t create.
I have quite a different intuition on this, and I’m curious if you have a particular justification for expecting non-simulated training for multi-agent problems.
In certain domains, there are very strong economic incentives to train agents that will act in a real-world multi-agent environment, where the ability to simulate the environment is limited (e.g. trading in stock markets and choosing content for social media users).
Some off-the-cuff thoughts:
It seems plausible that transformative agents will be trained exclusively on real-world data (without using simulated environments) [EDIT: in “data” I mean to include the observation/reward signal from the real-world environment in an online RL setup]; including social media feed-creation algorithms, and algo-trading algorithms. In such cases, the researchers don’t choose how to implement the “other agents” (the other agents are just part of the real-world environment that the researchers don’t control).
Focusing on agents that are trained on simulated environments that involve multiple agents: For a lab to use copies of other labs’ agents, the labs will probably need to cooperate (or some other process that involves additional actors may need to exist). In any case, using copies of the agent that is being trained (i.e. self-play) seems to me very plausible. (Like, I think both AlphaZero and OpenAI Five were trained via self-play and that self-play is generally considered to be a very prominent technique for RL-in-simulated-environments-that-involve-multiple-agents).
I have quite a different intuition on this, and I’m curious if you have a particular justification for expecting non-simulated training for multi-agent problems. Some reasons I expect otherwise:
At the very least, in the early days, you simply won’t have much (accurate, up-to-date) data on the behavior of AI systems made by other labs because they’re new. So you’ll probably need to use some simulations and/or direct coordination with those other labs.
Even as the systems become more mature, if there’s rapid improvement over time as seems likely, again the relevant data that would let you accurately predict the behavior of the other systems will be sparse.
Even if data are abundant, presumably for some strategic purposes developers won’t want to design their AIs to behave so predictably that learning from such data will be sufficient. You’ll probably need to use some combination of game theoretically informed models and augmentation of the data to account for distributional shifts, if you want to put your AI to use in any task that involves interacting with AIs you didn’t create.
In certain domains, there are very strong economic incentives to train agents that will act in a real-world multi-agent environment, where the ability to simulate the environment is limited (e.g. trading in stock markets and choosing content for social media users).