I’m Jérémy Perret. Based in France. PhD in AI (NLP). AI Safety & EA meetup organizer. Information sponge. Mostly lurking since 2014. Seeking more experience, and eventually a position, in AI safety/governance.
Extremely annoyed by the lack of an explorable framework for AI risk/benefits. Working on that.
I second this, and expansions of these ideas.
Thank you, that is clearer!
But let’s suppose that the first team of people who build a superintelligence first decide not to turn the machine on and immediately surrender our future to it. Suppose they recognize the danger and decide not to press “run” until they have solved alignment.
The section ends here but… isn’t there a paragraph missing? I was expecting the standard continuation along the lines of “Will the second team make the same decision, once they reach the same capability? Will the third, or the fourth?” and so on.
Thank you for this post, I find this distinction very useful and would like to see more of it. Has the talk been recorded, by any chance (or will you give it again)?
Thank you, that’s was my understanding. Looking forward to the second competition! And, good luck sorting out all the submissions for this one.
[Meta comment]
The deadline is past, should we keep the submissions coming or is it too late? Some of the best arguments I could find elsewhere are rather long, in the vein of the Superintelligence FAQ. I did not want to copy-paste chunks of it and the arguments stand better as part of a longer format.
Anyway, signalling that the lack of money incentive will not stop me from trying to generate more compelling arguments… but I’d rather do it in French instead of posting here (I’m currently working on some video scripts on AI alignment, there’s not enough French content of that type).
(Policymakers) We have a good idea of what make bridges safe, through physics, materials science and rigorous testing. We can anticipate the conditions they’ll operate in.
The very point of powerful AI systems is to operate in complex environments better than we can anticipate. Computer science can offer no guarantees if we don’t even know what to check. Safety measures aren’t catching up quickly enough.
We are somehow tolerating the mistakes of current AI systems. Nothing’s ready for the next scale-up.
(ML researchers) We still don’t have a robust solution to specification gaming: powerful agents find ways to get high reward, but not in the way you’d want. Sure, you can tweak your objective, add rules, but this doesn’t solve the core problem, that your agent doesn’t seek what you want, only a rough operational translation.
What would a high-fidelity translation would look like? How would create a system that doesn’t try to game you?
(Policymakers) There is outrage right now about AI systems amplifying discrimination and polarizing discourse. Consider that this was discovered after they were widely deployed. We still don’t know how to make them fair. This isn’t even much of a priority.
Those are the visible, current failures. Given current trajectories and lack of foresight of AI research, more severe failures will happen in more critical situations, without us knowing how to prevent them. With better priorities, this need not happen.
(Tech execs) “Don’t ask if artificial intelligence is good or fair, ask how it shifts power”. As a corollary, if your AI system is powerful enough to bypass human intervention, it surely won’t be fair, nor good.
(ML researchers) Most policies are unsafe in a large enough search space; have you designed yours well, or are you optimizing through a minefield?
(Policymakers) AI systems are very much unlike humans. AI research isn’t trying to replicate the human brain; the goal is, however, to be better than humans at certain tasks. For the AI industry, better means cheaper, faster, more precise, more reliable. A plane flies faster than birds, we don’t care if it needs more fuel. Some properties are important (here, speed), some aren’t (here, consumption).
When developing current AI systems, we’re focusing on speed and precision, and we don’t care about unintended outcomes. This isn’t an issue for most systems: a plane autopilot isn’t making actions a human pilot couldn’t do; a human is always there.
However, this constant supervision is expensive and slow. We’d like our machines to be autonomous and quick. They perform well on the “important” things, so why not give them more power? Except, here, we’re creating powerful, faster machines that will reliably do thing we didn’t have time to think about. We made them to be faster than us, so we won’t have time to react to unintended consequences.
This complacency will lead us to unexpected outcomes. The more powerful the systems, the worse they may be.
(Tech execs) Tax optimization is indeed optimization under the constraints of the tax code. People aren’t just stumbling on loopholes, they’re actually seeking them, not for the thrill of it, but because money is a strong incentive.
Consider now AI systems, built to maximize a given indicator, seeking whatever strategy is best, following your rules. They will get very creative with them, not for the thrill of it, but because it wins.
Good faith rules and heuristics are no match for adverse optimization.
(ML researchers) Powerful agents are able to search through a wide range of actions. The more efficient the search, the better the actions, the higher the rewards. So we are building agents that are searching in bigger and bigger spaces.
For a classic pathfinding algorithm, some paths are suboptimal, but all of them are safe, because they follow the map. For a self-driving car, some paths are suboptimal, but some are unsafe. There is no guarantee that the optimal path is safe, because we really don’t know how to tell what is safe or not, yet.
A more efficient search isn’t a safer search!
(Policymakers) The goals and rules we’re putting into machines are law to them. What we’re doing right now is making them really good at following the letter of this law, but not the spirit.
Whatever we really mean by those rules, is lost on the machine. Our ethics don’t translate well. Therein lies the danger: competent, obedient, blind, just following the rules.
Thank you for curating this, I had missed this one and it does provide a useful model of trying to point to particular concepts.
Hi! Thank you for this project, I’ll attempt to fill the survey.
My apologies if you already encountered the following extra sources I think are relevant to this post:
the Modeling Transformative AI Risk (MTAIR) project (an attempt to map out the relationships between key hypotheses and cruxes involved in debates about catastrophic risks from advanced AI);
Turchin & Derkenberger’s Classification of global catastrophic risks connected with artificial intelligence (lists and categorizes a wide range of catastrophic scenarios, from narrow or general AI, near-term or long-term, misuse or accidents, and many other factors, with references);
Hi! Thank you for this outline. I would like some extra details on the following points:
“They will find bugs! Maybe stack virtual boxes with hard limits”—Why is bug-finding an issue, here? Is your scheme aimed at producing agents that will not want to escape, or agents that we’d have to contain?
“Communicate in a manner legible to us”—How would you incentivize this kind of legibility, instead of letting communication shift to whatever efficient code is most useful for agents to coordinate and get more XP?
“Have secret human avatars steal, lie and aggress to keep the agents on their toes”—What is the purpose of this part? How is this producing aligned agents from definitely adversarial behavior from humans?
Congratulations on your launch!
As Michaël Trazzi in the other post, I’m interested in the kind of products you’ll develop, but more specifically in how the for-profit part interacts with both the conceptual research part and the incubator part. Are you expecting the latter two to yield new products as they make progress? Do these activities have different enough near-term goals that they mostly just coexist within Conjecture?
(also, looking forward to the pluralism sequence, this sounds great)
Answers in order: there is none, there were, there are none yet.
(Context starts, feel free to skip, this is the first time I can share this story)
After posting this, I was contacted by Richard Mallah, who (if memory serves right) created the map, compiled the references and wrote most of the text in 2017, to help with the next iteration of the map. The goal was to build a Body of Knowledge for AI Safety, including AGI topics but also more current-capabilities ML Safety methods.
This was going to happen in conjunction with the contributions of many academic & industry stakeholders, under the umbrella of CLAIS (Consortium on the Landscape of AI Safety), mentioned here.
There were design documents for the interactivity of the resource, and I volunteered Back in 2020 I had severely overestimated both my web development skills and ability to work during a lockdown, never published a prototype interface, and for unrelated reasons the CLAIS project… winded down.
(End of context)
I do not remember Richard mentioning a review of the map contents, apart from the feedback he received back when he wrote them. The map has been a bit tucked in a corner of the Internet for a while now.
The plans to update/expand it failed as far as I can tell. There is no new version and I’m not aware of any new plans to create one. I stopped working on this in April 2021.
There is no current map with this level of interactivity and visualization, but there has been a number of initiatives trying to be more comprehensive and up-to-date!