What I was thinking was “would you expect a FAI to do its own research about what it needs to for people to be physically safe enough, or should something on the subject be built in?
Ahh. Yeah, I’d expect that kind of content is way too specific to be built into initial FAI designs. There are multiple reasons for this, but off the top of my head,
I expect design considerations for Seed AI to favor smaller designs that only emphasize essential components for both superior ability to show desirable provability criteria, as well as improving design timelines.
All else equal, I expect that the less arbitrary decisions or content the human programmers provide to influence the initial dynamic of FAI, the better.
And my broadest answer is it’s not a core-Friendliness problem, so it’s not on the critical path to solving FAI. Even if an initial FAI design did need medical content or other things along those lines, this would be something that we could hire an expert to create towards the end of solving the more fundamental Friendliness and AI portions of FAI.
Note that this actually has very little to do with most of the seemingly hard parts of FAI theory. Much of it would be just as important if we wanted to create a recursively self modifying paper-clip maximizer, and be sure that it wouldn’t accidentally end up with the goal of “do the right thing”.
The actual implementation is probably far enough away that these issues aren’t even on the radar screen yet.
What I was thinking was “would you expect a FAI to do its own research about what it needs to for people to be physically safe enough, or should something on the subject be built in?
Ahh. Yeah, I’d expect that kind of content is way too specific to be built into initial FAI designs. There are multiple reasons for this, but off the top of my head,
I expect design considerations for Seed AI to favor smaller designs that only emphasize essential components for both superior ability to show desirable provability criteria, as well as improving design timelines.
All else equal, I expect that the less arbitrary decisions or content the human programmers provide to influence the initial dynamic of FAI, the better.
And my broadest answer is it’s not a core-Friendliness problem, so it’s not on the critical path to solving FAI. Even if an initial FAI design did need medical content or other things along those lines, this would be something that we could hire an expert to create towards the end of solving the more fundamental Friendliness and AI portions of FAI.
Note that this actually has very little to do with most of the seemingly hard parts of FAI theory. Much of it would be just as important if we wanted to create a recursively self modifying paper-clip maximizer, and be sure that it wouldn’t accidentally end up with the goal of “do the right thing”.
The actual implementation is probably far enough away that these issues aren’t even on the radar screen yet.