Consider how Ofria failed. Somebody told me that in that context deception is a property of the environment and its observer. However, it seems to me that the objective of the designer of the system needs to be factored in.
Although in general an observer can be deceived, this is not the case here I would argue. Ofria just designed a system that did not do what he wanted it to do. It failed transparently.
It would seem that this is similar to you wanting to build a rocket that goes to the moon, but then building a rocket that just explodes, because your understanding of rocketry is poor, and then saying “The rocket deceived me into thinking it would not explode”.
Imagine you want to build an honest system. You build a system that looks to you like it would be honest. But it ends up deceiving you because your understanding is flawed. In that case, the system would be deceptive, but I wouldn’t call the fact that you were mistaken about what would constitute a non-deceptive system deception. That is a particular kind of incompetence.
Also, note that the failure mode here is different from “Humans use condoms” type things. When Ofria created his system the organism would perform well in the overall system. The overall system stayed constant. It was not the case that at some point ice cream was introduced into the organism’s environment.
Instead, the failure mode was to think that by creating a special environment, the behavior observed in that environment would generalize widely by default.
Deceived by your own Incompetence
Consider how Ofria failed. Somebody told me that in that context deception is a property of the environment and its observer. However, it seems to me that the objective of the designer of the system needs to be factored in.
Although in general an observer can be deceived, this is not the case here I would argue. Ofria just designed a system that did not do what he wanted it to do. It failed transparently.
It would seem that this is similar to you wanting to build a rocket that goes to the moon, but then building a rocket that just explodes, because your understanding of rocketry is poor, and then saying “The rocket deceived me into thinking it would not explode”.
Imagine you want to build an honest system. You build a system that looks to you like it would be honest. But it ends up deceiving you because your understanding is flawed. In that case, the system would be deceptive, but I wouldn’t call the fact that you were mistaken about what would constitute a non-deceptive system deception. That is a particular kind of incompetence.
Also, note that the failure mode here is different from “Humans use condoms” type things. When Ofria created his system the organism would perform well in the overall system. The overall system stayed constant. It was not the case that at some point ice cream was introduced into the organism’s environment.
Instead, the failure mode was to think that by creating a special environment, the behavior observed in that environment would generalize widely by default.