There is a lot of complexity that arises when people try to reason about powerful optimizing processes.
I think part of this is because there are “naturally” a lot of feelings here. Like basically all human experiences proximate to naturally occurring instances of powerful optimization processes are colored by the vivid personal realities of it. Parents. Governments. Chainsaws. Forest fires. Championship sporting events. Financial schemes. Plague evolution. Etc.
By making a toy model of an effectively goal-pursuing thing (where the good and the bad are just numbers), the the essential mechanical predictability of the idea that “thermostats aim for what thermostats aim for because they are built to aim at things” can be looked at while still having a “safe feeling” despite the predictable complexity of the discussions… and then maybe people can plan for important things without causing the kind of adrenaline levels that normally co-occur with important things :-)
Another benefit of smallness and abstractness (aside from routing around “psychological defense mechanisms”) is that whatever design you posit is probably simple enough to be contained fully in the working memory of a person after relatively little study! So the educational benefit here is probably very very large!
This seems like good and important work!
There is a lot of complexity that arises when people try to reason about powerful optimizing processes.
I think part of this is because there are “naturally” a lot of feelings here. Like basically all human experiences proximate to naturally occurring instances of powerful optimization processes are colored by the vivid personal realities of it. Parents. Governments. Chainsaws. Forest fires. Championship sporting events. Financial schemes. Plague evolution. Etc.
By making a toy model of an effectively goal-pursuing thing (where the good and the bad are just numbers), the the essential mechanical predictability of the idea that “thermostats aim for what thermostats aim for because they are built to aim at things” can be looked at while still having a “safe feeling” despite the predictable complexity of the discussions… and then maybe people can plan for important things without causing the kind of adrenaline levels that normally co-occur with important things :-)
Another benefit of smallness and abstractness (aside from routing around “psychological defense mechanisms”) is that whatever design you posit is probably simple enough to be contained fully in the working memory of a person after relatively little study! So the educational benefit here is probably very very large!