One interpretation of this criticism is that it implies that feedback-optimization systems are too dumb to do relevant long-term reasoning, even with substantial work in reward engineering.
If this is true, it seems like a really important point that I need to understand better. Any chance you can surface this argument into a top-level post, so more people can see it and chime in with their thoughts? In particular I’d like to understand whether the problem is caused by current ML approaches not offering good/useful enough performance guarantees, which might change in the future, or if this a fundamental problem with ML and feedback-optimization that can’t be overcome. Also, can you suggest ways to test this empirically?
(I also can’t quite tell to what extent Paul’s response has addressed your criticism. If you decide to write a post maybe you can explain that as well?)
If this is true, it seems like a really important point that I need to understand better. Any chance you can surface this argument into a top-level post, so more people can see it and chime in with their thoughts? In particular I’d like to understand whether the problem is caused by current ML approaches not offering good/useful enough performance guarantees, which might change in the future, or if this a fundamental problem with ML and feedback-optimization that can’t be overcome. Also, can you suggest ways to test this empirically?
(I also can’t quite tell to what extent Paul’s response has addressed your criticism. If you decide to write a post maybe you can explain that as well?)