I’m sorry that I missed this.
I don’t have a great answer for you. It is largely speculation about counterfactual benefits. I would like to delude myself into thinking it’s good counterfactual speculation, but that’s probably just my ego talking.
I’m mostly basing this around things that continue to be problems for years. For instance, I work as an AI research engineer at a large industry lab, and a common task is to launch experiments that use O(100) GPUs. We have a system that automatically schedules these jobs on different data centers depending on availability. It does not have the ability to choose a data center that satisfies constraints on both spot instances and on-demand instances. The scheduler can only do one of these. As a result, we have to manually choose cells. It has been like this for >4 years.
I have a bunch of additional examples like this that have gone unsolved for years. This is a log way of saying that, to directly answer your question, I don’t have much more than speculation.
That’s a really great way to think about it- thank you for that metaphor.