The idea is not that humans are perfect consquentialists, but that they are able to work at all to produce future-steering outputs, insofar as humans actually do work at all, by an inner overlap of the shape of inner parts which has a shape resembling consequentialism, and the resemblance is what does the work. That is, your objection has the same flavor as “But humans aren’t Bayesian! So how can you say that updating on evidence is what’s doing their work of mapmaking?”
The idea is not that humans are perfect consquentialists, but that they are able to work at all to produce future-steering outputs, insofar as humans actually do work at all, by an inner overlap of the shape of inner parts which has a shape resembling consequentialism, and the resemblance is what does the work. That is, your objection has the same flavor as “But humans aren’t Bayesian! So how can you say that updating on evidence is what’s doing their work of mapmaking?”
To be clear I think I agree with your overall position. I just don’t think the argument you gave for it (about bureaucracies etc.) was compelling.