Armchair Transit Enthusiast here! Hopefully I can share some of my understanding on the situation. In short I think you’ve hit most of the answer in your last paragraph.
As background for those new to the topic (which it OP seems to know most of this already), this was actually how large swaths of pre-WW1 cities were developed and that method has largely disappeared by car-centric development that we’ve see explode post-WW2. I’m sure there are a variety of different reasons for this but if acceptable I can put a few of them in a quick list (willing to expand upon it if there is interest).
-
Governmental City Planning favoring cars the advantage-it was simply the zeitgeist for 60 years, lower (apparent) cost and perhaps some corruption (there is some evidence of a “automobile lobby” going for regulatory capture route)
-
Federal Government subsiding car infrastructure for defense encourages secondary development taking advantage of the new infrastructure.
-
Perceived market preference for car based development
-
Lower investment cost of non-transit development (dense=expensive and less dense development favors decentralized vs centralized transit)
-
Increase in governmental outreach and inclusion of affected voices- Overall good yes, but does come with a notable increase in costs and project timelines There still exists
However, there still are projects being built in this method though! Notably, Brightline East and West with the more important link being this, which is the parent company developing the adjacent real estate. I think time will show if these projects are initially successful, then we will see further interest and attempts being made.
And now for the low effort response to OPs question:
“Are cities unwilling to agree to high density for newly transit-served locations?”
Absolutely, though not the “City” per se but a large enough minority of citizens to make it exceedingly difficult. Time, and time again during the outreach phase of development there are always backlashes from citizens which make it very difficult to get ANY density built in cities. This pressures governmental officials which then result in needing project reworks to get approval, which still are not good enough for the concerned citizens, etc etc etc, ad nauseam. All of this balloons cost and timelines.
“Are the costs too unpredictable?”
Kinda, but I would say its just far to more capital intensive. First land purchasing costs, which key here is that not only is it a large track of land but also large tracts that are contiguous. Then once you start needing to get planning permission of right of way from Federal, State DOTs and local officials, trying to get everyone on board can take years. Then you have to deal with the hellscape of up-zoning and getting approval/permits (see below). All the while you better hope no environmentalist decides your transit route is gonna intersect a wildlife corridor and takes up the noble lawsuit fight (okay my bias is bleeding though a little). See next for the tie together.
“Are private developers too car-focused?”
Yes, but I would say not because they are ignorant but rather inexperienced. Think of how much new/different expertise is needed here in the way of engineering and legal. There simply is no experience or latent knowledge even in planning, hiring, and working with experts in the respective fields. This combined with the heavy investment needed and the few data points in recent decades make this a hard sell to decision makers.