lennart comments on Google’s new 540 billion parameter language model

[lennart]

From their paper:

We trained PaLM-540B on 6144 TPU v4 chips for 1200 hours and 3072 TPU v4 chips for 336 hours including some downtime and repeated steps.

That’s 64 days.