[Question] Have we seen any “ReLU instead of sigmoid-type improvements” recently

[KvmanThinking]

I read this and, it said:

there are huge low hanging fruit that any AI or random person designing AI in their garage can find by just grasping in the dark a bit, to get huge improvements at accelerating speeds.

have we found anything like this? at all? have we seen any “weird tricks” discovered that make AI way more powerful for no reason?