Train on everything, finetune if needed. Big enough and tuning seems pointless too. It’s not like humans need much tuning for everything. Why do we train and then throw away the network each time to start anew?
Why have separate modalities? That’s really dumb. Info is info, just give it all at once.
Cloze deletion should be network’s default task
Why bother cleaning data so much just dump more and more of it like those pots of neverending soup
I wonder if all this will finally teach us complex systems
If it picks up language, we could just try talking to it
I bet people will say it’s ”not real intelligence”. whatever bro, it’s more coherent than you
All these architectures don’t seem to matter that much
If it ever learns reasoning I feel like we’re screwed
normalization seems really important but batch norm is weird
When To Stop
Link post
(Old notes from 2016 that I stumbled upon).
Train on everything, finetune if needed. Big enough and tuning seems pointless too. It’s not like humans need much tuning for everything. Why do we train and then throw away the network each time to start anew?
Why have separate modalities? That’s really dumb. Info is info, just give it all at once.
Cloze deletion should be network’s default task
Why bother cleaning data so much just dump more and more of it like those pots of neverending soup
I wonder if all this will finally teach us complex systems
If it picks up language, we could just try talking to it
I bet people will say it’s ”not real intelligence”. whatever bro, it’s more coherent than you
All these architectures don’t seem to matter that much
If it ever learns reasoning I feel like we’re screwed
normalization seems really important but batch norm is weird
statistical learning theory doesn’t seem helpful
why can a network be compressed so much?
is parity the hardest function to learn