The data for GPT-2 has been replicated by the open source OpenWebText project. To my knowledge the same dataset was utilised for GPT-3, so accessing it is not a problem.
The parallelizability of GPT-3 is something I’ve been looking into. The current implementation of zero-2 seems like the best way to memory-optimally train a 170B parameter transformer model.
The data for GPT-2 has been replicated by the open source OpenWebText project. To my knowledge the same dataset was utilised for GPT-3, so accessing it is not a problem.
The parallelizability of GPT-3 is something I’ve been looking into. The current implementation of zero-2 seems like the best way to memory-optimally train a 170B parameter transformer model.