This feels somewhat similar to medical research with and without a detailed understanding of cell biochemistry. You can get very far without a deep understanding of the low level details and often this seems to be the best method for producing useful drugs (as I understand it most drugs are discovered through trial and error).
But understanding the low level details can get you important breakthroughs like mRNA vaccines. And if you’re maybe betting the future of humanity that you can create a cure for a new type of deadly virus (superintelligent AI) on the first try you would prefer to be confident that you know what you’re doing.
I agree, but I wouldn’t bet the future of humanity on mRNA vaccines that haven’t been thoroughly tested in practice or at least in very analogous situations either. If you code, this is like deploying code in production—it almost always goes badly if you haven’t created and tested a fake production environment first.
I think we need a holistic approach for this exact reason. It feels like we aren’t thinking enough about doing alignment “in production”, but instead focusing only on the theoretical “mRNA vaccine” for a subject that we haven’t taken sufficient time to interact with “in it’s own language”.
This feels somewhat similar to medical research with and without a detailed understanding of cell biochemistry. You can get very far without a deep understanding of the low level details and often this seems to be the best method for producing useful drugs (as I understand it most drugs are discovered through trial and error).
But understanding the low level details can get you important breakthroughs like mRNA vaccines. And if you’re maybe betting the future of humanity that you can create a cure for a new type of deadly virus (superintelligent AI) on the first try you would prefer to be confident that you know what you’re doing.
I agree, but I wouldn’t bet the future of humanity on mRNA vaccines that haven’t been thoroughly tested in practice or at least in very analogous situations either. If you code, this is like deploying code in production—it almost always goes badly if you haven’t created and tested a fake production environment first.
I think we need a holistic approach for this exact reason. It feels like we aren’t thinking enough about doing alignment “in production”, but instead focusing only on the theoretical “mRNA vaccine” for a subject that we haven’t taken sufficient time to interact with “in it’s own language”.