a dramatic hardware shift like that is likely going to mean a significant portion of progress up until that shift in topics like interpretability and alignment may be going out the window.
It’s still early to tell, as the specific characteristics of a photonic or optoelectronic neural network are still formulating in the developing literature.
For example, in my favorite work of the year so far, the researchers found they could use sound waves to reconfigure an optical neural network as the sound waves effectively preserved a memory of previous photon states as they propagated: https://www.nature.com/articles/s41467-024-47053-6
If you have bidirectionality where previously you didn’t, it’s not a reach to expect that the way in which data might encode in the network, as well as how the vector space is represented, might not be the same. And thus, that mechanistic interpretability gains may get a bit of a reset.
And this is just one of many possible ways it may change by the time the tech finalizes. The field of photonics, particularly for neural networks, is really coming along nicely. There may yet be future advances (I think this is very likely given the pace to date), and advantages the medium offers that electronics haven’t.
It’s hard to predict exactly what’s going to happen when two different fields which have each had unexpected and significant gains over the past 5 years collide, but it’s generally safe to say that it will at very least result in other unexpected things too.
Why is this the case?
It’s still early to tell, as the specific characteristics of a photonic or optoelectronic neural network are still formulating in the developing literature.
For example, in my favorite work of the year so far, the researchers found they could use sound waves to reconfigure an optical neural network as the sound waves effectively preserved a memory of previous photon states as they propagated: https://www.nature.com/articles/s41467-024-47053-6
In particular, this approach is a big step forward for bidirectional ONN, which addresses what I think is the biggest current flaw in modern transformers—their unidirectionality. I discussed this more in a collection of thoughts on directionality impact on data here: https://www.lesswrong.com/posts/bmsmiYhTm7QJHa2oF/looking-beyond-everett-in-multiversal-views-of-llms
If you have bidirectionality where previously you didn’t, it’s not a reach to expect that the way in which data might encode in the network, as well as how the vector space is represented, might not be the same. And thus, that mechanistic interpretability gains may get a bit of a reset.
And this is just one of many possible ways it may change by the time the tech finalizes. The field of photonics, particularly for neural networks, is really coming along nicely. There may yet be future advances (I think this is very likely given the pace to date), and advantages the medium offers that electronics haven’t.
It’s hard to predict exactly what’s going to happen when two different fields which have each had unexpected and significant gains over the past 5 years collide, but it’s generally safe to say that it will at very least result in other unexpected things too.