it requires racks of power hungry GPUs to achieve the above that are mounted in data centers
Inference with models trained for ternary quantization (which uses massively fewer multiplications and so less power) only needs hardware that can take advantage of it, doesn’t significantly lose quality compared to full precision. Though I don’t know if there is a good RNN-like block to enable large context while still able to mostly avoid multiplications with ternary weights (as opposed to activations, which need to be more precise), which seems crucial for video. A more pressing issue might be latency.
You’re right. I’ve also worked on the hardware for the SOTA for self driving pre-transformers. (2020-2022). We had enough DDR for smaller models, not 50B+ like https://robotics-transformer-x.github.io/ . Quantized support for int8 and fp16.
I’m thinking there are 2 factors working against self driving that won’t apply to other domains:
1. Stationary hardware in a data center gets utilized better and is going to be faster than the embedded boards mounted in a vehicle, and upgraded more often. (the robots can pause operations to handle higher priority requests, a factory doesn’t have to wait on people to summon an SDC taxi but can work 24⁄7.)
2. Liability/legal delays. What I was thinking is that if transformer models really do scale to general purpose robots that have human like manipulation and generality, you could use them without any delays in human free areas. I don’t think there are many legal requirements to do this. Initially in caged off areas:
Later you would just set the safety switches at the doors and make all human workers leave the building before enabling the robots. You could restock and clean retail stores, for example : make all the customers and workers leave before letting the robots go to work.
Ideally the area the robots are in would count as an “appliance” and thus avoid building codes as well. I don’t know precisely how this works, just I don’t think Samsung has to cater to the (slightly different from everywhere else) requirements of Beaumont, Texas to sell a microwave at a store there, but anyone wanting to build a shed or factory does.
Inference with models trained for ternary quantization (which uses massively fewer multiplications and so less power) only needs hardware that can take advantage of it, doesn’t significantly lose quality compared to full precision. Though I don’t know if there is a good RNN-like block to enable large context while still able to mostly avoid multiplications with ternary weights (as opposed to activations, which need to be more precise), which seems crucial for video. A more pressing issue might be latency.
You’re right. I’ve also worked on the hardware for the SOTA for self driving pre-transformers. (2020-2022). We had enough DDR for smaller models, not 50B+ like https://robotics-transformer-x.github.io/ . Quantized support for int8 and fp16.
I’m thinking there are 2 factors working against self driving that won’t apply to other domains:
1. Stationary hardware in a data center gets utilized better and is going to be faster than the embedded boards mounted in a vehicle, and upgraded more often. (the robots can pause operations to handle higher priority requests, a factory doesn’t have to wait on people to summon an SDC taxi but can work 24⁄7.)
2. Liability/legal delays. What I was thinking is that if transformer models really do scale to general purpose robots that have human like manipulation and generality, you could use them without any delays in human free areas. I don’t think there are many legal requirements to do this. Initially in caged off areas:
Later you would just set the safety switches at the doors and make all human workers leave the building before enabling the robots. You could restock and clean retail stores, for example : make all the customers and workers leave before letting the robots go to work.
Ideally the area the robots are in would count as an “appliance” and thus avoid building codes as well. I don’t know precisely how this works, just I don’t think Samsung has to cater to the (slightly different from everywhere else) requirements of Beaumont, Texas to sell a microwave at a store there, but anyone wanting to build a shed or factory does.