[EDIT: oops, I thought you were talking about the direct power consumption of the computation, not the extra hardware weight. My bad.]
It’s not about the power consumption.
The air conditioner in your car uses 3 kW, and GPT-3 takes 0.4 kWH for 100 pages of output—thus a dedicated computer on AC power could produce 700 pages per hour, going substantially faster than AI Dungeon (literally and metaphorically). So a model as large as GPT-3 could run on the electricity of a car.
The hardware would be more expensive, of course. But that’s different.
Huh, thanks—I hadn’t run the numbers myself, so this is a good wake-up call for me. I was going off what Elon said. (He said multiple times that power efficiency was an important design constraint on their hardware because otherwise it would reduce the range of the car too much.) So now I’m just confused. Maybe Elon had the hardware weight in mind, but still...
Maybe the real problem is just that it would add too much to the price of the car?
[EDIT: oops, I thought you were talking about the direct power consumption of the computation, not the extra hardware weight. My bad.]
It’s not about the power consumption.
The air conditioner in your car uses 3 kW, and GPT-3 takes 0.4 kWH for 100 pages of output—thus a dedicated computer on AC power could produce 700 pages per hour, going substantially faster than AI Dungeon (literally and metaphorically). So a model as large as GPT-3 could run on the electricity of a car.
The hardware would be more expensive, of course. But that’s different.
Huh, thanks—I hadn’t run the numbers myself, so this is a good wake-up call for me. I was going off what Elon said. (He said multiple times that power efficiency was an important design constraint on their hardware because otherwise it would reduce the range of the car too much.) So now I’m just confused. Maybe Elon had the hardware weight in mind, but still...
Maybe the real problem is just that it would add too much to the price of the car?
Yes. GPU/ASICs in a car will have to sit idle almost all the time, so the costs of running a big model on it will be much higher than in the cloud.