Avoid the abbreviation “FLOPs” – use “FLOP” or “FLOP/s” instead
Especially in discussions about AI, the abbreviation “FLOPs” is being used for both “floating point operations per second” (a measure of computational power) and “floating point operations” (a measure of total computations, and equivalent to the previous term times seconds). This is ambiguous and confusing. For clarity, I propose people avoid this specific abbreviation and instead use the alternatives of “FLOP” (for floating point operations) and “FLOP/s” (for floating point operations per second).
- Analysing a 2036 Takeover Scenario by (6 Oct 2022 20:48 UTC; 9 points)
- Analysing a 2036 Takeover Scenario by (EA Forum; 6 Oct 2022 20:48 UTC; 4 points)
- 's comment on Inflection.ai is a major AGI lab by (10 Aug 2023 5:04 UTC; 4 points)
- 's comment on Is it a coincidence that GPT-3 requires roughly the same amount of compute as is necessary to emulate the human brain? by (10 Feb 2023 21:30 UTC; 2 points)
“Floating point operations per second” is usually abbreviated FLOPS. I think using FLOPs as a synonym for FLOPS seems wrong in a particularly confusing way. It really suggests that FLOP is the acronym and the “s” is for a plural. Compare to someone saying SMs instead of SMS.
That said, I agree that using FLOPs as the plural of “floating point operation” is confusing since it is pretty similar to FLOPS. It’s also not something I’ve seen much outside of the futurist crowd (though I am guilty of using it fairly often). Another way in which the ML and futurist usage is nonstandard is that we often use FLOPS sloppily to talk about very low precision or even integer arithmetic.
I haven’t personally seen this cause very much confusion, in my experience it maybe increases the amount of “compute stock vs flow” confusion by like 25-50% from a small-ish baseline? I suspect it’s small enough that accessibility and using standard notation is significantly more important. (Though again, my sense is that FLOPs for “floating point operations” is not really standard and not particularly accessible, so it seems more plausible futurists should stop doing that.)
Not sure what’s realistic to do here. I’m a bit inclined to use “operations” or “ops” to both avoid this ambiguity and be more inclusive of integer arithmetic, but I kind of wouldn’t expect that to stick and maybe it’s bad in some other way. Using FLOP as a plural noun is pretty unsatisfying and I suspect it is unlikely to stick for fundamental linguistic reasons. Switching to using FLOP/s instead of FLOPS may be realistic, though I think flop/s is a bit less weird-looking to me.
ETA: though I think openphil has been using FLOP/s and FLOP as you suggest in their writing, so maybe I’m wrong about whether it’s realistic. I think the draw to add an s to plurals is strongest in spoken language, though unfortunately that’s also where the ambiguity is worst.
As for using FLOP as a plural noun, that’s how other units work. We use 5 m for 5 meters, 5 s for 5 seconds, 5 V for 5 volts, etc. so it’s not that weird.
This feels easier to me when writing and significantly harder for spoken language (where I have a strong inclination to add the s). I guess it’s probably worth thinking separately about talking vs writing, and this post is mostly about writing, in which case I’m probably sold.
In spoken language, you could expand the terms to “floating-point operations” vs “floating-point operations per second” (or just “operations (per second)” if that felt more apt)
Could write “FLOPS-seconds” maybe, which, although a bit wordy, resembles the common usage of “kilowatt-hours” instead of a more directly joule-based unit.
Can you point to example usages that are causing confusion?
Here’s a quote from someone on EleutherAI discord 2 weeks ago:
I believe the first “flops” is FLOP/s and the second “flops” is FLOP, although I’m not 100% confident. I replied asking for clarification and didn’t get a straight answer.
FWIW, I am ~100% confident that this is correct in terms of what they refer to. Typical estimates of the brain are that it uses ~10^15 FLOP/s (give or take a few OOM) and the fastest supercomputer in the world uses ~10^18 FLOP/s when at maximum (so there’s no way GPT-3 was trained on 10^23 FLOP/s).
If we assume the exact numbers here are correct, then the actual conclusion is that GPT-3 was trained on the amount of compute the brain uses in 10 million seconds, or around 100 days.
That’s a great example, thanks!
(I’m used to “flops” always being used to describe the capacity of machines (as in, per second) and not total computation as in this example.)
Agree that this discussion is surprisingly often confusing and people use the terms interchangeably. Unfortunately, readers often referred to our training compute measurement as a measure of performance, rather than a quantity of executed operations. However, I don’t think that this is necessarily due to the abbreviations but also due to the lack of understanding of what one measures. Next to making the distinction more clear with the terms, one should probably also explain it more and use terms such as quantity and performance.
For my research, I’ve been trying to be consistent with FLOPs (smaller case s) referring to the quantity. While FLOPS or FLOP/s refer to the performance: operations per second. (FWIW, during my time in computer engineering, it has been the norm to use FLOPs for quantity and FLOPS for performance.)
The term Petaflop/s-days also helps—outlining how long the performance (Petaflop/s) runs for how many days, therefore measuring a quantity of operations.
Note it gets even more complicated once we take the number representation (floating point 32bit, 16bit, or even bfloat16) into consideration. Therefore, I’m also in favor of maybe switching at one point towards OPs and OP/s and also document the used number representation for actual technical documentation (such as reporting the compute of ML models).
Note that @lennart has since recommended using FLOP for quantity and FLOP/s (or FLOPS) for performance.
This seems like a good idea, although I feel that the number of people who use FLOPS to mean FLOP per second are still going to grow faster than those who switch to using FLOP/s, as most introductory material from a simple google search still refer to FLOPS as FLOP per second. Perhaps an edit to the wiki page could expedite the switch?
Also, will FLOP/s still be pronounced ‘flops’ or will it now be ‘flop per second’?
i can’t think of another (pseudo-)acronym which gets used as an all-caps unit, off the top of my head. i may toy around with “Flop” as a unit, like GFlop for a billion operations, GFlop/s for a billion operations per second.