Error
Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?Unrecognized LW server error:
Field "fmCrosspost" of type "CrosspostOutput" must have a selection of subfields. Did you mean "fmCrosspost { ... }"?
Tiny remark regarding your post about the nomenclature of FLOP. Would it make sense for this series of intro posts to edit the occurrences of FLOPs to FLOP etc. to be consistent with your newly proposed nomenclature? I am currently upskilling in compute governance and as a newcomer, I was confused at first. I understand that it does not make sense to edit every post or article, but I just thought that it might be useful for those “intro” posts where a lot of basics are explained. Or maybe put in a link to the new nomenclature when you explain it for the first time? :)
Do you mean the first of the data points on the chart? The GPU was used for DL long before AlexNet. References: [1], [2], [3], [4], [5].
Thanks for the correction and references. I just followed my “common sense” from lectures and other pieces.
What do you think made AlexNet stand out? Is it the depth and use of GPUs?
I do not know the opinions of experts on this issue. And I lack competence for such conclusions, sorry.
I was slightly disappointed by this post not because it was bad but because it didn’t provide much new or interesting. I see this more as a recap and hope for the next posts in this sequence to build on this.
Thanks for the feedback, Gunnar. You’re right—it’s more of a recap and introduction. I think the “newest” insight is probably the updates in Section 2.3.
I also would be curious to know in which aspects and questions you’re most interested in.
The update in 2.3 was a valuable update. Based on the title (and my interests) I was hoping for
some integration of the limits for compute, memory, and interconnect. Like you say they limit each other but it is not very clear how the limits interrelate and scale with each other. Empirically, it would be interesting to see the relative sizes of these parts over time.
some comparison of the relative sizes of the human brain responsible for processing where we do have algorithms that are comparable to what the brain does, e.g. image processing and object and scene detection in the visual cortex.
Thanks!
I’m working with a colleague on the trends of the three components (compute, memory, and interconnect) over time of compute systems and then comparing it to our best estimates for the human brain (or other biological anchors). However, this will still take some time but I hope we will be able to share it in the future (≈ till the end of the year).
Cool. Looking forward to it.