As each new writing deadline approaches, I return to The Builders, a poem by Longfellow:
All are architects of Fate,
Working in these walls of Time;
Some with massive deeds and great,
Some with ornaments of rhyme.
Nothing useless is, or low;
Each thing in its place is best;
And what seems but idle show
Strengthens and supports the rest.
For the structure that we raise,
Time is with materials filled;
Our to-days and yesterdays
Are the blocks with which we build.
Truly shape and fashion these;
Leave no yawning gaps between;
Think not, because no man sees,
Such things will remain unseen.
In the elder days of Art,
Builders wrought with greatest care
Each minute and unseen part;
For the Gods see everywhere.
Let us do our work as well,
Both the unseen and the seen;
Make the house, where Gods may dwell,
Beautiful, entire, and clean.
Else our lives are incomplete,
Standing in these walls of Time,
Broken stairways, where the feet
Stumble as they seek to climb.
Build to-day, then, strong and sure,
With a firm and ample base;
And ascending and secure
Shall to-morrow find its place.
Thus alone can we attain
To those turrets, where the eye
Sees the world as one vast plain,
And one boundless reach of sky.
Each year the exhortation to Let us do our work as well becomes harder to follow, both because the pace of ML keeps increasing, and because more students show up to my lab so my attention is split between more projects. This creates pressure to triage and to leave some broken stairways as they are.
I’m sure that I inevitably do leave some stairways broken—questioning experiments in less detail than I would have as a PhD student, or leaving weak or ambiguous arguments in a paper because we ran out of time before the deadline. That being said, I strongly desire to build a culture that is worthy of the elder days of Art.
Is it possible to do this without permanently sacrificing scalability? I think yes, for the following reasons:
Standards propagate into culture. If you take the time to teach high standards to junior students, they will eventually become senior students and pass those standards down to the next generation. So, mending broken stairways is an investment that lets newcomers climb higher faster.
Scaling through processes. As a manager, catching bugs by exhaustively examining each experiment is not only time-consuming, it’s also error-prone. Instead I need to build processes where students can catch their own errors—through code reviews, standard sanity checks, a culture of following up on inconsistencies, etc. I’m still learning the best ways to do this (if you have ideas please comment!) but in the long-term I think this will simultaneously be more scalable and higher-quality.
Perhaps the common thread is that scaling and high standards may seem at odds in the short-term, but in the longer term they are aligned: you can’t scale well without also having processes in place to create consistently high-quality work.
As someone who has also struggled with similar issues, although in a different context than writing papers, I found some of the answers here helpful and could imagine some of them as good “tactical advice” to go along with cultural norms. I also ended up looking through Google’s SRE book as recommended in Gwern’s answer and benefited from it even though it’s focused on software infrastructure. In particular, the idea of treating knowledge production as a complex system helped knock me out of my “just be careful” mindset, which I think is often one of the harder things to scale. Of course, YMMV.
Thanks, really appreciate the references!