Ah! Were you the one who decorated the Rose Garden Inn? I’m really curious how you made the lighting that looks like the sun coming through a cracked door / coming through the cracks between bricks.
Picture below—I took a ton of pictures when I was there to steal your interior decoration ideas.
Aside which the original author may be interested in—there has been some work done to reduce the scaling of the context window below O(n^2) -- e.g. https://arxiv.org/pdf/1904.10509v1.pdf. I also think of OpenAI’s jukebox which uses a hierarchical strategy in addition to factorized self-attention for generating tokens to effectively increase the context window (https://openai.com/blog/jukebox/)
Gotcha, and thank you so much for writing this post!