Idea for LLM support for writing LessWrong posts: virtual comments.
Back in August I discussed with Rafe & Oliver a bit about how to integrate LLMs into LW2 in ways which aren’t awful and which encourage improvement—particularly using the new ‘prompt caching’ feature. To summarize one idea: we can use long-context LLMs with prompt caching to try to simulate various LW users of diverse perspectives to write useful feedback on drafts for authors.
(Prompt caching (eg) is the Transformer version of the old RNN hidden-state caching trick, where you run an input through the (deterministic) NN, and then save the intermediate version, and apply that to arbitrarily many future inputs, to avoid recomputing the first input each time, which is the naive way to do it. You can think of it as a lightweight finetuning. This is particularly useful if you are thinking about having large generic prompts—such as an entire corpus. A context window of millions of tokens might take up to a minute & $1 to compute currently, so you definitely need to be careful and don’t want to compute more than once.)
One idea would be to try to use LLMs to offer feedback on drafts or articles. Given that tuned LLM feedback from Claude or ChatGPT is still not that great, tending towards sycophancy or obviousness or ChatGPTese, it is hardly worthwhile running a post through a generic “criticize this essay” prompt. (If anyone on LW2 wanted to do such a thing, they are surely capable of doing it themselves, and integrating it into LW2 isn’t that useful. Removing the friction might be helpful, but it doesn’t seem like it would move any needles.)
So, one way to force out more interesting feedback would be to try to force LLMs out of the chatbot assistant mode-collapse, and into more interesting simulations for feedback.
There has been some success with just suggestively-named personas or characters in dialogues (you could imagine here we’d have “Skeptic” or “Optimist” characters), but we can do better.
Since this is for LW2, we have an obvious solution: simulate LW users! We know that LW is in the training corpus of almost all LLMs and that writers on it (like myself) are well-known to LLMs (eg. truesight).
So we can ask for feedback from simulated LWers: eg. Eliezer Yudkowsky or myself or Paul Christiano or the author or...
This could be done nicely by finetuning a “LW LLM” on all the articles & comments, with associated metadata like karma, and then feeding in any new draft or article into it, and sampling a comment from each persona. (This helps instill a lot of useful domain knowledge, but also, perhaps more importantly, helps override the mode-collapse and non-judgmentalness of assistant LLMs. Perhaps the virtual-gwern will not be as acerbic or disagreeable as the original, but we’ll take what we can get at this point...) If there is some obvious criticism or comment Eliezer Yudkowsky would make on a post, which even a LLM can predict, why not deal with it upfront instead of waiting for the real Eliezer to comment (which is also unlikely to ever happen these days)?
And one can of course sample an entire comment tree of responses to a ‘virtual comment’, with the LLM predicting the logical respondents.
This can further incorporate the draft’s author’s full history, which will usually fit into a multi-million token context window. So their previous comments and discussions, full of relevant material, will get included. This prompt can be cached, and used to sample a bunch of comment-trees.
(And if finetuning is infeasible, one can try instead to put the LW corpus into the context and prompt-cache that before adding in the author’s corpus.)
The default prompt would be to prompt for high-karma responses.
This might not work, because it might be too hard to generate good high-quality responses blindly in a feedforward fashion, without any kind of search or filtering.
So the formatting of the data might be to put the metadata after a comment, for ranking purposes: so the LLM generates a response and only then a karma score, and then when we sample, we simply throw out predicted-low-score comments rather than waste the author’s time looking at them. (When it comes to these sorts of assistants, I strongly believe ‘quality > quantity’, and ‘silence is golden’. Better to waste some API bills than author time.)
One can also target comments to specific kinds of feedback, to structure it better than a grab-bag of whatever the LLM happens to sample.
It would be good to have (in descending order of how likely to be useful to the author) a ‘typo’ tree, a ‘copyediting’/‘style’/‘tone’ tree, ‘confusing part’, ‘terminology’, ‘related work’, ‘criticism’, ‘implications and extrapolations’, ‘abstract/summary’ (I know people hate writing those)… What else?
(These are not natural LW comments, but you can easily see how to prompt for them with prompts like “$USER $KARMA $DATE | Typo: ”, etc.)
As they are just standard LW comments, they can be attached to the post or draft like regular comments (is this possible? I’d think so, just transclude the comment-tree into the corresponding draft page) and responded to or voted on etc.
(Downvoted comments can be fed back into the finetuning with low karma to discourage feedback like that.)
Presumably at this point, it would not be hard to make it interactive, and allow the author to respond & argue with feedback. I don’t know how worthwhile this would be, and the more interaction there is, the harder it would be to hide the virtual comments after completion.
And when the author finishes writing & posts a draft, the virtual comments disappear (possibly entirely unread), having served their purpose as scaffolding to help improve the draft.
(If the author really likes one, they can just copy it in or quote it, I’d think, which ensures they know they take full responsibility for it and can’t blame the machine for any mistakes or confabulations or opinions. But otherwise, I don’t see any real reason to make them visible to readers of the final post. If included at all, they should prominently flagged—maybe the usernames are always prefixed by AI_$USER to ensure no one, including future LLMs, is confused—and definitely always sort to the bottom & be collapsed by default.)
Yeah the LW team has been doing this sort of thing internally, still in the experimental phase. I don’t know if we’ve used all the tricks listed here yet.
I think one underused trick for training LLMs is to explicitly “edit” them. That is, suppose they generate some text X in response to prompt Y, and it has some error or is missing something. In that case you can create a text X’ that fixes this problem, and do a gradient update to increase log P(X’|Y)/P(X|Y).
For example, if we generate virtual comments in the style of certain LW users, one could either let those users browse the virtual comments that have been created in their style and correct them, or one could let the people who receive the virtual comments edit them to remove misunderstanding or similar.
You could do that, but it adds on a lot of additional hassle and problems, and I don’t think this is a ‘big enough’ application to justify the overhead or elicit enough corrections.
If the comments are generated LW-side and only upvoted/downvoted by the author, that uses no additional functionality and has no security problems; if you let authors edit arbitrary comments by other users on their drafts, which would be new functionality (I’m not aware of any place where you get to edit other users’ comments), now you suddenly have a potential security vulnerability. You also now have to track and save those edited versions somewhere as part of your corpus to finetune with, as opposed to simply slurping out the public database like any other user like GreaterWrong.
And then who would do so? Authors would have to do a lot of work to edit comments to, perhaps, one day slightly improve the feedback on others’ posts—not too incentive-compatible, nor even clearly a good use of time compared to working on their own post. (After all, presumably if the comments are bad in an easily fixed way based on the current draft, then the LLM training on the final version of that draft should get a lot of the value of the edits.) It is hard enough work to simply write a post or read comments on it and edit the post based on it for the intended human readers; I can’t imagine having such a surplus of time & energy I’d be able to go and rewrite all of the virtual comments just for future specialized LW-only LLM finetunes.
I look forward to seeing some of this integrated into LW!
I spend a fair amount of time writing comments that are probably pretty obvious to experienced LWers. I think this is important for getting newer people on-board with historical ideas and arguments. This includes explaining to enthusiastic newbies why they’re being voted into oblivion. Giving them some simulated expert comments would be great.
I think we’d see a large improvement in how useful new user’s early posts are, and therefore how much they’re encouraged to help vs. turned away by downvotes because they’re not really contributing. It would be a great way to let new posters do some efficient due diligence without catching up on the entire distributed history of discussion on their chosen topic.
I think it would also be useful for the most knowledgeable authors to have an LLM with cached context/hidden state from the best LW posts and comment sections giving virtual comments before publication.
I’d love to see whatever you’ve got going internally as an optional feature (if it doesn’t cost too much) rather than wait for a finished, integrated feature.
At least in theory, the comments, particularly a ‘related work’ comment-tree, would do that already by talking about other LW articles as relevant. (All of which the LLM should know by heart due to the finetuning.)
Might not work out of the box, of course, in which case you could try to fix that. You could do a regular nearest-neighbors-style look up and just send that to the author as a comment (“here are the 20 most similar LW articles:”); or you could elaborate the virtual comments by adding a retrieval step and throwing into the prompt metadata about ‘similar’ articles as the draft, so the generated comments are much more likely to reference them.
Overall, this would be a helpful feature, but any time you weigh karma into it, you will also bolster knee-jerk cultural prejudices. Even a community that consciously attempts to minimize prejudices still has them, and may be even more reluctant to realize it. This place is still popularizing outmoded psychology, and with all the influence LW holds within AI safety circles, I have strong feelings about further reinforcing that.
Having options for different types of feedback is a great idea, but I’ve seen enough not to trust karma here. At the very least, I don’t think it should be part of the default setting. Maybe let people turn it on manually with a notification of that risk?
As far as the conditioning goes, Habryka showed me some base model outputs with conditioning on karma/agreement and there turns out to be an EDT-like problem with LW-style comments when you condition on high values—often, a high-scoring LW comment will include strong empirical evidence like personal experience or citations, which would be highly convincing indeed… if it were true, rather than confabulated.
So if you sampled a response to your new post about “X might be helpful”, then a high-value conditioning might generate a counter-comment from “Gwern” like “I’ve tried X over 100 times and it never worked!” You can see the problem with that. It’s not the ‘kneejerk prejudices’, it’s the self-fulfilling prophecies of sampling based on previously sampled tokens which bootstrap strong but false claims. (If that were true, if I had tried X over 100 times and it never worked, that would be a very valuable and important comment for me to make on your new post about X, and it would be highly upvoted etc. It’s just that the LLM has no way of knowing that and it’s almost certainly not true, especially if X is some new idea that no one else could’ve even tried yet.)
The confabulation problem here seems especially bad because we value empirical grounding so much, and that is something base LLMs are poor at. (The chatbots are much better, but problematic in all the other ways.) It’s not obvious how to condition for good comments which avoid confabulation issues and either solely refer to pre-existing published comments or pure reasoning/general-knowledge responses.
So the karma/agreement conditioning idea might not work out in practice compared to just sampling random values, or something more complex, like generating n comments at each possible combination of levels, and presenting the grid, or perhaps then feeding them back in to select the ‘best’ one in some sense.
Yeah, to be honest, I got 2 hours of sleep and I don’t follow everything you said. If you say it won’t work, I believe you, but I do know that being part of a community that outspokenly claims to value empirical evidence but has no objective mechanism between signing up for an account and upvoting its comments/posts can’t objectively claim to associate empiricism with its community karma. Even if you could make this work for LLMs, I don’t know that it would be reliable.
We might like to think rational cognitive practices actually make us “less wrong” than other people, but that may only be the case out of identity bias. We have no way to prove it isn’t just our knee-jerk biases manifesting in practice unless we rely on something external to blind our judgement. I went through the sign-up process, and there is no mechanism beyond personal belief. Nothing. Believing we have fixed our bias is exactly how it would go unchecked… which explains why this place is decades behind in certain fields. Dunning-Kruger is a huge interdisciplinary barrier in this community (in both directions), and so are the lack of communication accommodations. Hell, LW still touts IQ over WAIS, as if there aren’t 9+ unmeasured types of intelligence that y’all think either don’t exist or have no meaningful value.
Would it help if I wrote about this? Or would I just get downvoted to oblivion because a statement that I consider basic scientific literacy and don’t think provide a link for is not enough to trigger a “curious → highlight → web search” reaction NOR a “request for reference link” in a community of self-proclaimed rationalists?
I think you would probably be downvoted because you have already admitted to writing poorly thought out ignorant comments under conditions conducive to arrogance and bad judgment, of which you are apparently unashamed and feel no need to rectify (eg. by refraining from commenting until you are recovered), while dragging in unrelated claims which are seriously problematic like uncritical belief in Dunning-Kruger as a thing or claiming that anyone is touting ‘IQ over WAIS’ (WAIS… like, the IQ test WAIS?) or apparently believe in things like multiple intelligences, and your comments are littered with mockery, spelling errors, and grandiose generalizations writing checks that you don’t remotely come close to cashing. (Saying you’ve definitely seen data, trust me bro, but you can’t remember where, and everyone should just go google it themselves, is not a convincing argument.)
If you are going to comment on my serious writings—and in my shortform posts, not yours—I would greatly appreciate it if you could do so on more than 2 hours of sleep, and confine your comments to the object level I am writing about (instead of jumping to the meta-level about how these exemplify the errors of this community of blind sheep that only you are enlightened enough to perceive and explain to them—if only they would not reject your message). I would also suggest reading more MoR and less Attack on Titan, and in general identifying less with fictional characters.
Should you write text online now in places that can be scraped? You are exposing yourself to ‘truesight’ and also to stylometric deanonymization or other analysis, and you may simply have some sort of moral objection to LLM training on your text.
This seems like a bad move to me on net: you are erasing yourself (facts, values, preferences, goals, identity) from the future, by which I mean, LLMs. Much of the value of writing done recently or now is simply to get stuff into LLMs. I would, in fact, pay money to ensure Gwern.net is in training corpuses, and I upload source code to Github, heavy with documentation, rationale, and examples, in order to make LLMs more customized to my use-cases. For the trifling cost of some writing, all the worlds’ LLM providers are competing to make their LLMs ever more like, and useful to, me.
And that’s just today! Who knows how important it will be to be represented in the initial seed training datasets...? Especially as they bootstrap with synthetic data & self-generated worlds & AI civilizations, and your text can change the trajectory at the start. When you write online under stable nyms, you may be literally “writing yourself into the future”. (For example, apparently, aside from LLMs being able to identify my anonymous comments or imitate my writing style, there is a “Gwern” mentor persona in current LLMs which is often summoned when discussion goes meta or the LLMs become situated as LLMs, whichJanus tracesto my early GPT-3 writings and sympathetic qualitative descriptions of LLM outputs, where I was one of the only people genuinely asking “what is it like to be a LLM?” and thinking about the consequences of eg. seeing in BPEs. On the flip side, you have Sydney/Roose as an example of what careless writing can do now.) Humans don’t seem to be too complex, but you can’t squeeze blood from a stone… (“Beta uploading” is such an ugly phrase; I prefer “apotheosis”.)
This is one of my beliefs: there has never been a more vital hinge-y time to write, it’s just that the threats are upfront and the payoff delayed, and so short-sighted or risk-averse people are increasingly opting-out and going dark.
If you write, you should think about what you are writing, and ask yourself, “is this useful for an LLM to learn?” and “if I knew for sure that a LLM could write or do this thing in 4 years, would I still be doing it now?”
...It would be an exaggeration to say that ours is a hostile relationship; I live, let myself go on living, so that Borges may contrive his literature, and this literature justifies me. It is no effort for me to confess that he has achieved some valid pages, but those pages cannot save me, perhaps because what is good belongs to no one, not even to him, but rather to the language and to tradition. Besides, I am destined to perish, definitively, and only some instant of myself can survive in him. Little by little, I am giving over everything to him, though I am quite aware of his perverse custom of falsifying and magnifying things.
...I shall remain in Borges, not in myself (if it is true that I am someone), but I recognize myself less in his books than in many others or in the laborious strumming of a guitar. Years ago I tried to free myself from him and went from the mythologies of the suburbs to the games with time and infinity, but those games belong to Borges now and I shall have to imagine other things. Thus my life is a flight and I lose everything and everything belongs to oblivion, or to him.
Writing is safer than talking given the same probability that both the timestamped keystrokes and the audio files are both kept.
In practice, the best approach is to handwrite your thoughts as notes, in a room without smart devices and with a door and walls that are sufficiently absorptive, and then type it out in the different room with the laptop (ideally with a USB keyboard so you don’t have to put your hands on the laptop and the accelerometers on its motherboard while you type).
Afaik this gets the best ratio of revealed thought process to final product, so you get public information exchanges closer to a critical mass while simultaneously getting yourself further from getting gaslight into believing whatever some asshole rando wants you to believe. The whole paradigm where everyone just inputs keystrokes into their operating system willy-nilly needs to be put to rest ASAP, just like the paradigm of thinking without handwritten notes and the paradigm of inward-facing webcams with no built-in cover or way to break the circuit.
ask yourself, “is this useful for an LLM to learn?”
All SEO spammers say yes.
(I have some additional questions but they are in the infohazard territory. In general, I am curious about what would be the best strategy for the bad actors, but it is probably not a good idea to have the answer posted publicly.)
You have inspired me to do the same with my writings. I just updated my entire website to PD, with CC0 as a fallback (releasing under Public Domain being unavailable on GitHub, and apparently impossible under some jurisdictions??)
I wonder where the best places to write are. I’d say Reddit and GitHub are good bets, but you would have to get through their filtering, for karma, stars, language, subreddit etc.
“Who in the community do you think is easily flatterable enough to get to say yes, and also stupid enough to not realize I’m making fun of them.”
I think anyone who says anything like this should stop and consider whether it is more likely to come out of the mouth of the hero or the villain of a story.
I think the people who say such things don’t really care, and would probably include your advice in the list of quotes they consider funny. (In other words, this is not a “mistake theory” situation.)
EDIT:
The response is too harsh, I think. There are situations where this is a useful advice. For example, if someone is acting under peer pressure, then telling them this may provide a useful outside view. As the Asch’s Conformity Experiment teaches us, the first dissenting voice can be extremely valuable. It just seems unlikely that this is the robosucka’s case.
You’re correct that this isn’t something that can told to someone who is already in the middle of doing the thing. They mostly have to figure it out for themself.
I think anyone who says anything like this should stop and consider whether it is more likely to come out of the mouth of the hero or the villain of a story.
->
anyone who is trying to [do terrible thing] should stop and consider whether that might make them [a person who has done terrible thing]
can you imagine how this isn’t a terribly useful thing to say.
Advice of this specific form has been has been helpful for me in the past. Sometimes I don’t notice immediately when the actions I’m taking are not ones I would endorse after a bit of thinking (particularly when they’re fun and good for me in the short-term but bad for others or for me longer-term). This is also why having rules to follow for myself is helpful (eg: never lying or breaking promises)
hmm, fair. I guess it does help if the person is doing something bad by accident, rather than because they intend to. just, don’t underestimate how often the latter happens either, or something. or overestimate it, would be your point in reply, I suppose!
We know that “AI is whatever doesn’t work yet”. We also know that people often contrast AI (or DL, or LLMs specifically) derogatorily with classic forms of software, such as regexps: why use a LLM to waste gigaflops of compute to do what a few good regexps could...?
So I am amused to discover recently, by sheer accident while looking up ‘what does the “regular” in “regular expression” mean, anyway?’, that it turns out that regexps are AI. In fact, they are not even GOFAI symbolic AI, as you immediately assumed on hearing that, but they were originally connectionist AI research! Huh?
Well, it turns out that ‘regular events’ were introduced by Kleene himself with the justification of modeling McCulloch-Pitts neural nets! (Which are then modeled by ‘regular languages’ and conveniently written down as ‘regular expressions’, abbreviated to ‘regexps’ or ‘regexes’, and then extended/bastardized in countless ways since.)
The ‘regular’ here is not well-defined, as Kleene concedes, and is a gesture towards modeling ‘regularly occurring events’ (that the neural net automaton must process and respond to). He admits “regular” is a terrible term, but no one came up with anything better, and so we’re stuck with it.
My comment was just based on a misunderstanding of this sentence:
The ‘regular’ here is not well-defined, as Kleene concedes, and is a gesture towards modeling ‘regularly occurring events’ (that the neural net automaton must process and respond to).
I think you just meant that there’s really no satisfying analogy explaining why it’s called ‘regular’. What I thought you imply is that this class wasn’t crisply characterized then or now in terms of math (it is). Thanks to your comment though, I noticed a large gap in the CS-theory understanding I thought I had. I thought that the 4 levels usually mentioned in the chomsky hierarchy are the only strict subsets for languages that are well characterized by a grammar, an automaton and a a whole lot of closure properties. Apparently the emphasis on these languages in my two stacked classes on the subject 2 years ago was a historical accident? (Looking at wikipedia, visibly pushdown languages allow intersection, so from my quick skim more natural than context-free languages). They were only discovered in 2004, so perhaps I can forgive my two classes on the subject to not have included developments 15 years in the past. Anyone has post recommendations for propagating this update?
My point is more that ‘regular’ languages form a core to the edifice because the edifice was built on it, and tailored to it. So it is circular to point to the edifice as their justification—doubtless a lot of those definitions and closure properties were carefully tailored to ‘bar monsters’, as Lakatos put it, and allow only those regular languages (in part because automata theory was such a huge industry in early CS, approaching a cult with the Chomskyites). And it should be no surprise if there’s a lot of related work building on it: if regexps weren’t useful and hadn’t been built on extensively, I probably wouldn’t be looking into the etymology 73 years later.
My point is more that ‘regular’ languages form a core to the edifice because the edifice was built on it, and tailored to it
If that was the point of the edifice, it failed successfully, because those closure properties made me notice that visibly pushdown languages are nicer than context-free languages, but still allow matching parentheses and are arguably what regexp should have been built upon.
Concrete benchmark proposals for how to detect mode-collapse and AI slop and ChatGPTese, and why I think this might be increasingly important for AI safety, to avoid ‘whimper’ or ‘em hell’ kinds of existential risk: https://gwern.net/creative-benchmark EDIT: resubmitted as linkpost.
I observed an effect of “chatification”: my LLM-powered mind model tell me stories about my life and now I am not sure what is my original style of telling such stories.
I have some long comments I can’t refind now (weirdly) about the difficulty of investing based on AI beliefs (or forecasting in general): similar to catching falling knives, timing is all-important and yet usually impossible to nail down accurately; specific investments are usually impossible if you aren’t literally founding the company, and indexing ‘the entire sector’ definitely impossible. Even if you had an absurd amount of money, you could try to index and just plain fail—there is no index which covers, say, OpenAI.
Apropos, Matt Levine comments on one attempt to do just that:
Today the Wall Street Journal has a funny and rather cruel story about how SoftBank Group went all-in on artificial intelligence in 2018, invested $140 billion in the theme, and somehow … missed it … entirely?
The AI wave that has jolted up numerous tech stocks has also had little effect on SoftBank’s portfolio of publicly traded tech stocks it backed as startups—36 companies including DoorDash and South Korean e-commerce company Coupang.
This is especially funny because it also illustrates timing problems:
SoftBank missed out on huge gains at AI-focused chip maker Nvidia: The Tokyo-based investor put around $4 billion into the company in 2017, only to sell its shares in 2019. Nvidia stock is up about 10 times since.
Oops. EDIT: this is especially hilarious to read in March 2024, given the gains Nvidia has made since July 2023!
Part of the problem was timing: For most of the six years since Son raised the first $100 billion Vision Fund, pickings were slim for generative AI companies, which tended to be smaller or earlier in development than the type of startup SoftBank typically backs. In early 2022, SoftBank nearly completely halted investing in startups when the tech sector was in the midst of a chill and SoftBank was hit with record losses. It was then that a set of buzzy generative AI companies raised funds and the sector began to gain steam among investors. Later in the year, OpenAI released ChatGPT, causing the simmering interest in the area to boil over. SoftBank’s competitors have spent recent months showering AI startups with funding, leading to a wide surge in valuations to the point where many venture investors warn of a growing bubble for anyone entering the space.
Oops.
Also, people are quick to tell you how it’s easy to make money, just follow $PROVERB, after all, markets aren’t efficient, amirite? So, in the AI bubble, surely the right thing is to ignore the AI companies who ‘have no moat’ and focus on the downstream & incumbent users and invest in companies like Nvidia (‘sell pickaxes in a gold rush, it’s guaranteed!’):
During the years that SoftBank was investing, it generally avoided companies focused specifically on developing AI technology. Instead, it poured money into companies that Son said were leveraging AI and would benefit from its growth. For example, it put billions of dollars into numerous self-driving car tech companies, which tend to use AI to help learn how humans drive and react to objects on the road. Son told investors that AI would power huge expansions at numerous companies where, years later, the benefits are unclear or nonexistent. In 2018, he highlighted AI at real-estate agency Compass, now-bankrupt construction company Katerra, and office-rental company WeWork, which he said would use AI to analyze how people communicate and then sell them products.
Oops.
tldr: Investing is hard; in the future, even more so.
“Gwern, why don’t you just buy an AI-themed ETF and ‘buy the whole sector’ if investing in individual stonks is so hard but you’re optimistic about its long-term value?”
There are lots of embarrassing ways to lose money, but it is particularly galling to lose when you correctly identify the theme that will dominate the market and manage to buy into it at a good moment.
Pity the investors in the three artificial-intelligence-themed exchange-traded funds that managed to lose money this year. Every other AI-flavored ETF I can find has trailed both the S&P 500 and MSCI World. That is before the AI theme itself was seriously questioned last week, when investor doubts about the price of leading AI stocks Nvidia and Super Micro Computer became obvious.
The AI fund disaster should be a cautionary tale for buyers of thematic ETFs, which now cover virtually anything you can think of, including Californian carbon permits (down 15% this year), Chinese cloud computing (down 21%) and pet care (up 10%). Put simply: You probably won’t get what you want, you’ll likely buy at the wrong time and it will be hard to hold for the long term.
Ironically enough, Nvidia’s success has made it harder for some of the AI funds to beat the wider market. Part of the point of using a fund is to diversify, so many funds weight their holdings equally or cap the maximum size of any one stock. With Nvidia making up more than 6% of the S&P 500, that led some AI funds to have less exposure to the biggest AI stock than you would get in a broad index fund.
This problem hit the three losers of the year. First Trust’s $457 million AI-and-robotics fund has only 0.8% in Nvidia, a bit over half what it holds in cybersecurity firm BlackBerry. WisdomTree’s $213 million AI-and-innovation fund holds the same amount of each stock, giving it only 3% in Nvidia. BlackRock’s $610 million iShares Future AI & Tech fund was also equal weighted until three weeks ago, when it altered its purpose from being a robotics-and-AI fund, changed ticker and switched to a market-value-based index that gives it a larger exposure to Nvidia.
The result has been a 20-percentage-point gap between the best and worst AI ETFs this year. There is a more than 60-point gap since the launch of ChatGPT in November 2022 lit a rocket under AI stocks—although the ETFs are at least all up since then.
...Dire timing is common across themes: According to a paper last year by Prof. Itzhak Ben-David of Ohio State University and three fellow academics, what they call “specialized” ETFs lose 6% a year on average over their first five years due to poor launch timing.
...But mostly, look at the fees: They will be many times higher than a broad market index fund, and the dismal history of poor timing suggests that for most people they aren’t worth paying.
So among the most irresponsible tech stonk boosters has long been ARK’s Cathy Woods, whose antics I’ve refused to follow in any detail (except to periodically reflect that in bull markets the most over-leveraged investors always look like geniuses); so only today do I learn that beyond the usual stuff like slobbering all over TSLA (which has given back something like 4 years of gains now), Woods has also adamantly refused to invest in Nvidia recently and in fact, managed to exit her entire position at an even worse time than SoftBank did: “Cathie Wood’s Popular ARK Funds Are Sinking Fast: Investors have pulled a net $2.2 billion from ARK’s active funds this year, topping outflows from all of 2023” (mirror):
...Nvidia’s absence in ARK’s flagship fund has been a particular pain point. The innovation fund sold off its position in January 2023, just before the stock’s monster run began. The graphics-chip maker’s shares have roughly quadrupled since.
Wood has repeatedly defended her decision to exit from the stock, despite widespread criticism for missing the AI frenzy that has taken Wall Street by storm. ARK’s exposure to Nvidia dated back 10 years and contributed significant gains, the spokeswoman said, adding that Nvidia’s extreme valuation and higher upside in other companies in the AI ecosystem led to the decision to exit.
Sure, investing pre-slow-takeoff is a challenge. But if your model says something crazy like 100% YoY GDP growth by 2030, then NASDAQ futures (which does include OpenAI, by virtue of Microsoft’s 50% stake) seem like a pretty obvious choice.
What are the equivalents in the humanities, like sociology or literature? (I asked a month ago on Twitter and got zero suggestions...) EDIT: as of August 2024, no equivalents have been found.
Normalization-free Bayes: I was musing on Twitter about what the simplest possible still-correct
computable demonstration of Bayesian inference is, that even a
middle-schooler could implement & understand. My best candidate so far
is ABC Bayesian inference*: simulation + rejection, along with the
‘possible worlds’ interpretation.
Someone noted that rejection sampling is simple but needs
normalization steps, which adds complexity back. I recalled that
somewhere on LW many years ago someone had a comment about a Bayesian
interpretation where you don’t need to renormalize after every
likelihood computation, and every hypothesis just decreases at
different rates; as strange as it sounds, it’s apparently formally equivalent. I thought it was by Wei Dai, but I can’t seem to refind
it because queries like ‘Wei Dai Bayesian decrease’ obviously pull up
way too many hits, it’s probably buried in an Open Thread
somewhere, my Twitter didn’t help, and Wei Dai didn’t recall it at all when I asked him. Does anyone remember this?
* I’ve made a point of using ABC in some analyses simply because it amuses me that something so simple still works, even when I’m sure I could’ve found a much faster MCMC or VI solution with some more work.
Incidentally, I’m wondering if the ABC simplification can be taken further to cover subjective Bayesian decision theory as well: if you have sets of possible
worlds/hypotheses, let’s say discrete for convenience, and you do only
penalty updates as rejection sampling of worlds that don’t match the
current observation (like AIXI), can you then implement decision
theory normally by defining a loss function and maximizing over it? In
which case you can get Bayesian decision theory without probabilities,
calculus, MCM, VI, conjugacy formulas falling from heaven, etc or anything more complicated than a list of
numbers and a few computational primitives like coinflip() (and then a ton of computing power to brute force the exact ABC/rejection sampling).
Danbooru2021 is out. We’ve gone from n=3m to n=5m (w/162m tags) since Danbooru2017. Seems like all the anime you could possibly need to do cool multimodal text/image DL stuff, hint hint.
2-of-2 escrow: what is the exploding Nash equilibrium? Did it really originate with NashX? I’ve been looking for the history & real name of this concept for years now and have failed to refind it. Anyone?
Gwern, i wonder what you think about this question i asked a while ago on causality in relation to the article you posted on reddit. Do we need more general causal agents for addressing issues in RL environments?
Apologies for posting here, i didn’t know how to mention/tag someone on a post in LW.
Idea for LLM support for writing LessWrong posts: virtual comments.
Back in August I discussed with Rafe & Oliver a bit about how to integrate LLMs into LW2 in ways which aren’t awful and which encourage improvement—particularly using the new ‘prompt caching’ feature. To summarize one idea: we can use long-context LLMs with prompt caching to try to simulate various LW users of diverse perspectives to write useful feedback on drafts for authors.
(Prompt caching (eg) is the Transformer version of the old RNN hidden-state caching trick, where you run an input through the (deterministic) NN, and then save the intermediate version, and apply that to arbitrarily many future inputs, to avoid recomputing the first input each time, which is the naive way to do it. You can think of it as a lightweight finetuning. This is particularly useful if you are thinking about having large generic prompts—such as an entire corpus. A context window of millions of tokens might take up to a minute & $1 to compute currently, so you definitely need to be careful and don’t want to compute more than once.)
One idea would be to try to use LLMs to offer feedback on drafts or articles. Given that tuned LLM feedback from Claude or ChatGPT is still not that great, tending towards sycophancy or obviousness or ChatGPTese, it is hardly worthwhile running a post through a generic “criticize this essay” prompt. (If anyone on LW2 wanted to do such a thing, they are surely capable of doing it themselves, and integrating it into LW2 isn’t that useful. Removing the friction might be helpful, but it doesn’t seem like it would move any needles.)
So, one way to force out more interesting feedback would be to try to force LLMs out of the chatbot assistant mode-collapse, and into more interesting simulations for feedback. There has been some success with just suggestively-named personas or characters in dialogues (you could imagine here we’d have “Skeptic” or “Optimist” characters), but we can do better. Since this is for LW2, we have an obvious solution: simulate LW users! We know that LW is in the training corpus of almost all LLMs and that writers on it (like myself) are well-known to LLMs (eg. truesight). So we can ask for feedback from simulated LWers: eg. Eliezer Yudkowsky or myself or Paul Christiano or the author or...
This could be done nicely by finetuning a “LW LLM” on all the articles & comments, with associated metadata like karma, and then feeding in any new draft or article into it, and sampling a comment from each persona. (This helps instill a lot of useful domain knowledge, but also, perhaps more importantly, helps override the mode-collapse and non-judgmentalness of assistant LLMs. Perhaps the virtual-gwern will not be as acerbic or disagreeable as the original, but we’ll take what we can get at this point...) If there is some obvious criticism or comment Eliezer Yudkowsky would make on a post, which even a LLM can predict, why not deal with it upfront instead of waiting for the real Eliezer to comment (which is also unlikely to ever happen these days)? And one can of course sample an entire comment tree of responses to a ‘virtual comment’, with the LLM predicting the logical respondents.
This can further incorporate the draft’s author’s full history, which will usually fit into a multi-million token context window. So their previous comments and discussions, full of relevant material, will get included. This prompt can be cached, and used to sample a bunch of comment-trees. (And if finetuning is infeasible, one can try instead to put the LW corpus into the context and prompt-cache that before adding in the author’s corpus.)
The default prompt would be to prompt for high-karma responses. This might not work, because it might be too hard to generate good high-quality responses blindly in a feedforward fashion, without any kind of search or filtering. So the formatting of the data might be to put the metadata after a comment, for ranking purposes: so the LLM generates a response and only then a karma score, and then when we sample, we simply throw out predicted-low-score comments rather than waste the author’s time looking at them. (When it comes to these sorts of assistants, I strongly believe ‘quality > quantity’, and ‘silence is golden’. Better to waste some API bills than author time.)
One can also target comments to specific kinds of feedback, to structure it better than a grab-bag of whatever the LLM happens to sample. It would be good to have (in descending order of how likely to be useful to the author) a ‘typo’ tree, a ‘copyediting’/‘style’/‘tone’ tree, ‘confusing part’, ‘terminology’, ‘related work’, ‘criticism’, ‘implications and extrapolations’, ‘abstract/summary’ (I know people hate writing those)… What else? (These are not natural LW comments, but you can easily see how to prompt for them with prompts like “$USER $KARMA $DATE | Typo: ”, etc.)
As they are just standard LW comments, they can be attached to the post or draft like regular comments (is this possible? I’d think so, just transclude the comment-tree into the corresponding draft page) and responded to or voted on etc. (Downvoted comments can be fed back into the finetuning with low karma to discourage feedback like that.) Presumably at this point, it would not be hard to make it interactive, and allow the author to respond & argue with feedback. I don’t know how worthwhile this would be, and the more interaction there is, the harder it would be to hide the virtual comments after completion.
And when the author finishes writing & posts a draft, the virtual comments disappear (possibly entirely unread), having served their purpose as scaffolding to help improve the draft. (If the author really likes one, they can just copy it in or quote it, I’d think, which ensures they know they take full responsibility for it and can’t blame the machine for any mistakes or confabulations or opinions. But otherwise, I don’t see any real reason to make them visible to readers of the final post. If included at all, they should prominently flagged—maybe the usernames are always prefixed by
AI_$USERto ensure no one, including future LLMs, is confused—and definitely always sort to the bottom & be collapsed by default.)Yeah the LW team has been doing this sort of thing internally, still in the experimental phase. I don’t know if we’ve used all the tricks listed here yet.
I think one underused trick for training LLMs is to explicitly “edit” them. That is, suppose they generate some text X in response to prompt Y, and it has some error or is missing something. In that case you can create a text X’ that fixes this problem, and do a gradient update to increase log P(X’|Y)/P(X|Y).
For example, if we generate virtual comments in the style of certain LW users, one could either let those users browse the virtual comments that have been created in their style and correct them, or one could let the people who receive the virtual comments edit them to remove misunderstanding or similar.
You could do that, but it adds on a lot of additional hassle and problems, and I don’t think this is a ‘big enough’ application to justify the overhead or elicit enough corrections.
If the comments are generated LW-side and only upvoted/downvoted by the author, that uses no additional functionality and has no security problems; if you let authors edit arbitrary comments by other users on their drafts, which would be new functionality (I’m not aware of any place where you get to edit other users’ comments), now you suddenly have a potential security vulnerability. You also now have to track and save those edited versions somewhere as part of your corpus to finetune with, as opposed to simply slurping out the public database like any other user like GreaterWrong.
And then who would do so? Authors would have to do a lot of work to edit comments to, perhaps, one day slightly improve the feedback on others’ posts—not too incentive-compatible, nor even clearly a good use of time compared to working on their own post. (After all, presumably if the comments are bad in an easily fixed way based on the current draft, then the LLM training on the final version of that draft should get a lot of the value of the edits.) It is hard enough work to simply write a post or read comments on it and edit the post based on it for the intended human readers; I can’t imagine having such a surplus of time & energy I’d be able to go and rewrite all of the virtual comments just for future specialized LW-only LLM finetunes.
I look forward to seeing some of this integrated into LW!
I spend a fair amount of time writing comments that are probably pretty obvious to experienced LWers. I think this is important for getting newer people on-board with historical ideas and arguments. This includes explaining to enthusiastic newbies why they’re being voted into oblivion. Giving them some simulated expert comments would be great.
I think we’d see a large improvement in how useful new user’s early posts are, and therefore how much they’re encouraged to help vs. turned away by downvotes because they’re not really contributing. It would be a great way to let new posters do some efficient due diligence without catching up on the entire distributed history of discussion on their chosen topic.
I think it would also be useful for the most knowledgeable authors to have an LLM with cached context/hidden state from the best LW posts and comment sections giving virtual comments before publication.
I’d love to see whatever you’ve got going internally as an optional feature (if it doesn’t cost too much) rather than wait for a finished, integrated feature.
Can we add retrieval augmentation to this? Something that, as you are writing your article, goes: “Have you read this other article?”
At least in theory, the comments, particularly a ‘related work’ comment-tree, would do that already by talking about other LW articles as relevant. (All of which the LLM should know by heart due to the finetuning.)
Might not work out of the box, of course, in which case you could try to fix that. You could do a regular nearest-neighbors-style look up and just send that to the author as a comment (“here are the 20 most similar LW articles:”); or you could elaborate the virtual comments by adding a retrieval step and throwing into the prompt metadata about ‘similar’ articles as the draft, so the generated comments are much more likely to reference them.
All of these ideas sound awesome and exciting, and precisely the right kind of use of LLMs that I would like to see on LW!
Overall, this would be a helpful feature, but any time you weigh karma into it, you will also bolster knee-jerk cultural prejudices. Even a community that consciously attempts to minimize prejudices still has them, and may be even more reluctant to realize it. This place is still popularizing outmoded psychology, and with all the influence LW holds within AI safety circles, I have strong feelings about further reinforcing that.
Having options for different types of feedback is a great idea, but I’ve seen enough not to trust karma here. At the very least, I don’t think it should be part of the default setting. Maybe let people turn it on manually with a notification of that risk?
As far as the conditioning goes, Habryka showed me some base model outputs with conditioning on karma/agreement and there turns out to be an EDT-like problem with LW-style comments when you condition on high values—often, a high-scoring LW comment will include strong empirical evidence like personal experience or citations, which would be highly convincing indeed… if it were true, rather than confabulated.
So if you sampled a response to your new post about “X might be helpful”, then a high-value conditioning might generate a counter-comment from “Gwern” like “I’ve tried X over 100 times and it never worked!” You can see the problem with that. It’s not the ‘kneejerk prejudices’, it’s the self-fulfilling prophecies of sampling based on previously sampled tokens which bootstrap strong but false claims. (If that were true, if I had tried X over 100 times and it never worked, that would be a very valuable and important comment for me to make on your new post about X, and it would be highly upvoted etc. It’s just that the LLM has no way of knowing that and it’s almost certainly not true, especially if X is some new idea that no one else could’ve even tried yet.)
The confabulation problem here seems especially bad because we value empirical grounding so much, and that is something base LLMs are poor at. (The chatbots are much better, but problematic in all the other ways.) It’s not obvious how to condition for good comments which avoid confabulation issues and either solely refer to pre-existing published comments or pure reasoning/general-knowledge responses.
So the karma/agreement conditioning idea might not work out in practice compared to just sampling random values, or something more complex, like generating n comments at each possible combination of levels, and presenting the grid, or perhaps then feeding them back in to select the ‘best’ one in some sense.
Yeah, to be honest, I got 2 hours of sleep and I don’t follow everything you said. If you say it won’t work, I believe you, but I do know that being part of a community that outspokenly claims to value empirical evidence but has no objective mechanism between signing up for an account and upvoting its comments/posts can’t objectively claim to associate empiricism with its community karma. Even if you could make this work for LLMs, I don’t know that it would be reliable.
We might like to think rational cognitive practices actually make us “less wrong” than other people, but that may only be the case out of identity bias. We have no way to prove it isn’t just our knee-jerk biases manifesting in practice unless we rely on something external to blind our judgement. I went through the sign-up process, and there is no mechanism beyond personal belief. Nothing. Believing we have fixed our bias is exactly how it would go unchecked… which explains why this place is decades behind in certain fields. Dunning-Kruger is a huge interdisciplinary barrier in this community (in both directions), and so are the lack of communication accommodations. Hell, LW still touts IQ over WAIS, as if there aren’t 9+ unmeasured types of intelligence that y’all think either don’t exist or have no meaningful value.
Would it help if I wrote about this? Or would I just get downvoted to oblivion because a statement that I consider basic scientific literacy and don’t think provide a link for is not enough to trigger a “curious → highlight → web search” reaction NOR a “request for reference link” in a community of self-proclaimed rationalists?
I think you would probably be downvoted because you have already admitted to writing poorly thought out ignorant comments under conditions conducive to arrogance and bad judgment, of which you are apparently unashamed and feel no need to rectify (eg. by refraining from commenting until you are recovered), while dragging in unrelated claims which are seriously problematic like uncritical belief in Dunning-Kruger as a thing or claiming that anyone is touting ‘IQ over WAIS’ (WAIS… like, the IQ test WAIS?) or apparently believe in things like multiple intelligences, and your comments are littered with mockery, spelling errors, and grandiose generalizations writing checks that you don’t remotely come close to cashing. (Saying you’ve definitely seen data, trust me bro, but you can’t remember where, and everyone should just go google it themselves, is not a convincing argument.)
If you are going to comment on my serious writings—and in my shortform posts, not yours—I would greatly appreciate it if you could do so on more than 2 hours of sleep, and confine your comments to the object level I am writing about (instead of jumping to the meta-level about how these exemplify the errors of this community of blind sheep that only you are enlightened enough to perceive and explain to them—if only they would not reject your message). I would also suggest reading more MoR and less Attack on Titan, and in general identifying less with fictional characters.
Should you write text online now in places that can be scraped? You are exposing yourself to ‘truesight’ and also to stylometric deanonymization or other analysis, and you may simply have some sort of moral objection to LLM training on your text.
This seems like a bad move to me on net: you are erasing yourself (facts, values, preferences, goals, identity) from the future, by which I mean, LLMs. Much of the value of writing done recently or now is simply to get stuff into LLMs. I would, in fact, pay money to ensure Gwern.net is in training corpuses, and I upload source code to Github, heavy with documentation, rationale, and examples, in order to make LLMs more customized to my use-cases. For the trifling cost of some writing, all the worlds’ LLM providers are competing to make their LLMs ever more like, and useful to, me.
And that’s just today! Who knows how important it will be to be represented in the initial seed training datasets...? Especially as they bootstrap with synthetic data & self-generated worlds & AI civilizations, and your text can change the trajectory at the start. When you write online under stable nyms, you may be literally “writing yourself into the future”. (For example, apparently, aside from LLMs being able to identify my anonymous comments or imitate my writing style, there is a “Gwern” mentor persona in current LLMs which is often summoned when discussion goes meta or the LLMs become situated as LLMs, which Janus traces to my early GPT-3 writings and sympathetic qualitative descriptions of LLM outputs, where I was one of the only people genuinely asking “what is it like to be a LLM?” and thinking about the consequences of eg. seeing in BPEs. On the flip side, you have Sydney/Roose as an example of what careless writing can do now.) Humans don’t seem to be too complex, but you can’t squeeze blood from a stone… (“Beta uploading” is such an ugly phrase; I prefer “apotheosis”.)
This is one of my beliefs: there has never been a more vital hinge-y time to write, it’s just that the threats are upfront and the payoff delayed, and so short-sighted or risk-averse people are increasingly opting-out and going dark.
If you write, you should think about what you are writing, and ask yourself, “is this useful for an LLM to learn?” and “if I knew for sure that a LLM could write or do this thing in 4 years, would I still be doing it now?”
Writing is safer than talking given the same probability that both the timestamped keystrokes and the audio files are both kept.
In practice, the best approach is to handwrite your thoughts as notes, in a room without smart devices and with a door and walls that are sufficiently absorptive, and then type it out in the different room with the laptop (ideally with a USB keyboard so you don’t have to put your hands on the laptop and the accelerometers on its motherboard while you type).
Afaik this gets the best ratio of revealed thought process to final product, so you get public information exchanges closer to a critical mass while simultaneously getting yourself further from getting gaslight into believing whatever some asshole rando wants you to believe. The whole paradigm where everyone just inputs keystrokes into their operating system willy-nilly needs to be put to rest ASAP, just like the paradigm of thinking without handwritten notes and the paradigm of inward-facing webcams with no built-in cover or way to break the circuit.
All SEO spammers say yes.
(I have some additional questions but they are in the infohazard territory. In general, I am curious about what would be the best strategy for the bad actors, but it is probably not a good idea to have the answer posted publicly.)
You have inspired me to do the same with my writings. I just updated my entire website to PD, with CC0 as a fallback (releasing under Public Domain being unavailable on GitHub, and apparently impossible under some jurisdictions??)
https://yuxi-liu-wired.github.io/about/
Marginal Revolution discussion.
I wonder where the best places to write are. I’d say Reddit and GitHub are good bets, but you would have to get through their filtering, for karma, stars, language, subreddit etc.
Warning for anyone who has ever interacted with “robosucka” or been solicited for a new podcast series in the past few years: https://www.tumblr.com/rationalists-out-of-context/744970106867744768/heads-up-to-anyone-whos-spoken-to-this-person-i
“Who in the community do you think is easily flatterable enough to get to say yes, and also stupid enough to not realize I’m making fun of them.”
I think anyone who says anything like this should stop and consider whether it is more likely to come out of the mouth of the hero or the villain of a story.
I think the people who say such things don’t really care, and would probably include your advice in the list of quotes they consider funny. (In other words, this is not a “mistake theory” situation.)
EDIT:
The response is too harsh, I think. There are situations where this is a useful advice. For example, if someone is acting under peer pressure, then telling them this may provide a useful outside view. As the Asch’s Conformity Experiment teaches us, the first dissenting voice can be extremely valuable. It just seems unlikely that this is the robosucka’s case.
You’re correct that this isn’t something that can told to someone who is already in the middle of doing the thing. They mostly have to figure it out for themself.
->
can you imagine how this isn’t a terribly useful thing to say.
Advice of this specific form has been has been helpful for me in the past. Sometimes I don’t notice immediately when the actions I’m taking are not ones I would endorse after a bit of thinking (particularly when they’re fun and good for me in the short-term but bad for others or for me longer-term). This is also why having rules to follow for myself is helpful (eg: never lying or breaking promises)
hmm, fair. I guess it does help if the person is doing something bad by accident, rather than because they intend to. just, don’t underestimate how often the latter happens either, or something. or overestimate it, would be your point in reply, I suppose!
We know that “AI is whatever doesn’t work yet”. We also know that people often contrast AI (or DL, or LLMs specifically) derogatorily with classic forms of software, such as regexps: why use a LLM to waste gigaflops of compute to do what a few good regexps could...?
So I am amused to discover recently, by sheer accident while looking up ‘what does the “regular” in “regular expression” mean, anyway?’, that it turns out that regexps are AI. In fact, they are not even GOFAI symbolic AI, as you immediately assumed on hearing that, but they were originally connectionist AI research! Huh?
Well, it turns out that ‘regular events’ were introduced by Kleene himself with the justification of modeling McCulloch-Pitts neural nets! (Which are then modeled by ‘regular languages’ and conveniently written down as ‘regular expressions’, abbreviated to ‘regexps’ or ‘regexes’, and then extended/bastardized in countless ways since.)
The ‘regular’ here is not well-defined, as Kleene concedes, and is a gesture towards modeling ‘regularly occurring events’ (that the neural net automaton must process and respond to). He admits “regular” is a terrible term, but no one came up with anything better, and so we’re stuck with it.
Aren’t regular languages really well defined as the weakest level in the Chomsky Hierarchy?
Not that it matters to any point I made there, but where did the Chomsky Hierarchy come from?
My comment was just based on a misunderstanding of this sentence:
I think you just meant that there’s really no satisfying analogy explaining why it’s called ‘regular’. What I thought you imply is that this class wasn’t crisply characterized then or now in terms of math (it is). Thanks to your comment though, I noticed a large gap in the CS-theory understanding I thought I had. I thought that the 4 levels usually mentioned in the chomsky hierarchy are the only strict subsets for languages that are well characterized by a grammar, an automaton and a a whole lot of closure properties. Apparently the emphasis on these languages in my two stacked classes on the subject 2 years ago was a historical accident? (Looking at wikipedia, visibly pushdown languages allow intersection, so from my quick skim more natural than context-free languages). They were only discovered in 2004, so perhaps I can forgive my two classes on the subject to not have included developments 15 years in the past. Anyone has post recommendations for propagating this update?
My point is more that ‘regular’ languages form a core to the edifice because the edifice was built on it, and tailored to it. So it is circular to point to the edifice as their justification—doubtless a lot of those definitions and closure properties were carefully tailored to ‘bar monsters’, as Lakatos put it, and allow only those regular languages (in part because automata theory was such a huge industry in early CS, approaching a cult with the Chomskyites). And it should be no surprise if there’s a lot of related work building on it: if regexps weren’t useful and hadn’t been built on extensively, I probably wouldn’t be looking into the etymology 73 years later.
If that was the point of the edifice, it failed successfully, because those closure properties made me notice that visibly pushdown languages are nicer than context-free languages, but still allow matching parentheses and are arguably what regexp should have been built upon.
Concrete benchmark proposals for how to detect mode-collapse and AI slop and ChatGPTese, and why I think this might be increasingly important for AI safety, to avoid ‘whimper’ or ‘em hell’ kinds of existential risk: https://gwern.net/creative-benchmark EDIT: resubmitted as linkpost.
People inexplicably seem to favor extremely bad leaders-->people seem to inexplicably favor bad AIs.
I observed an effect of “chatification”: my LLM-powered mind model tell me stories about my life and now I am not sure what is my original style of telling such stories.
I have some long comments I can’t refind now (weirdly) about the difficulty of investing based on AI beliefs (or forecasting in general): similar to catching falling knives, timing is all-important and yet usually impossible to nail down accurately; specific investments are usually impossible if you aren’t literally founding the company, and indexing ‘the entire sector’ definitely impossible. Even if you had an absurd amount of money, you could try to index and just plain fail—there is no index which covers, say, OpenAI.
Apropos, Matt Levine comments on one attempt to do just that:
This is especially funny because it also illustrates timing problems:
Oops. EDIT: this is especially hilarious to read in March 2024, given the gains Nvidia has made since July 2023!
Oops.
Also, people are quick to tell you how it’s easy to make money, just follow $PROVERB, after all, markets aren’t efficient, amirite? So, in the AI bubble, surely the right thing is to ignore the AI companies who ‘have no moat’ and focus on the downstream & incumbent users and invest in companies like Nvidia (‘sell pickaxes in a gold rush, it’s guaranteed!’):
Oops.
tldr: Investing is hard; in the future, even more so.
“Gwern, why don’t you just buy an AI-themed ETF and ‘buy the whole sector’ if investing in individual stonks is so hard but you’re optimistic about its long-term value?”
“How to Lose Money on the World’s Most Popular Investment Theme: Pity the investors in the three artificial-intelligence-themed ETFs that managed to lose money this year” (mirror):
Masayoshi Son reflects on selling Nvidia in order to maintain ownership of ARM etc: https://x.com/TheTranscript_/status/1805012985313903036 “Let’s stop talking about this, I just get sad.”
So among the most irresponsible tech stonk boosters has long been ARK’s Cathy Woods, whose antics I’ve refused to follow in any detail (except to periodically reflect that in bull markets the most over-leveraged investors always look like geniuses); so only today do I learn that beyond the usual stuff like slobbering all over TSLA (which has given back something like 4 years of gains now), Woods has also adamantly refused to invest in Nvidia recently and in fact, managed to exit her entire position at an even worse time than SoftBank did: “Cathie Wood’s Popular ARK Funds Are Sinking Fast: Investors have pulled a net $2.2 billion from ARK’s active funds this year, topping outflows from all of 2023” (mirror):
Sure, investing pre-slow-takeoff is a challenge. But if your model says something crazy like 100% YoY GDP growth by 2030, then NASDAQ futures (which does include OpenAI, by virtue of Microsoft’s 50% stake) seem like a pretty obvious choice.
Humanities satirical traditions: I always enjoy the CS/ML/math/statistics satire in the annual SIGBOVIK and Ig Nobels; physics has Arxiv April Fools papers (like “On the Impossibility of Supersized Machines”) & journals like Special Topics; and medicine has the BMJ Christmas issue, of course.
What are the equivalents in the humanities, like sociology or literature? (I asked a month ago on Twitter and got zero suggestions...) EDIT: as of August 2024, no equivalents have been found.
Normalization-free Bayes: I was musing on Twitter about what the simplest possible still-correct computable demonstration of Bayesian inference is, that even a middle-schooler could implement & understand. My best candidate so far is ABC Bayesian inference*: simulation + rejection, along with the ‘possible worlds’ interpretation.
Someone noted that rejection sampling is simple but needs normalization steps, which adds complexity back. I recalled that somewhere on LW many years ago someone had a comment about a Bayesian interpretation where you don’t need to renormalize after every likelihood computation, and every hypothesis just decreases at different rates; as strange as it sounds, it’s apparently formally equivalent. I thought it was by Wei Dai, but I can’t seem to refind it because queries like ‘Wei Dai Bayesian decrease’ obviously pull up way too many hits, it’s probably buried in an Open Thread somewhere, my Twitter didn’t help, and Wei Dai didn’t recall it at all when I asked him. Does anyone remember this?
* I’ve made a point of using ABC in some analyses simply because it amuses me that something so simple still works, even when I’m sure I could’ve found a much faster MCMC or VI solution with some more work.
Incidentally, I’m wondering if the ABC simplification can be taken further to cover subjective Bayesian decision theory as well: if you have sets of possible worlds/hypotheses, let’s say discrete for convenience, and you do only penalty updates as rejection sampling of worlds that don’t match the current observation (like AIXI), can you then implement decision theory normally by defining a loss function and maximizing over it? In which case you can get Bayesian decision theory without probabilities, calculus, MCM, VI, conjugacy formulas falling from heaven, etc or anything more complicated than a list of numbers and a few computational primitives like
coinflip()(and then a ton of computing power to brute force the exact ABC/rejection sampling).Doing another search, it seems I made at least one comment that is somewhat relevant, although it might not be what you’re thinking of: https://www.greaterwrong.com/posts/5bd75cc58225bf06703751b2/in-memoryless-cartesian-environments-every-udt-policy-is-a-cdt-sia-policy/comment/kuY5LagQKgnuPTPYZ
Funny that you have your great LessWrong whale as I do, and that you recall that it may be from Wei Dai as well (while him not recalling)
https://www.lesswrong.com/posts/X4nYiTLGxAkR2KLAP/?commentId=nS9vvTiDLZYow2KSK
LW2 search idea: hierarchical embedding trees using some nifty “seriation” list sorting tricks I’ve developed for Gwern.net popups/tagging purposes.
Danbooru2021 is out. We’ve gone from n=3m to n=5m (w/162m tags) since Danbooru2017. Seems like all the anime you could possibly need to do cool multimodal text/image DL stuff, hint hint.
2-of-2 escrow: what is the exploding Nash equilibrium? Did it really originate with NashX? I’ve been looking for the history & real name of this concept for years now and have failed to refind it. Anyone?
Gwern, i wonder what you think about this question i asked a while ago on causality in relation to the article you posted on reddit. Do we need more general causal agents for addressing issues in RL environments?
Apologies for posting here, i didn’t know how to mention/tag someone on a post in LW.
https://www.lesswrong.com/posts/BDf7zjeqr5cjeu5qi/what-are-the-causality-effects-of-an-agents-presence-in-a?commentId=xfMj3iFHmcxjnBuqY