I haven’t experimented very much, but here’s one example prompt.
Please describe what you mentally picture when reading the following block of text:
“ A Shutdown Problem Proposal
First things first: this is not (yet) aimed at solving the whole corrigibility problem, or even the whole shutdown problem.
The main thing this proposal is intended to do is to get past the barriers MIRI found in their old work on the shutdown problem. In particular, in a toy problem basically-identical to the one MIRI used, we want an agent which:
Does not want to manipulate the shutdown button Does respond to the shutdown button Does want to make any child-agents it creates responsive-but-not-manipulative to the shutdown button, recursively (i.e. including children-of-children etc) If I understand correctly, this is roughly the combination of features which MIRI had the most trouble achieving simultaneously. ”
This one produced basically-decent results from GPT-4.
Although I don’t have the exact prompt on hand at the moment, I’ve also asked GPT-4 to annotate a piece of code line-by-line with a Fermi estimate of its runtime, which worked pretty well.
Don’t really need comments which are non-obvious to an expert. Part of what makes LLMs well-suited to building external cognitive tools is that external cognitive tools can create value by just tracking “obvious” things, thereby freeing up the user’s attention/working memory for other things.
So kinda like spellcheckers (most typos you could figure out, but why spend time and attention on proofreading if the program can do that for you), but… thought-checkers.
Like, if a part of your article contradicts another part, it would be underlined.
I’ve long wanted this, but it’s not clear how to do it. Long-context LLMs are still expensive and for authors who need it most, context windows are still too small: me or Yudkowsky, for example, would still exceed the context window of almost all LLMs except possibly the newest Gemini. And then you have their weak reasoning. You could try to RAG it, but embeddings are not necessarily tuned to encode logically contradictory or inconsistent claims: probably if I wrote “the sky is blue” in one place and “the sky is red” in another, a retrieval would be able to retrieve both paragraphs and a LLM point out that they are contradictory, but such blatant contradictions are probably too rare to be useful to check for. You want something more subtle, like where you say “the sky is blue” and elsewhere “I looked up from the ground and saw the color of apples”. You could try to brute force it and consider every pairwise comparison of 2 reasonable sized chunks of text and ask for contradictions, but this is quadratic and will get slow and expensive and probably turn up too many false positives. (And how do you screen off false positives and mark them ‘valid’?)
My general thinking these days is that these truly useful ‘tools for thought’ LLMs are going to require either much better & cheaper LLMs, so smart that they can provide useful assistance despite being used in a grossly unnatural way input-wise or safety-tuned to hell, or biting the bullet of finetuning/dynamic-evaluation (see my Nenex proposal).
A LLM finetuned on my corpus can hope to quickly find, with good accuracy, contradictions because it was trained to know ‘the sky was blue’ when I wrote that at the beginning of the corpus, and it gets confused when it hits ‘the color of ____’ and it gets the prediction totally wrong. And RAG on an embedding tailored to the corpus can hope to surface the contradictions because it sees the two uses are the same in the essays’ context, etc. (And if you run them locally, and they don’t need a large context window because of the finetuning, they will be fast and cheap, so you can more meaningfully apply the brute force approach; or you could just run multiple epoches on your data, with an auxiliary prompt asking for a general critique, which would cover contradictions. ‘You say here X, but don’t I recall you saying ~X back at the beginning? What gives?’)
Feed it a shorter text (that fits in the window) and ask it to provide a short summary focusing on factual statements. Then hopefully all short versions could fit in the window. Find the contradiction—report the two contradicting factual statements and which section they appeared in. Locate the statement in the original text.
I may have. Just gwern.net is, I think, somewhere around 2m, and it’s not comprehensive. Also, for contradictions, I would want to detect contradictions against citations/references as well (detecting miscitations would be more important than self-consistency IMO), and as a rough ballpark, the current Gwern.net annotation* corpus is approaching 4.3m words, looks like, and is also not comprehensive. So, closer than one might think! (Anyway, doesn’t deal with the cost or latency: as you can see in the demos, we are talking minutes, not seconds, for these million-token calls and the price is probably going to be in the dollar+ regime per call.)
* which are not fulltext. It would be nice to throw in all of the hosted paper & book & webpage fulltexts, but then that’s probably more like 200m+ words.
There may not be any ‘clear’ technical obstruction, but it has failed badly in the past. ‘Add more parallelism’ (particularly hierarchically) is one of the most obvious ways to improve attention, and people have spent the past 5 years failing to come up with efficient attentions that do anything but move along a Pareto frontier from ‘fast but doesn’t work’ to ‘slow and works only as well as the original dense attention’. It’s just inherently difficult to know what tokens you will need across millions of tokens without input from all the other tokens (unless you are psychic), implying extensive computation of some sort, which makes things inherently serial and costs you latency, even if you are rich enough to spend compute like water. You’ll note that when Claude-2 was demoing the ultra-long attention windows, it too spent a minute or two churning. While the most effective improvements in long-range attention like Flash Attention or Ring Attention are just hyperoptimizing dense attention, which is inherently limited.
I haven’t experimented very much, but here’s one example prompt.
This one produced basically-decent results from GPT-4.
Although I don’t have the exact prompt on hand at the moment, I’ve also asked GPT-4 to annotate a piece of code line-by-line with a Fermi estimate of its runtime, which worked pretty well.
Yeah i was thinking your specs were, well
Wrap gpt-4 and Gemini, columned output over a set of text, applying prompts to each section? Prototype in a weekend.
Make the AI able to meaningfully contribute non obvious comments to help someone who already is an expert?
https://xkcd.com/1425/
Don’t really need comments which are non-obvious to an expert. Part of what makes LLMs well-suited to building external cognitive tools is that external cognitive tools can create value by just tracking “obvious” things, thereby freeing up the user’s attention/working memory for other things.
So kinda like spellcheckers (most typos you could figure out, but why spend time and attention on proofreading if the program can do that for you), but… thought-checkers.
Like, if a part of your article contradicts another part, it would be underlined.
I’ve long wanted this, but it’s not clear how to do it. Long-context LLMs are still expensive and for authors who need it most, context windows are still too small: me or Yudkowsky, for example, would still exceed the context window of almost all LLMs except possibly the newest Gemini. And then you have their weak reasoning. You could try to RAG it, but embeddings are not necessarily tuned to encode logically contradictory or inconsistent claims: probably if I wrote “the sky is blue” in one place and “the sky is red” in another, a retrieval would be able to retrieve both paragraphs and a LLM point out that they are contradictory, but such blatant contradictions are probably too rare to be useful to check for. You want something more subtle, like where you say “the sky is blue” and elsewhere “I looked up from the ground and saw the color of apples”. You could try to brute force it and consider every pairwise comparison of 2 reasonable sized chunks of text and ask for contradictions, but this is quadratic and will get slow and expensive and probably turn up too many false positives. (And how do you screen off false positives and mark them ‘valid’?)
My general thinking these days is that these truly useful ‘tools for thought’ LLMs are going to require either much better & cheaper LLMs, so smart that they can provide useful assistance despite being used in a grossly unnatural way input-wise or safety-tuned to hell, or biting the bullet of finetuning/dynamic-evaluation (see my Nenex proposal).
A LLM finetuned on my corpus can hope to quickly find, with good accuracy, contradictions because it was trained to know ‘the sky was blue’ when I wrote that at the beginning of the corpus, and it gets confused when it hits ‘the color of ____’ and it gets the prediction totally wrong. And RAG on an embedding tailored to the corpus can hope to surface the contradictions because it sees the two uses are the same in the essays’ context, etc. (And if you run them locally, and they don’t need a large context window because of the finetuning, they will be fast and cheap, so you can more meaningfully apply the brute force approach; or you could just run multiple epoches on your data, with an auxiliary prompt asking for a general critique, which would cover contradictions. ‘You say here X, but don’t I recall you saying ~X back at the beginning? What gives?’)
Perhaps you could do it in multiple steps.
Feed it a shorter text (that fits in the window) and ask it to provide a short summary focusing on factual statements. Then hopefully all short versions could fit in the window. Find the contradiction—report the two contradicting factual statements and which section they appeared in. Locate the statement in the original text.
Did you write more than 7 million words yet @gwern? https://www.google.com/amp/s/blog.google/technology/ai/google-gemini-next-generation-model-february-2024/amp/
Basically it’s the “lazy wait” calculation. Get something to work now or wait until the 700k or 7m word context window ships.
I may have. Just gwern.net is, I think, somewhere around 2m, and it’s not comprehensive. Also, for contradictions, I would want to detect contradictions against citations/references as well (detecting miscitations would be more important than self-consistency IMO), and as a rough ballpark, the current Gwern.net annotation* corpus is approaching 4.3m words, looks like, and is also not comprehensive. So, closer than one might think! (Anyway, doesn’t deal with the cost or latency: as you can see in the demos, we are talking minutes, not seconds, for these million-token calls and the price is probably going to be in the dollar+ regime per call.)
* which are not fulltext. It would be nice to throw in all of the hosted paper & book & webpage fulltexts, but then that’s probably more like 200m+ words.
There isn’t any clear technical obstruction to getting this time down pretty small with more parallelism.
There may not be any ‘clear’ technical obstruction, but it has failed badly in the past. ‘Add more parallelism’ (particularly hierarchically) is one of the most obvious ways to improve attention, and people have spent the past 5 years failing to come up with efficient attentions that do anything but move along a Pareto frontier from ‘fast but doesn’t work’ to ‘slow and works only as well as the original dense attention’. It’s just inherently difficult to know what tokens you will need across millions of tokens without input from all the other tokens (unless you are psychic), implying extensive computation of some sort, which makes things inherently serial and costs you latency, even if you are rich enough to spend compute like water. You’ll note that when Claude-2 was demoing the ultra-long attention windows, it too spent a minute or two churning. While the most effective improvements in long-range attention like Flash Attention or Ring Attention are just hyperoptimizing dense attention, which is inherently limited.