This reminds me a lot of a toy project I have in the back of my mind but will probably never get around to:
Which is to train a transformer on the sequences generated by the logic models from the apperception engine paper (which in the paper are inferred by the apperception engine from the sequences) with the aim of predicting the logic model.
The wikipedia page on the technological singularity has some background on that. https://en.wikipedia.org/wiki/Technological_singularity
About 1.) That GPT4 performance jumps most in non-char tests seems to point towards two sources of difficulty in H-tests, with one being tokenization hiding char-level information.
About 2.) To me your results look completely consistent with scale solving H-tests. There are many benchmarks where a certain level has to be reached to leave random performance behind. For your benchmark that level is pretty high, but Claude and GPT4 seem to be above it.
If it’s not scale, what makes Claude and GPT4 capable of making a dent in your benchmark?
About 3.) Finetuning doesn’t convey enough information to completely revamp the representation of the spelling of different tokens. Finetuning mostly doesn’t teach models skills they don’t have. It instead points them squarely at the task they should be doing.
To me the most interesting question is to what extend your network learns to do reasoning/search vs pure pattern recognition.
I trained a transformer to predict tournament chess moves a few years back and my impression was that it played strongly in the opening and always made sensible looking moves but had absolutely no ability to look ahead.
I am currently working on a benchmark of positions that require reasoning and can’t be solved by highly trained intuition alone. Would you be interested in running such a benchmark?
99.2% square accuracy is consistent with 50% position accuracy. Did you check position accuracy?
There was another Chess-GPT investigation into that question recently by Adam Karvonen: https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world-models.html
The linear probe accuracy for the board state actually peaks in the sixth layer (out of eight). To predict the next move it already discards some of the state information. Well, maybe that is unsurprising.
It also doesn’t reach terribly impressive accuracy. 99.2% sounds a lot, but it is per square, which means it might get a square wrong in every second position.
I think more important than how easy it is to extract the information, is how necessary it is to extract the information. You can probably be somewhat fuzzy about board state details and still get great accuracy.
There is a step in move prediction where you go beyond the intuitive move selection and have to calculate to find deeper reasons for and against moves. This feels similar to me to attending to your uncertainty about the placement of particular pieces beyond the immediate necessity. And all these models have not taken this step yet.
I’m actually doing an analysis right now to nail it down that GPTs don’t calculate ahead when trained on move prediction and stay completely in the intuitive move selection regime, but it’s not easy to separate intuitive move selection and calculation in a bulletproof way.
System 2 thinking that takes a detour over tokens is fundamentally limited compared to something that continuously modifies a complex and highly dimensional vector representation.
Integration of senses will happen, but is the information density of non-text modalities high enough to contribute to the intelligence of future models?
What I call “learning during problem solving” relies on the ability to extract a lot of information from a single problem. To investigate and understand the structure of this one problem. To in the process of doing that building a representation of this problem that can be leveraged in the future to solve problems that have similar aspects.
I think you have defined me as not really talking as I am on the autism spectrum and have trouble telling emotions from tone.
No, he didn’t. Talking is not listening and there’s a big difference between being bad at understanding emotional nuance because of cognitive limitations and the information that would be necessary for understanding emotional nuance never even reaching you brain.
Was Stephen Hawking able to talk (late in life)? No, he wasn’t. He was able to write and his writing was read by a machine. Just like GPT4.
If I read a book to my daughter, does the author talk to her? No. He might be mute or dead. Writing and then having your text read by a different person or system is not talking.
But in the end, these are just words. It’s a fact that GPT4 has no control over how what it writes is read, nor can it hear how what it has written is being read.
If the entire representation of a complex task or problem is collapsed into a text, reading that text and trying to push further is not really “reasoning across calls”. I expect that you can go further with that, but not much further. At least that’s what it looks like currently.
I don’t think you can learn to solve very specific complex problems with the kind of continuous learning that would be possibly to implement with current models. Some of the theorem-prover papers have continuous learning loops that basically try to do this but those still seem very inefficient and are applied to only highly formalised problems whose solutions can be automatically verified.
Yes, multi-modality is not a hard limitation.
I know these approaches and they don’t work. Maybe they will start working at some point, but to me very unclear when and why that should happen. All approaches that use recurrence based on token-output are fundamentally impoverished compared to real recurrence.
About multi-modality:
I expect these limitations to largely vanish as models are scaled up and trained end-to-end on a large variety of modalities.
Yes, maybe Gemini will be able to really hear and see.
Your views were called “morally atrocious” because you stated that human extinction would not necessarily be bad. Seems very clear from the context in the comment frankly.
I agree that massive population growth would also be dangerous. We have that in Africa, so I worry about it for Afrika. We don’t have it anywhere else, so I don’t worry about it for any other place.
Empirically, resource wars are much less likely than internecine ethnic strife.
After we have automated much of the economy, there won’t be side effects on the economy. The trick is actually getting there.
I don’t know what Zvi and Robin Hanson would celebrate, but I personally worry about fast population decline in those “geographical/cultural pockets” that are responsible for scientific and technological progress.
And I worry because I see the possibility that the decline of innovation and tech will not be as gradual as even fast population decline generally is, but that this decline will be exacerbated by the political instability and/or political sclerosis that comes from two many old people / too much immigration + a shrinking pie.
It is the change that is bad, not necessarily the future total size of the population.
Edit: Maybe I should unpack that a bit. I also think more people is better, because life is good and innovation is proportional to the number of innovators, but apart from that:
A decreasing population leads to economic stagnation and innovation slowdown. Both can be observed in Japan. South Korea, China, Taiwan are on track to tank their population much faster than Japan ever did. Hows that going to work out for them?
In a permanent recession will investment dry up killing whatever dynamism there might still be?
If the age pyramid is inverted old people have too much political power for the country to ever reverse course and support the young towards family formation.
If you allow massive immigration to fix the labor shortage you also invite ethnic strife down the line. Almost all violent conflicts are based on having two or more ethnic groups within one country.
Will young people emigrate if they are burdened with caring for too many old people in a shrinking economy?
My view is that the progress we observe in the last centuries is more fragile than it seems and it is certainly possible that we will kill it almost completely if we continue to remove or weaken many of the preconditions for it.
Does the post ever mention the target of growing the population? I only recall mentions of replacement fertility.
The next question would be the success-rate. Even successful somatic gene editing I’ve read about so far have modified only a small fraction of cells. Is it realistic to modify a double digit percentage of neurons in the brain?
One thought: One could probably do mice studies where instead of maximizing a polygenic score, non-consensus variants are edited to reduce mutational load. If that had positive effects it would be a huge result.
Somatic gene editing was in cards for a while now, but I assumed that so far off-target effects would make that pretty risky, especially for a large number of variants.
What is the current situation regarding off-target effects for large numbers of edits?
https://www.sciencedirect.com/science/article/pii/S0004370220301855#se0050
https://www.sciencedirect.com/science/article/pii/S0004370221000722