D𝜋

Karma: 178

D𝜋 Jan 4, 2022, 11:38 AM
2 points
in reply to: A Ray’s comment on: Self-Organised Neural Networks: A simple, natural and efficient way to intelligence
Thank you for the congrats, it helps.
Note, that I only claim to reach SOTA, not to beat it.
It would be preposterous to convince anybody with this limited evidence. The goal is to raise interest so some will spend some time to look deeper into it. Most will not, of course, for many reasons, and yours is a valid one.
The advantage of this one is its simplicity. At this point any coder can take it up and build on it. This has to be turned into a new type of construction set. I would like this to provide the 15 years old of today the pleasure my first computer (machine language) gave me, and Legos before that.
You got the last bit correctly. That is what self-organisation provides: ad-hoc selection.

D𝜋 Jan 4, 2022, 9:34 AM
10 points
in reply to: jimrandomh’s comment on: Self-Organised Neural Networks: A simple, natural and efficient way to intelligence
See my answer to mlem_mlem_mlem for the second part of your comment.
You are bringing another interesting point: scaling up and tuning.
As I indicated in the roadmap, nature has chosen the way of width to that of depth.
The cortical sheet is described as a 6 layers structure, but only 3 are neurons and 2 pyramidal neurons. That is not deep. Then we see columns, functional ‘zones’, ‘regions’… There is an organisation, but it is not very deep. The number of columns in each ‘zone’ is very large. Also note that the neuron is deemed ‘stochastic’, so precision is not possible. Lastly, note (sad but true) that those who got the prize worked on technical simplification for practical use.
There is two options at this stage:
We consider, as the symbolic school has since 1969, that the underlying substrate is unimportant and, if we can find mathematical ways to describe it, we will be able to reproduce it, or...
We consider that nature has done the work (we are here to attest to that), properly, and we should look at how it did it.
1986 was an acceptable compromise, for a time. 2026, will mark one century of the 5th Solvay conference.

D𝜋 Jan 4, 2022, 8:54 AM
5 points
in reply to: mlem_mlem_mlem’s comment on: Self-Organised Neural Networks: A simple, natural and efficient way to intelligence
You are comparing step and ladder (I had to seize on it !).
If you look at Table 2 in your last reference, you will see that they, carefully, show results improving has steps are added. Ladder is just another step (an optimisation one). There is a reason why researchers use PI-MNIST: it is to reduce the size of the ladder to make comparisons clearer.
What I am trying to bring here is a new first step.
I could have tried a 784-25-10 BP/SGD network (784*25 = 19600 parameters) to compare with this system with 196 neurons and 10 connections. I have managed to get 98% with that. How much for the same with BP/SGD ?
The current paradigm has been building up since 1986, and was itself based on the perceptron from 1958.
Here, I take the simplest form of the perceptron (single layer), only adjoin a, very basic, quantiliser to drive it, and already get near SOTA. I also point out that this quantiliser is just another form of neuron.
I am trying to show it might be an interesting step to take.
What links here?
- Bucky's comment on D𝜋′s Spiking Network by lsusr (Jan 5, 2022, 2:11 PM; 3 points)

D𝜋 Jan 3, 2022, 5:57 PM
6 points
in reply to: lsusr’s comment on: Self-Organised Neural Networks: A simple, natural and efficient way to intelligence
I wrote a comment on that but this is a better to place for it.
I changed the update value from 1000 to 500 for that network size (in the code).
1000 is for the large network (98.9%). At size 792 (for 98.6%) it is too much, and the accuracy goes down after reaching the top. I did not take the time to check properly before publishing. My fault.
If you check it out now, it will get to >98.6% and stay there (tested up to 10 millions, three times, random).

D𝜋 Jan 3, 2022, 3:08 PM
9 points
on: Self-Organised Neural Networks: A simple, natural and efficient way to intelligence
Update:
I changed the adjustment values for the 98.65% version to 500/-500 (following lsusr comments).
1000/-1000 is good for the larger network (98.9%), but too much for the smaller ones. It makes the accuracy reduce after it has reached the peak value.
I was too fast publishing and did not go through all the required verifications. My fault.
I am running a series of tests to confirm. The first two are in spec and stable at 10 million steps.
Larger values speed up the convergence, and I was trying to make it as fast as possible, to not waste the time of those who would spend it verifying. Sorry about that.

D𝜋 Jan 3, 2022, 2:53 PM
5 points
in reply to: procgen’s comment on: Self-Organised Neural Networks: A simple, natural and efficient way to intelligence
They belong to the same, forgotten, family.
T.Kuhn said it all.

D𝜋 Jan 3, 2022, 2:14 PM
4 points
in reply to: sullyj3’s comment on: Self-Organised Neural Networks: A simple, natural and efficient way to intelligence
No, there isn’t, but it is interesting.
I gave it a quick look. It seems to be closer to this (this is closer to the point)
I was heavily influenced, back in the 70s, by the works of Mandelbrot and the chaos theory that developed at the time, and has gone nowhere.
The concept of self-organisation has been around for a long time but it is hard to study from the mathematical point of view, and, probably for that reason, it has never ‘picked up’.
So, of course, there are similarities, and, please, go back to all of those old papers and re-think it all.
You will benefit from an hands-on approach rather then a theoretical one. First you experiment, then you find, then you analyse and finally, you theorise. This is not quantum physics and we have the tools (computers) to easily conduct experiments.
This is just another exemple, one that could prove very useful. That’s it.

D𝜋 Jan 3, 2022, 11:30 AM
5 points
in reply to: Lech Mazur’s comment on: Self-Organised Neural Networks: A simple, natural and efficient way to intelligence
The only function that is implemented in CUDA is the test one (test_gpu).
It is also implement for CPU (test_mt_one), identically.
What matters is all clearly (I hope) explained in the text. It is simple enough that its reach is not limited to ML researchers and clearly within that of a lot of coders. The IT revolution started when amateurs got PCs.
In this version of the code, I had to make a tradeoff between completeness, usability and practicality. Write your own code, it does not matter. It is the concept that does.
The (upcoming) website will give separate, readable, versions. I am waiting to get a proper idea of what is demanded before I do that, so thank you for that input.

D𝜋 Jan 3, 2022, 10:34 AM
12 points
in reply to: Lech Mazur’s comment on: Self-Organised Neural Networks: A simple, natural and efficient way to intelligence
I have, deliberately, taken away everything relating to geometry from this presentation.
It took 12 years (1986-1998) and (how much research effort ?) ,to go from BP/SGD to convolutions.
This is a one man effort, on my own personal time (20,000 hours over the past 6 years), that I am giving away for the community to freely take over. I am really sorry if it is not enough. Their choice.
It is not an add-on to something that exist but a complete restart. One thing at a time.
As for CUDA, if you have a lot of threads, it is bearable, and you can use old, cheap, GPUs with very little loss (they have been optimised recently for FP multiply/add, at the expense of ADD of INT).
FYI, I got >99.3% with only adding a fix layer of simple preset filters (no augmentation) and the idea behind can be readily extended. And you can also train, unsupervised, convolutions.

D𝜋 Jan 3, 2022, 9:59 AM
4 points
in reply to: Daniel Kokotajlo’s comment on: Self-Organised Neural Networks: A simple, natural and efficient way to intelligence
That is a question of low philosophical value, but of the highest practical importance.
At line 3,000,000 with the 98.9% setup, in the full log, there is these two informations:
‘sp: 1640 734 548’, and ‘nbupd: 7940.54 6894.20’ (and the test accuracy is exactly 98.9%)
It means that the average spike count for the IS-group is 1640 per sample, 734 for the highest ISNOT-group and 548 for the other ones. The number of weight updates per IS-learn is 7940.54 and 6894.20 per ISNOT-learn. With the coding scheme used, the average number of inputs per sample over the four cycles is 2162 (total number of activated pixels in the input matrix) for 784 pixels. There is 7920 neurons per group with 10 connections each (so ¹⁰⁄₇₈₄ th of the pixel matrix), for a total of 79200 neurons.
From those numbers:
The average number of integer additions done for all neurons in a group when a sample is presented is: 79200 * 2162 * ¹⁰⁄₇₈₄ = 2,184,061 integer additions in total.
And for the spike counts: 1640 + 734 + 8*548 = 6758 INCrements (counting the spikes).
When learning, for each IS-learn, there is 7940, and for each ISNOT-learn, 6894, weight updates . Those are additions of integers. So an average of (7940 + 9*6894) / 10 = 7000 additions of integer.
That is to be compared with a 3 fully connected, say 800 units (to make up for the difference between 98.90% and 98.94%) layers.
That would be at least 800*800*2 + 800*784 = 1,907,200 floating point multiplications, plus what would be used for Max-norm, ReLu,… that I am not qualified to evaluate, but might roughly double that ?
And the same for each update (low estimate).
Even with recent works on sparse updates that do reduce that by 93%, it is still more than 133,000 floating-point multiplications (against 7000 additions of integers).
I have managed to get over 98.5% with 2000 neurons (20,000 connections). I would like to know if BP/SGD can perform to that level with such a small number of parameters (that would be one fully connected layer of 25 units) ? And, as I said in the roadmap, that is what will matter for full real systems.
That is the basic building bloc. the 1x1x1 Lego brick. 1.5/1.1 = 36% improvement with 40 times the ressources is useless in practice.
And that is missing the real point laid out in the Roadmap: this system CAN and MUST be implemented in analog (hybrid until we get practical memristors), whereas BP/SGD CAN NOT.
There is, at least, another order of magnitude in efficiency to be gained there.
There is a lot of effort invested, at this time, in the industry to implement AI at IC-level. Now is the time.

D𝜋 Jan 2, 2022, 10:29 PM
14 points
in reply to: Randomized, Controlled’s comment on: Self-Organised Neural Networks: A simple, natural and efficient way to intelligence
I think I have expressed my views on the matter of responsibility quiet clearly in the conclusion.
I just checked Yudkowsky on Google. He founded this website, so good.
Here is not the place to argue my views on super-intelligence, but I clearly side with Russell and Norvig. Life is just too complex; luckily.
As for safety, the title of Jessica Taylor’s article is:
“Quantilizers: A Safer Alternative to Maximizers for Limited Optimization”.
I will just be glad to have proved that alternative to be effective.

D𝜋 Jan 2, 2022, 8:12 PM
4 points
in reply to: Randomized, Controlled’s comment on: Self-Organised Neural Networks: A simple, natural and efficient way to intelligence
I am not sure I understand your question (sorry, I do not know what is Yudkowsky’DMs)
I basically disclosed, to all, that the way we all think we think, does work.
What kind of responsibility could that bear ?

D𝜋 Jan 2, 2022, 4:46 PM
8 points
in reply to: Richard_Kennaway’s comment on: Self-Organised Neural Networks: A simple, natural and efficient way to intelligence
We never stop learning.
To kill the program, shoot ‘Ctrl+C’.
Seriously, this system works ‘online’. I gave the exemple of the kids and the Dutch to illustrate that in nature, things change around us and we adjust what we know to the new conditions. A learning process should not have a stopping criteria.
The system converges, on PI-MNIST, at 3-5 million steps. To compare, recent research papers stop at 1 million, but keep in mind that we only update about 2 out of 10 groups each time, so it is equivalent.
So you can use “for( ; b<5000000 ; b++ )” instead of “while( 1 == 1 )” in the batch() function.
After convergence, it stays within a 0.1% margin forever after. You can design a stop test around that if you want, or around the fact that weights stabilise, or anything of that kind.
If you were to use a specific selection of the dataset, wait until it stabilises and, then, use the whole set, the system would ‘start learning again’ and adjust to that change. Forever.
It is a feature, not a bug.

D𝜋 Jan 2, 2022, 1:15 PM
9 points
in reply to: tailcalled’s comment on: Self-Organised Neural Networks: A simple, natural and efficient way to intelligence
It is PI-MNIST.
Permutation Invariant. To keep it simple, you cannot use convolutions. It is all explained in the text.
Real SOTA on that version is 99.04% (Maxout), but that is with 65 Millions+ parameters. I do not have the hardware (or time).
I stopped at 98.9% with 750,000 connections (integers and additions) and this is close to what BP/SGD (table 2) gets with 3 hidden layer of 1024 units each, for a total of >3,000,000 parameters (floating-points and multiplication) with max-norm and Relu.
For a similar accuracy, the number of ‘parameters’ is almost an order of magnitude lower with this system and efficiency even more.
Remember, it is not supposed to work at all, and it is not optimised.

D𝜋 2 Jan 2022 10:05 UTC
1 point
on: Another view of quantilizers: avoiding Goodhart’s Law
Happy new year.
I have just posted on LessWrong the result of my work on AI.
Your work on quantilisers is the core mathematical evidence of what I propose (the code is another).
I would really appreciate your opinion on it.
Kind regards

D𝜋 2 Jan 2022 8:07 UTC
6 points
in reply to: lsusr’s comment on: Self-Organised Neural Networks: A simple, natural and efficient way to intelligence
The link I typed in and appears when hovering over is, indeed, ‘http://yann.lecun.com/exdb/mnist/’, and it works on my machine… Thanks for the additional link.

Self-Organised Neural Networks: A simple, natural and efficient way to intelligence

D𝜋1 Jan 2022 23:24 UTC

40 points

51 comments44 min readLW link

D𝜋

Self-Or­ganised Neu­ral Net­works: A sim­ple, nat­u­ral and effi­cient way to intelligence

Self-Organised Neural Networks: A simple, natural and efficient way to intelligence