One point which is never mentioned (as far as I can see) either in D𝜋′s article or in this one, but which is present in the code, is the minimum threshold for learning. D𝜋′s code only learns if the neuron’s potential is already above a certain minimal value (300,000 in the code as posted). D𝜋 says the maximum threshold is optional, but in my experiments, I found the minimum threshold absolutely crucial for the network to ever get to high levels of accuracy. Seems like otherwise a neuron gets distracted by a bunch of junk updates that aren’t really related to its particular purpose.
Welcome onboard this IT ship to baldly go where no one as gone before !
Indeed, I just wrote ‘when it spikes’ and further as the ‘low threshold’ and no more. I work in complete isolation and some things are so obvious inside my brain that I do not consider them as non obvious to others.
It is part of the ‘when’ aspect of learning, but uses an internal state of the neuron instead of an external information from the quantilisers.
If there is little reaction to a sample in a neuron (spiking does happen slowly, or not), it is meaningless and you should ignore it. If it comes too fast, it is already ‘in’ the system and there is no point in adding to it. You are right to say the first rule is more important than the second.
Originally, there was only one threshold instead of 3. When learning, the update would only take place if the threshold was reached after a minimum of two cycles (or 3, but then it converges unbearably slowly), and only for the connections that had been active at least twice. I ‘compacted’ it for use within one cycle (to make it look simpler), so it was 50% of the threshold minimum, and then adjusted (might as well) that value by scanning around and, then, added the upper threshold, but more to limit the number of updates than to improve the accuracy (although it contributes a small bit). The best result is with 30% and 120%, whatever the size or the other parameters.
Before I write this, I quickly checked on PI-F-MNIST. It is still ongoing, but it seems to hold true even on that dataset (BTW: use quantUpP = 3.4 and quantUpN = 40.8 to get to 90.2% with 792 neurons and 90.5% with 7920).
As it seems you are interested, feel free to contact me through private message. There is plenty more in my bag than can fit in a post or comment. I can provide you some more complete code (this one is triple distilled).
One point which is never mentioned (as far as I can see) either in D𝜋′s article or in this one, but which is present in the code, is the minimum threshold for learning. D𝜋′s code only learns if the neuron’s potential is already above a certain minimal value (300,000 in the code as posted). D𝜋 says the maximum threshold is optional, but in my experiments, I found the minimum threshold absolutely crucial for the network to ever get to high levels of accuracy. Seems like otherwise a neuron gets distracted by a bunch of junk updates that aren’t really related to its particular purpose.
Welcome onboard this IT ship to baldly go where no one as gone before !
Indeed, I just wrote ‘when it spikes’ and further as the ‘low threshold’ and no more. I work in complete isolation and some things are so obvious inside my brain that I do not consider them as non obvious to others.
It is part of the ‘when’ aspect of learning, but uses an internal state of the neuron instead of an external information from the quantilisers.
If there is little reaction to a sample in a neuron (spiking does happen slowly, or not), it is meaningless and you should ignore it. If it comes too fast, it is already ‘in’ the system and there is no point in adding to it. You are right to say the first rule is more important than the second.
Originally, there was only one threshold instead of 3. When learning, the update would only take place if the threshold was reached after a minimum of two cycles (or 3, but then it converges unbearably slowly), and only for the connections that had been active at least twice. I ‘compacted’ it for use within one cycle (to make it look simpler), so it was 50% of the threshold minimum, and then adjusted (might as well) that value by scanning around and, then, added the upper threshold, but more to limit the number of updates than to improve the accuracy (although it contributes a small bit). The best result is with 30% and 120%, whatever the size or the other parameters.
Before I write this, I quickly checked on PI-F-MNIST. It is still ongoing, but it seems to hold true even on that dataset (BTW: use quantUpP = 3.4 and quantUpN = 40.8 to get to 90.2% with 792 neurons and 90.5% with 7920).
As it seems you are interested, feel free to contact me through private message. There is plenty more in my bag than can fit in a post or comment. I can provide you some more complete code (this one is triple distilled).
Thank you very much for your interest.