I know very little about artificial neural networks, like I’m not really even comfortable saying that I know what they are. (Without googling I’m basically like, “Well, probably they’re still systems of logic gates at ground level, though maybe some of them involve quantum computing circuits that are fishy-to-me in their non-binariness or something, and those systems have properties that resemble non-artificial neural networks such as nodes and weighted edges, which causes them to behave like non-artificial neural networks with stuff like association cascades and trigger-action patterns and predictive-processing-type stuff?”) [Edit: Oh snap that actually sounds a lot like the Wikipedia page!]
But I can easily imagine someone who’s very interested in artificial neural networks and has so far studied them by reading about them and talking to people about them. It’s a very different kind of thing to try to design one, at all or even from scratch, to try to use one for various purposes, to provide certain inputs and statistically analyze patterns of outputs, to reason mathematically about what seem to you to be necessary properties of artificial neural networks and then find out whether an actual neural network in front of you behaves as you’ve predicted.
So yeah if you’ve mostly been in “reading stuff” territory, that interactive demo looks to me like a great step in the right direction. But if I were in that position, I would be asking myself “what is the very simplest thing that would technically count as an artificial neural network, and what would it take for me to build that thing myself?”
If you’re not in the “mostly I’ve been reading stuff” boat and have already been doing the kinds of things I’ve described so far, then I expect that increasing the directness of your contact will look less like interacting with a different kind of thing, and more like adopting different mental patterns as you interact. How much of you is showing up to your investigations? What parts of you are asking questions, what parts of you are generating hypotheses? How many methods are you employing for turning your central puzzles around and around to see them from different angles, and what is the range of those methods? What work are you doing to let different activities of your daily life participate in your processes of observation and analysis? What are you doing to become sensitive to subtle patterns in your observations that can only become apparent over time? That kind of thing.
Remember that there are THREE entities needed for contact with the territory: The territory, the person making contact, and sensation at the point of contact. You can change your contact by changing any three of those entities.
In case you haven’t seen it, Chris Olah’s work in neural network interpretability is extremely concordant with naturalism.
It’s common for people to say that neural networks are something like “a mass of inscrutable tensors” and I feel like Olah is one of the only people whose response was something like “did you try literally scrutinizing them” and the answer is no, no one did, because they looked complicated and icky. And then Olah did, and when he looked, he saw things.
Ok I’m actually pretty curious about this myself now. The basic element of an ANN is a neuron I think, and maybe I could personally build a single neuron out of household materials? It doesn’t gotta do much, right?
It needs to be able to receive at least one input (though to be at all interesting it probably ought to receive at least 2).
It needs to sum its inputs.
Something about an activation function. Does this happen before or after the summing? My guess is after; so maybe it’s stuff like “if the sum is less than 4 then make the output be two less than the sum, but if it’s more than 4 make it be the sum times six”.
Then it’s gotta be able to output, ideally to something observable.
So I’m imagining a little circuit board of logic gates with copper wires attached to batteries and a lightbulb that can glow brighter when you give it more juice, with the activation function business happening in a series of insulators of varying strengths [uh, conductors of varying resistances?] and the variable inputs also coming from currents run through different insulators, or perhaps from different strengths of batteries.
What do you think ML people, am I on the right track? Have I sketched an artificial neuron?
I know very little about artificial neural networks, like I’m not really even comfortable saying that I know what they are. (Without googling I’m basically like, “Well, probably they’re still systems of logic gates at ground level, though maybe some of them involve quantum computing circuits that are fishy-to-me in their non-binariness or something, and those systems have properties that resemble non-artificial neural networks such as nodes and weighted edges, which causes them to behave like non-artificial neural networks with stuff like association cascades and trigger-action patterns and predictive-processing-type stuff?”) [Edit: Oh snap that actually sounds a lot like the Wikipedia page!]
But I can easily imagine someone who’s very interested in artificial neural networks and has so far studied them by reading about them and talking to people about them. It’s a very different kind of thing to try to design one, at all or even from scratch, to try to use one for various purposes, to provide certain inputs and statistically analyze patterns of outputs, to reason mathematically about what seem to you to be necessary properties of artificial neural networks and then find out whether an actual neural network in front of you behaves as you’ve predicted.
So yeah if you’ve mostly been in “reading stuff” territory, that interactive demo looks to me like a great step in the right direction. But if I were in that position, I would be asking myself “what is the very simplest thing that would technically count as an artificial neural network, and what would it take for me to build that thing myself?”
If you’re not in the “mostly I’ve been reading stuff” boat and have already been doing the kinds of things I’ve described so far, then I expect that increasing the directness of your contact will look less like interacting with a different kind of thing, and more like adopting different mental patterns as you interact. How much of you is showing up to your investigations? What parts of you are asking questions, what parts of you are generating hypotheses? How many methods are you employing for turning your central puzzles around and around to see them from different angles, and what is the range of those methods? What work are you doing to let different activities of your daily life participate in your processes of observation and analysis? What are you doing to become sensitive to subtle patterns in your observations that can only become apparent over time? That kind of thing.
Remember that there are THREE entities needed for contact with the territory: The territory, the person making contact, and sensation at the point of contact. You can change your contact by changing any three of those entities.
In case you haven’t seen it, Chris Olah’s work in neural network interpretability is extremely concordant with naturalism.
It’s common for people to say that neural networks are something like “a mass of inscrutable tensors” and I feel like Olah is one of the only people whose response was something like “did you try literally scrutinizing them” and the answer is no, no one did, because they looked complicated and icky. And then Olah did, and when he looked, he saw things.
Agreed. Also had this experience recently, although I really like the way you put it here.
Ok I’m actually pretty curious about this myself now. The basic element of an ANN is a neuron I think, and maybe I could personally build a single neuron out of household materials? It doesn’t gotta do much, right?
It needs to be able to receive at least one input (though to be at all interesting it probably ought to receive at least 2).
It needs to sum its inputs.
Something about an activation function. Does this happen before or after the summing? My guess is after; so maybe it’s stuff like “if the sum is less than 4 then make the output be two less than the sum, but if it’s more than 4 make it be the sum times six”.
Then it’s gotta be able to output, ideally to something observable.
So I’m imagining a little circuit board of logic gates with copper wires attached to batteries and a lightbulb that can glow brighter when you give it more juice, with the activation function business happening in a series of insulators of varying strengths [uh, conductors of varying resistances?] and the variable inputs also coming from currents run through different insulators, or perhaps from different strengths of batteries.
What do you think ML people, am I on the right track? Have I sketched an artificial neuron?
Actually I think I’m yearning to do this with marbles and pipes and carefully balanced buckets so the weightings can be totally literal.
Yeah, that sounds right! I think this video supports that idea as well:
That sounds like it’s on the right track to me. Is there any chance you’ve made progress on this?