Well, good question. Frankly I don’t think it matters. I don’t believe that my claims are sensitive to the distributions (aside from some convoluted ones), or that giving you a specific distribution would help you to defend either position (feel free to prove me wrong). But when I want to feel rigorous, I assume that I’m starting off with a natural length-based distribution over all Turing machines (or maybe all neural networks), then discard all machines that fail to pass some relatively simple criteria about the output they generate (e.g. does it classify a given set of cat pictures correctly), keep the ones that passed, normalize and draw from that.
But really, by “random” I mean nearly anything that’s not entirely intentional. To use a metaphor for machine learning, if you pick a random point in the world map, then find the nearest point that’s 2km above sea level, you’ll find a “random” point that’s 2km above sea level. The algorithm has a non-random step, but the outcome is clearly random in a significant way. The distribution you get is different from the one I described in my previous paragraph (where you just filtered the initial point distribution to get the points at 2km), but they’ll most likely be close.
Well, good question. Frankly I don’t think it matters. I don’t believe that my claims are sensitive to the distributions (aside from some convoluted ones), or that giving you a specific distribution would help you to defend either position (feel free to prove me wrong). But when I want to feel rigorous, I assume that I’m starting off with a natural length-based distribution over all Turing machines (or maybe all neural networks), then discard all machines that fail to pass some relatively simple criteria about the output they generate (e.g. does it classify a given set of cat pictures correctly), keep the ones that passed, normalize and draw from that.
But really, by “random” I mean nearly anything that’s not entirely intentional. To use a metaphor for machine learning, if you pick a random point in the world map, then find the nearest point that’s 2km above sea level, you’ll find a “random” point that’s 2km above sea level. The algorithm has a non-random step, but the outcome is clearly random in a significant way. The distribution you get is different from the one I described in my previous paragraph (where you just filtered the initial point distribution to get the points at 2km), but they’ll most likely be close.
Maybe that answers your other comment too?