What’s your definition of ‘explicitly stochastic mechanisms’? Every physical-world sensor is stochastic to some degree.
“All” you really need to be stochastic is to ensure that you have some level of stochastic input, plus a chaotic feedback loop of some sort.
Some initial work/thoughts on solutions: Training an RNN on any set of random outputs will not work, it will just memorize the random strings. Can I train directly on the entropy of the output?
Train directly on maximizing the Lyapunov exponent, perhaps? That is, repeatedly:
Pick a random input x
Calculate the output F(x)
Make a random perturbation to said input: x′
Calculate the output F(x′)
Do a training update towards having said outputs diverge. That is, maximize |F(x′)−F(x)|.[1]
(...this seems suspiciously like ‘calculate and maximize the gradient’...)
How about outputing a string that has different amounts of structure and entropy?
Instead of maximizing the Lyapunov exponent, try optimizing towards a specific value of Lyapunov exponent instead?
Great idea! My intuition says this won’t work, as you’ll just capture half of the mechanism of the type of chaotic attractor we want. This will give you the “stretching” of points close in phase space to some elongated section, but not by itself the folding over of that stretched section, which at least in my current thinking is necessary. But it’s definitely worth trying, I could very well be wrong! Thanks for the idea :)
Similarly it’s not obvious to me that constraining the lyapanov exponent to a certain value gives you the correct “structure”. For instance, if instead of ..01R… I wanted to train on …10R… Or …11R… Etc. But maybe the training of the lyapanov would just be one part of the optimization, and then other factors could play into it.
This will give you the “stretching” of points close in phase space to some elongated section, but not by itself the folding over of that stretched section, which at least in my current thinking is necessary.
I mean, the only way to have “stretching” of ‘most’ points in phase space is to have some sort of ‘folding over’.
Of course, it’s an entirely different matter as to if a standard optimizer can actually make any headway in figuring that out.
What’s your definition of ‘explicitly stochastic mechanisms’? Every physical-world sensor is stochastic to some degree.
“All” you really need to be stochastic is to ensure that you have some level of stochastic input, plus a chaotic feedback loop of some sort.
Train directly on maximizing the Lyapunov exponent, perhaps? That is, repeatedly:
Pick a random input x
Calculate the output F(x)
Make a random perturbation to said input: x′
Calculate the output F(x′)
Do a training update towards having said outputs diverge. That is, maximize |F(x′)−F(x)|.[1]
(...this seems suspiciously like ‘calculate and maximize the gradient’...)
Instead of maximizing the Lyapunov exponent, try optimizing towards a specific value of Lyapunov exponent instead?
This isn’t quite right, I don’t think. Close though.
Great idea! My intuition says this won’t work, as you’ll just capture half of the mechanism of the type of chaotic attractor we want. This will give you the “stretching” of points close in phase space to some elongated section, but not by itself the folding over of that stretched section, which at least in my current thinking is necessary. But it’s definitely worth trying, I could very well be wrong! Thanks for the idea :)
Similarly it’s not obvious to me that constraining the lyapanov exponent to a certain value gives you the correct “structure”. For instance, if instead of ..01R… I wanted to train on …10R… Or …11R… Etc. But maybe the training of the lyapanov would just be one part of the optimization, and then other factors could play into it.
I mean, the only way to have “stretching” of ‘most’ points in phase space is to have some sort of ‘folding over’.
Of course, it’s an entirely different matter as to if a standard optimizer can actually make any headway in figuring that out.