I definitely agree with the claim that evolutionary strategies being effective would weaken my entire case. I do think that evolutionary methods like GAs are too hobbled by their inability to exploit white-box optimization, unlike SGD, but we shall see.
I genuinely don’t know if heuristic AIXI is a real thing or not, but if it is it combines the ability to search the whole space of possible algorithms (which evolution has but SGD doesn’t) with the ability to take advantage of higher order statistics (like SGD does but evolution doesn’t).
My best guess is that just as there was a “Deep learning” regime that only got unlocked once we had tons of compute from GPUs, there’s also a heuristic AIXI regime that unlocks at some level of compute.
I definitely agree with the claim that evolutionary strategies being effective would weaken my entire case. I do think that evolutionary methods like GAs are too hobbled by their inability to exploit white-box optimization, unlike SGD, but we shall see.
I genuinely don’t know if heuristic AIXI is a real thing or not, but if it is it combines the ability to search the whole space of possible algorithms (which evolution has but SGD doesn’t) with the ability to take advantage of higher order statistics (like SGD does but evolution doesn’t).
My best guess is that just as there was a “Deep learning” regime that only got unlocked once we had tons of compute from GPUs, there’s also a heuristic AIXI regime that unlocks at some level of compute.