I realize that my position might seem increasingly flippant, but I really think it is necessary to acknowledge that you’ve stated a core assumption as a fact.
Alignment doesn’t run on some nega-math that can’t be cast as an optimization problem.
I am not saying that the concept of “alignment” is some bizarre meta-physical idea that cannot be approximated by a computer because something something human souls etc, or some other nonsense.
However the assumption that “alignment is representable in math” directly implies “alignment is representable as an optimization problem” seems potentially false to me, and I’m not sure why you’re certain it is true.
There exist systems that can be 1.) represented mathematically, 2.) perform computations, and 3.) do not correspond to some type of min/max optimization, e.g. various analog computers or cellular automaton.
I don’t think it is ridiculous to suggest that what the human brain does is 1.) representable in math, 2.) in some type of way that we could actually understand and re-implement it on hardware / software systems, and 3.) but not as an optimization problem where there exists some reward function to maximize or some loss function to minimize.
There exist systems that can be 1.) represented mathematically, 2.) perform computations, and 3.) do not correspond to some type of min/max optimization, e.g. various analog computers or cellular automaton.
You don’t even have to go that far. What about, just, regular non-iterative programs? Are type(obj) or json.dump(dict) or resnet50(image) usefully/nontrivially recast as optimization programs? AFAICT there are a ton of things that are made up of normal math/computation and where trying to recast them as optimization problems isn’t helpful.
I realize that my position might seem increasingly flippant, but I really think it is necessary to acknowledge that you’ve stated a core assumption as a fact.
I am not saying that the concept of “alignment” is some bizarre meta-physical idea that cannot be approximated by a computer because something something human souls etc, or some other nonsense.
However the assumption that “alignment is representable in math” directly implies “alignment is representable as an optimization problem” seems potentially false to me, and I’m not sure why you’re certain it is true.
There exist systems that can be 1.) represented mathematically, 2.) perform computations, and 3.) do not correspond to some type of min/max optimization, e.g. various analog computers or cellular automaton.
I don’t think it is ridiculous to suggest that what the human brain does is 1.) representable in math, 2.) in some type of way that we could actually understand and re-implement it on hardware / software systems, and 3.) but not as an optimization problem where there exists some reward function to maximize or some loss function to minimize.
You don’t even have to go that far. What about, just, regular non-iterative programs? Are
type(obj)orjson.dump(dict)orresnet50(image)usefully/nontrivially recast as optimization programs? AFAICT there are a ton of things that are made up of normal math/computation and where trying to recast them as optimization problems isn’t helpful.