Software development is a poor metaphor for AI alignment. It reminds me of Ron Jeffries’ sad attempt to write a Sudoku solver by using “extreme programming” without knowing about backtracking search. He kinda blunders around for a few weeks and then stops. Another nice mockery is this, which (unbelievably) comes from an actual AI design by someone.
A better metaphor for AI alignment is fundamental science, where we’re supposed to understand things step by step. Your post makes several such steps at the start. But then you spend many pages on sketching a software design, which even after careful reading didn’t advance my understanding of AI alignment in any way. That’s the criterion I use.
Maybe we should explain to contestants that they should try to advance the frontier of understanding by one inch, not solve all of AI alignment in one go. The latter seems to lead people in unproductive directions.
It may be worth mentioning that the “someone” who produced the “actual AI design” is a known crank. (Whose handle must not be mentioned, for ancient legend says that if you speak his name then he will appear and be a big nuisance.)
>Software development is a poor metaphor for AI alignment.
I think I disagree, but let’s ignore the metaphor aspect and focus on the model. The same causal model can also be communicated using science & engineering as a metaphor. If you want to know what scientific insights to work towards to create some breakthrough technology, it’s valuable to periodically put on your engineer hat. Without it, you’ll do basic research that could end up leading anywhere. In search terms, an engineer hat offers an improved heuristic. If your scientist hat allows you to forward chain, your engineer hat allows you to backward chain.
I’d argue the engineer hat is critical for effective differential technological development.
Software development is a poor metaphor for AI alignment. It reminds me of Ron Jeffries’ sad attempt to write a Sudoku solver by using “extreme programming” without knowing about backtracking search. He kinda blunders around for a few weeks and then stops. Another nice mockery is this, which (unbelievably) comes from an actual AI design by someone.
A better metaphor for AI alignment is fundamental science, where we’re supposed to understand things step by step. Your post makes several such steps at the start. But then you spend many pages on sketching a software design, which even after careful reading didn’t advance my understanding of AI alignment in any way. That’s the criterion I use.
Maybe we should explain to contestants that they should try to advance the frontier of understanding by one inch, not solve all of AI alignment in one go. The latter seems to lead people in unproductive directions.
It may be worth mentioning that the “someone” who produced the “actual AI design” is a known crank. (Whose handle must not be mentioned, for ancient legend says that if you speak his name then he will appear and be a big nuisance.)
>Software development is a poor metaphor for AI alignment.
I think I disagree, but let’s ignore the metaphor aspect and focus on the model. The same causal model can also be communicated using science & engineering as a metaphor. If you want to know what scientific insights to work towards to create some breakthrough technology, it’s valuable to periodically put on your engineer hat. Without it, you’ll do basic research that could end up leading anywhere. In search terms, an engineer hat offers an improved heuristic. If your scientist hat allows you to forward chain, your engineer hat allows you to backward chain.
I’d argue the engineer hat is critical for effective differential technological development.