Imagine an astronomer in the year 1600, who frequently refers to the “giant inscrutable movements” of the stars. [...]
I think the analogy to {building intelligent systems} is unclear/weak. There seem to be many disanalogies:
In the astronomy case, we have
A phenomenon (the stars and their motions) that we cannot affect
That phenomenon is describable with simple laws.
The “piles of numbers” are detailed measurements of that phenomenon.
It is useful to take more measurements, doing so is helpful for finding the simple laws.
In the {building AIs} case, we have
A phenomenon (intelligence) which we can choose to implement in different ways, and which we want to harness for some purpose.
That phenomenon is probably not entirely implementable with simple programs. (As you point out yourself.)
The “piles of numbers” are the means by which some people are implementing the thing; as opposed to measurements of the one and only way the thing is implemented.
And so: implementing AIs as piles-of-numbers is not clearly helpful (and might be harmful) to finding better/simpler alternatives.
I think the analogy to {building intelligent systems} is unclear/weak. There seem to be many disanalogies:
In the astronomy case, we have
A phenomenon (the stars and their motions) that we cannot affect
That phenomenon is describable with simple laws.
The “piles of numbers” are detailed measurements of that phenomenon.
It is useful to take more measurements, doing so is helpful for finding the simple laws.
In the {building AIs} case, we have
A phenomenon (intelligence) which we can choose to implement in different ways, and which we want to harness for some purpose.
That phenomenon is probably not entirely implementable with simple programs. (As you point out yourself.)
The “piles of numbers” are the means by which some people are implementing the thing; as opposed to measurements of the one and only way the thing is implemented.
And so: implementing AIs as piles-of-numbers is not clearly helpful (and might be harmful) to finding better/simpler alternatives.