I wasn’t eager on this, but your justification updated me a bit. I think the most important distinction is indeed the ‘grown/evolved/trained/found, not crafted’, and ‘brainware’ didn’t immediately evoke that for me. But you’re right, brains are inherently grown, they’re very diverse, we can probe them but don’t always/ever grok them (yet), structure is somewhat visible, somewhat opaque, they fit into a larger computational chassis but adapt to their harness somewhat, properties and abilities can be elicited by unexpected inputs, they exhibit various kinds of learning on various timescales, …
Incidentally I noticed Yudkowsky uses ‘brainware’ in a few places (e.g. in conversation with Paul Christiano). But it looks like that’s referring to something more analogous to ‘architecture and learning algorithms’, which I’d put more in the ‘software’ camp when in comes to the taxonomy I’m pointing at (the ‘outer designer’ is writing it deliberately).
I wasn’t eager on this, but your justification updated me a bit. I think the most important distinction is indeed the ‘grown/evolved/trained/found, not crafted’, and ‘brainware’ didn’t immediately evoke that for me. But you’re right, brains are inherently grown, they’re very diverse, we can probe them but don’t always/ever grok them (yet), structure is somewhat visible, somewhat opaque, they fit into a larger computational chassis but adapt to their harness somewhat, properties and abilities can be elicited by unexpected inputs, they exhibit various kinds of learning on various timescales, …
Incidentally I noticed Yudkowsky uses ‘brainware’ in a few places (e.g. in conversation with Paul Christiano). But it looks like that’s referring to something more analogous to ‘architecture and learning algorithms’, which I’d put more in the ‘software’ camp when in comes to the taxonomy I’m pointing at (the ‘outer designer’ is writing it deliberately).