failure mode can be understood as trying to aristotle the problem, lack of experimentation
thinking about the nanotech ASI threat model, where it solves nanotech overnight and deploys adversarial proteins in all the bloodstreams of all the lifeforms.
These are sometimes justified by Drexler’s inside view of boundary conditions and physical limits.
But to dodge the aristotle problem, there would have to be an amount of bandwidth of what’s passing between sensors and actuators (which may roughly correspond to the number of do applications in pearl)
Can you use something like communication complexity https://en.wikipedia.org/wiki/Communication_complexity (between a system and an environment) to think about “lower bound on the number of sensor-actuator actions” mixed with sample complexity (statistical learning theory)
Like ok if you’re simulating all of physics you can aristotle nanotech, for a sufficient definition of “all” that you would run up against realizability problems and cost way more than you actually need to spend.
Like I’m thinking if there’s a kind of complexity theory of pearl (number of do applications needed to acquire some kind of “loss”), then you could direct that at something like “nanotech projects” to fermstimate the way AIs might tradeoff between applying aristotlean effort (observation and induction with no experiment) and spending sensor-actuator interactions (with the world).
There’s a scenario in the sequences if I recall correctly about which physics an AI infers from 3 frames of a video of an apple falling, and something about how security mindset suggests you shouldn’t expect your information-theoretic calculation that einsteinian physics is impossible to believe from the three frames to actually apply to the AI. Which is a super dumbed down way of opening up this sort of problem space.
messy, jotting down notes:
I saw this thread https://twitter.com/alexschbrt/status/1666114027305725953 which my housemate had been warning me about for years.
failure mode can be understood as trying to aristotle the problem, lack of experimentation
thinking about the nanotech ASI threat model, where it solves nanotech overnight and deploys adversarial proteins in all the bloodstreams of all the lifeforms.
These are sometimes justified by Drexler’s inside view of boundary conditions and physical limits.
But to dodge the aristotle problem, there would have to be an amount of bandwidth of what’s passing between sensors and actuators (which may roughly correspond to the number of
doapplications in pearl)Can you use something like communication complexity https://en.wikipedia.org/wiki/Communication_complexity (between a system and an environment) to think about “lower bound on the number of sensor-actuator actions” mixed with sample complexity (statistical learning theory)
Like ok if you’re simulating all of physics you can aristotle nanotech, for a sufficient definition of “all” that you would run up against realizability problems and cost way more than you actually need to spend.
Like I’m thinking if there’s a kind of complexity theory of pearl (number of
doapplications needed to acquire some kind of “loss”), then you could direct that at something like “nanotech projects” to fermstimate the way AIs might tradeoff between applying aristotlean effort (observation and induction with no experiment) and spending sensor-actuator interactions (with the world).There’s a scenario in the sequences if I recall correctly about which physics an AI infers from 3 frames of a video of an apple falling, and something about how security mindset suggests you shouldn’t expect your information-theoretic calculation that einsteinian physics is impossible to believe from the three frames to actually apply to the AI. Which is a super dumbed down way of opening up this sort of problem space.