I partially reject the premise in a way that leads to a similar conclusion: humans will be able to tell not by the quality of the construction, but by the choice of message. you might not be able to tell the difference between 1. a human who has ongoing life experiences and 2. a robot who has ongoing life experiences, but 3. a fixed-dataset computer with no motors and no egocentric video from which to observe the world would be detectable by the increasing staleness of the message. it can be perfectly able to replicate what any human up to its time would have done; humans and internet-interactive bots who get to see how the zeitgeist changes, those will get to have new experiences and make art which comments on the latest media in the zeitgeist. hence, I agree more or less with the conclusion, but with the addendum that the librarian must be ongoingly pointing to new rooms, as we stop visiting the exact rooms we’ve been to too many times.
It feels related to the assertion that DNNs can only interpolate between training data points, never extrapolate beyond them. (Technically they can extrapolate, but the results are hilarious/nonsensical/bad in proportion to how far beyond their training distribution they try to go.)
Here’s how I see your argument ‘formalised’ in terms of the two spaces (total combinatorial phase space and a post-threshold GenAI’s latent space over the same output length), please correct anything you think I’ve got wrong:
A model can only be trained on what already exists (though it can augment its dataset to reduce overfitting etc.), and what already exists approximately overlaps with what humans consider to be good, or at least sticky. What humans consider to be good or sticky changes over time; the points in phase space that are not in the latent space of and therefore not accessible to GenAI 1.0 are different from those not in the latent space of GenAI 2.0, assuming GenAI 2.0 has had access to new, external training data from human or other agents with ongoing life experiences. GenAI 1.0 therefore cannot claim to have subsumed GenAI 2.0 before the fact, whereas GenAI 1.0 + 2.0 has a provably broader gamut.
Though I like the spirit of this, it doesn’t quite refute the certainty of provenance-erasure, at least in theory. With a training dataset that ends at time t1, a sufficiently high-fidelity GenAI could (re-)create anything plausibly or actually human-made at or before t1, and only fresh creativity from agents with subjective experience between t1 and t2, when the model’s next generation begins training, is fully ‘protected’ (unless the fresh creativity is actively excluded from the next model’s training data).
Also: what about the limiting case of a CI-style model that is always in combined training-and-inference mode, so is never more than trivially out-of-date? (I know current architectures aren’t designed for this, but it doesn’t seem like a lunatic leap.)
Objections aside, the idea of dynamic interaction (and perhaps Hofstadterian feedback loops) between experiential agent and environment being the secret sauce that DNNs can never recreate is appealing. Can the idea be made rigorous and less dualism-adjacent? What exactly is the dynamic setup getting that DNNs aren’t? Maximal or optimal transfer of order/entropy between agent and environment? An irreducible procedure in which Kolmogorov complexity = string length, so that any attempt to compress or reduce dimensionality kills its Truly Generative spirit?
Though I like the spirit of this, it doesn’t quite refute the certainty of provenance-erasure, at least in theory
Oh indeed, like I said, I don’t think this really undoes your main point.
I think there’s something going on where a probability distribution of the future can’t ever be perfectly tight, and being as brains and social systems and the weather—three of the highest-impact interacting components of earth’s combined dynamical system—are all chaotic, it is effectively guaranteed that any finitely intelligent predictor of the future will be unable to perfectly constrain its predictions. (it can do something, and it can massively exceed us in terms of our sit-in-a-room-and-think prediction capability; but it can never be a perfect copy of the future ahead of time. I think. maybe. ask MIRI.)
so assuming that there’s no galaxy brain shortcut that lets you exactly predict the future, even having a perfectly calibrated distribution, your distribution is not as narrow as the timeline you end up on. and if superdeterminism is false (which is the current default expectation of physicists), quantum also is guaranteed to surprise you. you accumulate more information about what timeline you’re in and not continuing to observe it makes you unaware of goings on.
I partially reject the premise in a way that leads to a similar conclusion: humans will be able to tell not by the quality of the construction, but by the choice of message. you might not be able to tell the difference between 1. a human who has ongoing life experiences and 2. a robot who has ongoing life experiences, but 3. a fixed-dataset computer with no motors and no egocentric video from which to observe the world would be detectable by the increasing staleness of the message. it can be perfectly able to replicate what any human up to its time would have done; humans and internet-interactive bots who get to see how the zeitgeist changes, those will get to have new experiences and make art which comments on the latest media in the zeitgeist. hence, I agree more or less with the conclusion, but with the addendum that the librarian must be ongoingly pointing to new rooms, as we stop visiting the exact rooms we’ve been to too many times.
I like this.
It feels related to the assertion that DNNs can only interpolate between training data points, never extrapolate beyond them. (Technically they can extrapolate, but the results are hilarious/nonsensical/bad in proportion to how far beyond their training distribution they try to go.)
Here’s how I see your argument ‘formalised’ in terms of the two spaces (total combinatorial phase space and a post-threshold GenAI’s latent space over the same output length), please correct anything you think I’ve got wrong:
A model can only be trained on what already exists (though it can augment its dataset to reduce overfitting etc.), and what already exists approximately overlaps with what humans consider to be good, or at least sticky. What humans consider to be good or sticky changes over time; the points in phase space that are not in the latent space of and therefore not accessible to GenAI 1.0 are different from those not in the latent space of GenAI 2.0, assuming GenAI 2.0 has had access to new, external training data from human or other agents with ongoing life experiences. GenAI 1.0 therefore cannot claim to have subsumed GenAI 2.0 before the fact, whereas GenAI 1.0 + 2.0 has a provably broader gamut.
Though I like the spirit of this, it doesn’t quite refute the certainty of provenance-erasure, at least in theory. With a training dataset that ends at time t1, a sufficiently high-fidelity GenAI could (re-)create anything plausibly or actually human-made at or before t1, and only fresh creativity from agents with subjective experience between t1 and t2, when the model’s next generation begins training, is fully ‘protected’ (unless the fresh creativity is actively excluded from the next model’s training data).
Also: what about the limiting case of a CI-style model that is always in combined training-and-inference mode, so is never more than trivially out-of-date? (I know current architectures aren’t designed for this, but it doesn’t seem like a lunatic leap.)
Objections aside, the idea of dynamic interaction (and perhaps Hofstadterian feedback loops) between experiential agent and environment being the secret sauce that DNNs can never recreate is appealing. Can the idea be made rigorous and less dualism-adjacent? What exactly is the dynamic setup getting that DNNs aren’t? Maximal or optimal transfer of order/entropy between agent and environment? An irreducible procedure in which Kolmogorov complexity = string length, so that any attempt to compress or reduce dimensionality kills its Truly Generative spirit?
Oh indeed, like I said, I don’t think this really undoes your main point.
I think there’s something going on where a probability distribution of the future can’t ever be perfectly tight, and being as brains and social systems and the weather—three of the highest-impact interacting components of earth’s combined dynamical system—are all chaotic, it is effectively guaranteed that any finitely intelligent predictor of the future will be unable to perfectly constrain its predictions. (it can do something, and it can massively exceed us in terms of our sit-in-a-room-and-think prediction capability; but it can never be a perfect copy of the future ahead of time. I think. maybe. ask MIRI.)
so assuming that there’s no galaxy brain shortcut that lets you exactly predict the future, even having a perfectly calibrated distribution, your distribution is not as narrow as the timeline you end up on. and if superdeterminism is false (which is the current default expectation of physicists), quantum also is guaranteed to surprise you. you accumulate more information about what timeline you’re in and not continuing to observe it makes you unaware of goings on.