Yeah idk. I think it’s plausible that proofs and products trade off against eachother. There’s a sense in which theoretical legitimacy is a signal that priorities are not in order, that it selects for staff that get excited by the wrong things, whereas deeply product-oriented teams are focused on what users actually care about. In an arms race, or even just scarce modes like layoffs/cuts, we should expect theory-driven teams to either restructure themselves against theory or just lose.
One approach to drawing a reference class: central banking predates macroeconomic theory by centuries, gambling predates probability theory by millenia, enigma was cracked without formal information theory (though the lag here is under a decade). However, this approach fails in at least one of it’s exceptions that I’m thinking about: Godel/church/turing can be said to have provided the “theory of the x86 instruction set” a few decades in advance. Then there are ambiguous cases: like archimedean theory of pressure was available to the engineers of the roman aqueduct, but it’s not clear 1. that the theory was a game changer for the engineers or they would have figured it out anyway, or 2. whether it matters that the theory isn’t the same as the matured theory after civilization continued to iterate.
Importantly I think is observations in the blockchain space: ethereum is not theory driven, it’s very move fast break things, it’s subject to hundreds of millions of dollars of hacks or security failures, its contracting language is intentionally trying to appeal to javascript developers, etc. If theory gets you any gains at all, you should expect that proof-driven blockchain ecosystems to beat ethereum, if nothing else than for the security benefits of theoretical legitimacy. But that happened: a splinter group from the ethereum team went off and started a formal verification -friendly blockchain ecosystem, and (this is not investing advice) it hasn’t gone super well. Basically ethereum says “the whitepaper markets the product” and cardano says “the product implements the whitepaper”, and this strategy has not led to cardano winning. (Note: this example could be washed out by first mover advantage, ethereum was the first mover above an important threshold of contract expressibility, so no reasonable amount of technical superiority may viably work).
So yeah, in spite of not being convinced by the “engineering predates theory” reference class, I still think that an aligned takeoff strategy that rests on deployments abiding soundly and completely to specifications that everyone agrees are right has a deeply slim chance of getting us out of this thing.
Strongly upvoted, this was a very valuable objection.
My theory of change for theorems sounds like Cardano from your example:
My theory of change is using adequate/comprehensive selection theoretic results to design training setups/environments that select for safety properties we want of our systems.
[Copied from a comment I made on Discord.]
Perhaps it’s a pessimistic bet, but decorrelated alignment bets are good. 😋
Yeah idk. I think it’s plausible that proofs and products trade off against eachother. There’s a sense in which theoretical legitimacy is a signal that priorities are not in order, that it selects for staff that get excited by the wrong things, whereas deeply product-oriented teams are focused on what users actually care about. In an arms race, or even just scarce modes like layoffs/cuts, we should expect theory-driven teams to either restructure themselves against theory or just lose.
One approach to drawing a reference class: central banking predates macroeconomic theory by centuries, gambling predates probability theory by millenia, enigma was cracked without formal information theory (though the lag here is under a decade). However, this approach fails in at least one of it’s exceptions that I’m thinking about: Godel/church/turing can be said to have provided the “theory of the x86 instruction set” a few decades in advance. Then there are ambiguous cases: like archimedean theory of pressure was available to the engineers of the roman aqueduct, but it’s not clear 1. that the theory was a game changer for the engineers or they would have figured it out anyway, or 2. whether it matters that the theory isn’t the same as the matured theory after civilization continued to iterate.
Importantly I think is observations in the blockchain space: ethereum is not theory driven, it’s very move fast break things, it’s subject to hundreds of millions of dollars of hacks or security failures, its contracting language is intentionally trying to appeal to javascript developers, etc. If theory gets you any gains at all, you should expect that proof-driven blockchain ecosystems to beat ethereum, if nothing else than for the security benefits of theoretical legitimacy. But that happened: a splinter group from the ethereum team went off and started a formal verification -friendly blockchain ecosystem, and (this is not investing advice) it hasn’t gone super well. Basically ethereum says “the whitepaper markets the product” and cardano says “the product implements the whitepaper”, and this strategy has not led to cardano winning. (Note: this example could be washed out by first mover advantage, ethereum was the first mover above an important threshold of contract expressibility, so no reasonable amount of technical superiority may viably work).
So yeah, in spite of not being convinced by the “engineering predates theory” reference class, I still think that an aligned takeoff strategy that rests on deployments abiding soundly and completely to specifications that everyone agrees are right has a deeply slim chance of getting us out of this thing.
Strongly upvoted, this was a very valuable objection.
My theory of change for theorems sounds like Cardano from your example:
[Copied from a comment I made on Discord.]
Perhaps it’s a pessimistic bet, but decorrelated alignment bets are good. 😋