Was on Vivek Hebbar’s team at MIRI, now working with Adrià Garriga-Alonso onvarious empirical alignment projects.
I’m looking for projects in interpretability, activation engineering, and control/oversight; DM me if you’re interested in working with me.
I have signed no contracts or agreements whose existence I cannot mention.
The thought experiment is not about the idea that your VNM utility could theoretically be doubled, but instead about rejecting diminishing returns to actual matter and energy in the universe. SBF said he would flip with a 51% of doubling the universe’s size (or creating a duplicate universe) and 49% of destroying the current universe. Taking this bet requires a stronger commitment to utilitarianism than most people are comfortable with; your utility needs to be linear in matter and energy. You must be the kind of person that would take a 0.001% chance of colonizing the universe over a 100% chance of colonizing merely a thousand galaxies. SBF also said he would flip repeatedly, indicating that he didn’t believe in any sort of bound to utility.
This is not necessarily crazy—I think Nate Soares has a similar belief—but it’s philosophically fraught. You need to contend with the unbounded utility paradoxes, and also philosophical issues: what if consciousness is information patterns that become redundant when duplicated, so that only the first universe “counts” morally?
For context, I just trialed at METR and talked to various people there, but this take is my own.
I think further development of evals is likely to either get effective evals (informal upper bound on the future probability of catastrophe) or exciting negative results (“models do not follow reliable scaling laws, so AI development should be accordingly more cautious”).
The way to do this is just to examine models and fit scaling laws for catastrophe propensity, or various precursors thereof. Scaling laws would be fit to elicitation quality as well as things like pretraining compute, RL compute, and thinking time.
In a world where elicitation quality has very reliable scaling laws, we would observe that there are diminishing returns to better scaffolds. Elicitation quality is predictable, ideally an additive term on top of model quality, but more likely requiring some more information about the model. It is rare to ever discover a new scaffold that can 2x the performance of an already well-tested models.
In a world where elicitation quality is not reliably modelable, we would observe that different methods of elicitation routinely get wildly different bottom-line performance, and sometimes a new elicitation method makes models 10x smarter than before, making error bars on the best undiscovered elicitation method very wide. Different models may benefit from different elicitation methods, and some get 10x benefits while others are unaffected.
It is NOT KNOWN what world we are in (worst-case assumptions would put us in 2 though I’m optimistic we’re closer to 1 in practice), and determining this is just a matter of data collection. If our evals are still not good enough but we don’t seem to be in World 2 either, there are endless of tricks to add that make evals more thorough, some of which are already being used. Like evaluating models with limited human assistance, or dividing tasks into subtasks and sampling a huge number of tries for each.
What’s the most important technical question in AI safety right now?
Yes, lots of socioeconomic problems have been solved on a 5 to 10 year timescale.
I also disagree that problems will become moot after the singularity unless it kills everyone—the US has a good chance of continuing to exist, and improving democracy will probably make AI go slightly better.
I mention exactly this in paragraph 3.
The new font doesn’t have a few characters useful in IPA.
The CATXOKLA population is higher than the current swing state population, so it would arguably be a little less unfair overall. Also there’s the potential for a catchy pronunciation like /kæ′tʃoʊklə/.
Knowing now that he had an edge, I feel like his execution strategy was suspect. The Polymarket prices went from 66c during the order back to 57c on the 5 days before the election. He could have extracted a bit more money from the market if he had forecasted the volume correctly and traded against it proportionally.
I think it would be better to form a big winner-take-all bloc. With proportional voting, the number of electoral votes at stake will be only a small fraction of the total, so the per-voter influence of CA and TX would probably remain below the national average.
A third approach to superbabies: physically stick >10 infant human brains together while they are developing so they form a single individual with >10x the neocortex neurons as the average humans. Forget +7sd, extrapolation would suggest they are >100sd intelligence.
Even better, we could find some way of networking brains together into supercomputers using configurable software. This would reduce potential health problems and also allow us to harvest their waste energy. Though we would have to craft a simulated reality to distract the non-useful conscious parts of the computational substrate, perhaps modeled on the year 1999...
In many respects, I expect this to be closer to what actually happens than “everyone falls over dead in the same second” or “we definitively solve value alignment”. Multipolar worlds, AI that generally follows the law (when operators want it to, and modulo an increasing number of loopholes) but cannot fully be trusted, and generally muddling through are the default future. I’m hoping we don’t get instrumental survival drives though.
Claim 2: The world has strong defense mechanisms against (structural) power-seeking.
I disagree with this claim. It seems pretty clear that the world has defense mechanisms against
disempowering other people or groups
breaking norms in the pursuit of power
But it is possible to be power-seeking in other ways. The Gates Foundation has a lot of money and wants other billionaires’ money for its cause too. It influences technology development. It has to work with dozens of governments, sometimes lobbying them. Normal think tanks exist to gain influence over governments. Harvard University, Jane Street, and Goldman Sachs recruit more elite students than all the EA groups and control more money than OpenPhil. Jane Street and Goldman Sachs guard private information worth billions of dollars. The only one with a negative reputation is Goldman Sachs, which is due to perceived greed rather than power-seeking per se. So why is there so much more backlash against AI safety? I think it basically comes down to a few factors:
We are bending norms (billionaire funding for somewhat nebulous causes) and sometimes breaking them (FTX financial and campaign finance crimes)
We are not able to credibly signal that we won’t disempower others.
MIRI wanted a pivotal act to happen, and under that plan nothing would stop MIRI from being world dictators
AI is inherently a technology with world-changing military and economic applications whose governance is unsolved
An explicitly consequentialist movement will take power by any means necessary, and people are afraid of that.
AI labs have incentives to safetywash, making people wary of safety messaging.
The preexisting AI ethics and open-source movements think their cause is more important and x-risk is stealing attention.
AI safety people are bad at diplomacy and communication, leading to perceptions that they’re the same as the AI labs or have some other sinister motivation.
That said, I basically agree with section 3. Legitimacy and competence are very important. But we should not confuse power-seeking—something the world has no opinion on—with what actually causes backlash.
Fixed.
Yeah that’s right, I should have said market for good air filters. My understanding of the problem is that most customers don’t know to insist on high CADR at low noise levels, and therefore filter area is low. A secondary problem is that HEPA filters are optimized for single-pass efficiency rather than airflow, but they sell better than 70-90% efficient MERV filters.
The physics does work though. At a given airflow level, pressure and noise go as roughly the −1.5 power of filter area. What IKEA should be producing instead of the FÖRNUFTIG and STARKVIND is one of three good designs for high CADR:
a fiberboard box like the CleanAirKits End Table 7 which has holes for pre-installed fans and can accept at least 6 square feet of MERV 13 furnace filters or maybe EPA 11.
a box like the AirFanta 3Pro, ideally that looks nicer somehow.
a wall-mounted design with furnace filters in a V shape, like this DIY project.
I made a shortform and google slides presentation about this and might make it a longform if there is enough interest or I get more information.
Quiet air filters is an already solved problem technically. You just need enough filter area that the pressure drop is low, so that you can use quiet low-pressure PC fans to move the air. CleanAirKits is already good, but if the market were big enough cared enough, rather than CleanAirKits charging >$200 for a box with holes in it and fans, you would get a purifier from IKEA for $120 which is sturdy and 3db quieter due to better sound design.
Haven’t fully read the post, but I feel like that could be relaxed. Part of my intuition is that Aumann’s theorem can be relaxed to the case where the agents start with different priors, and the conclusion is that their posteriors differ by no more than their priors.
I agree that with superficial observations, I can’t conclusively demonstrate that something is devoid of intellectual value. However, the nonstandard use of words like “proof” is a strong negative signal on someone’s work.
If someone wants to demonstrate a scientific fact, the burden of proof is on them to communicate this in some clear and standard way, because a basic strategy of anyone practicing pseudoscience is to spend lots of time writing something inscrutable that ends in some conclusion, then claim that no one can disprove it and anyone who thinks it’s invalid is misunderstanding something inscrutable.
This problem is exacerbated when someone bases their work on original philosophy. To understand Forrest Landry’s work to his satisfaction someone will have to understand his 517-page book An Immanent Metaphysics, which uses words like “proof”, “theorem”, “conjugate”, “axiom”, and “omniscient” in a nonstandard sense, and also probably requires someone to have a background in metaphysics. I scanned the 134-page version, can’t make any sense of it, and found several concrete statements that sound wrong. I read about 50 pages of various articles on the website and found them to be reasonably coherent but often oddly worded and misusing words like entropy, with the same content quality as a ~10 karma LW post but super overconfident.
That researcher is now collaborating with Anders Sandberg to formalise an elegant model of AGI uncontainability in mathematical notation.
Ok. To be clear I don’t expect any Landry and Sandberg paper that comes out of this collaboration to be crankery. Having read the research proposal my guess is that they will prove something roughly like the Good Regulator Theorem or Rice’s theorem which will be slightly relevant to AI but not super relevant because the premises are too strong, like the average item in Yampolskiy’s list of impossibility proofs (I can give examples if you want of why these are not conclusive).
I’m not saying we should discard all reasoning by someone that claims an informal argument is a proof, but rather stop taking their claims of “proofs” at face value without seeing more solid arguments.
claiming the “proof” uses mathematical arguments from Godel’s theorem, Galois Theory,
Nope, I haven’t claimed either of that.
Fair enough. I can’t verify this because Wayback Machine is having trouble displaying the relevant content though.
Paul had zoned in on a statement of the conclusion, misinterpreted what was meant, and then moved on to dismissing the entire project. Doing this was not epistemically humble.
Paul expressed appropriate uncertainty. What is he supposed to do, say “I see several red flags, but I don’t have time to read a 517-page metaphysics book, so I’m still radically uncertain whether this is a crank or the next Kurt Godel”?
Read the core argument please (eg. summarised in point 3-5. above) and tell me where you think premises are unsound or the logic does not follow from the premises.
When you say failures will “build up toward lethality at some unknown rate”, why would failures build up toward lethality? We have lots of automated systems e.g. semiconductor factories, and failures do not accumulate until everyone at the factory dies, because humans and automated systems can notice errors and correct them.
Variants get evolutionarily selected for how they function across the various contexts they encounter over time. [...] The artificial population therefore converges on fulfilling their own expanding needs.
This is pretty similar to Hendrycks’s natural selection argument, but with the additional piece that the goals of AIs will converge to optimizing the environment for the survival of silicon-based life. He claims that there are various ways to counter evolutionary pressures, like “carefully designing AI agents’ intrinsic motivations, introducing constraints on their actions, and institutions that encourage cooperation”. In the presence of ways to change incentives such that benign AI systems get higher fitness, I don’t think you can get to 99% confidence. Evolutionary arguments are notoriously tricky and respected scientists get them wrong all the time, from Malthus to evolutionary psychology to the group selectionists.
I eat most meats (all except octopus and chicken) and have done this my entire life, except once when I went vegan for Lent. This state seems basically fine because it is acceptable from scope-sensitive consequentialist, deontic, and common-sense points of view, and it improves my diet enough that it’s not worth giving up meat “just because”.
According to EA-style consequentialism, eating meat is a pretty small percentage of your impact, and even if you’re not directly offsetting, the impact can be vastly outweighed by positive impact in your career or donations to other causes.
There is a finite amount of sadness I’m willing to put up with for the sake of impact, and it seems far more important to use the vast majority of my limited sadness budget in my career choice.
There is no universally accepted deontological rule against indirectly causing the expected number of tortured animals to increase by one, nor would this be viable as it requires tracking the consequences of your actions through the complicated world, which defeats the point of deontology. There might be a rule against benefiting from identifiable torture, but I don’t believe in deontology strongly enough to think this is definitive. Note there isn’t a good contractualist angle against torturing animals like there is for humans.
Common-sense morality says that meat-eating is traditional and not torturing the animals yourself does reduce how bad it is, and although this is pretty silly as a general principle, it applies to the other benefit of being vegan, which is less corrupted moral reasoning. My empathy and moral reasoning are less corrupted by eating meat than it would be working in a factory farm or slaughterhouse. I am still concerned about loss of empathy but I get around half the empathy benefits of veganism anyway, just by not eating chicken.
I do have some doubts; sometimes eating meat feels like being a slaveholder in 1800, which feels pretty bad. I hope history will not judge me harshly for what seem like reasonable decisions now, and plan to go vegan or move to a high-welfare-only diet when it’s easier.
The North Wind, the Sun, and Abadar
One day, the North Wind and the Sun argued about which of them was the strongest. Abadar, the god of commerce and civilization, stopped to observe their dispute. “Why don’t we settle this fairly?” he suggested. “Let us see who can compel that traveler on the road below to remove his cloak.”
The North Wind agreed, and with a mighty gust, he began his effort. The man, feeling the bitter chill, clutched his cloak tightly around him and even pulled it over his head to protect himself from the relentless wind. After a time, the North Wind gave up, frustrated.
Then the Sun tried his turn. Beaming warmly from the heavens, the Sun caused the air to grow pleasant and balmy. The man, feeling the growing heat, loosened his cloak and eventually took it off in the heat, resting under the shade of a tree. The Sun began to declare victory, but as soon as he turned away, the man put on the cloak again.
The god of commerce then approached the traveler and bought the cloak for five gold coins. The traveler tucked the money away and continued on his way, unbothered by either wind or heat. He soon bought a new cloak and invested the remainder in an index fund. The returns were steady, and in time he prospered far beyond the value of his simple cloak, while the cloak was Abadar’s permanently.
Commerce, when conducted wisely, can accomplish what neither force nor gentle persuasion alone can achieve, and with minimal deadweight loss.