Jessica Taylor. CS undergrad and Master’s at Stanford; former research fellow at MIRI.
I work on decision theory, social epistemology, strategy, naturalized agency, mathematical foundations, decentralized networking systems and applications, theory of mind, and functional programming languages.
Blog: unstableontology.com
Twitter: https://twitter.com/jessi_cata
I think this shades into dark forest theory. Broadly my theory about aliens in general is that they’re not effectively hiding themselves, and we don’t see them because any that exist are too far away.
Partially it’s a matter of, if aliens wanted to hide, could they? Sure, eating a star would show up in terms of light patterns, but also, so would being a civilization at the scale of 2025-earth. And my argument is that these aren’t that far-off in cosmological terms (<10K years).
So, I really think alien encounters are in no way an urgent problem: we won’t encounter them for a long time, and if they get light from 2025-Earth, they’ll already have some idea that something big is likely to happen soon on Earth.
Doesn’t have to expend the energy. It’s about reshaping the matter to machines. Computers take lots of mass-energy to constitute them, not to power them.
Things can go 6 orders of magnitude faster due to intelligence/agency, it’s not highly unlikely in general.
I agree that in theory the arguments here could be better. It might require knowing more physics than I do, and has the “how does Kasparov beat you at chess” problem.
I’m not sure what the details would look like, but I’m pretty sure ASI would have enough new technologies to figure something out within 10,000 years. And expending a bunch of waste heat could easily be worth it, if having more computers allows sending out Von Neumann probes faster / more efficiently to other stars. Since the cost of expending the Sun’s energy has to be compared with the ongoing cost of other stars burning.
I think partially it’s meant to go from some sort of abstract model of intelligence as a scalar variable that increases at some rate (like, on a x/y graph) to concrete, material milestones. Like, people can imagine “intelligence goes up rapidly! singularity!” and it’s unclear what that implies, I’m saying sufficient levels would imply eating the sun, that makes it harder to confuse with things like “getting higher scores on math tests”.
I suppose a more general category would be, the relevant kind of self-improving intelligence would be the sort that can re-purpose mass-energy to creating more computation that can run its intelligence, and “eat the Sun” is an obvious target given this background notion of intelligence.
(Note, there is skepticism about feasibility on Twitter/X, that’s some info about how non-singulatarians react)
Mostly speculation based on tech level. But:
To the extent temperature is an issue, energy can be used to transfer temperature from one place to another.
Maybe matter from the Sun can be physically expelled into more manageable chunks. The Sun already ejects matter naturally (though at a slow rate).
Nanotech in general (cell-like, self-replicating robots).
High energy availability with less-speculative tech like Dyson spheres.
I don’t habitually use the concept so I don’t have an opinion on how to use the term.
I’m not sure this captures what you mean, but, if you see a query, do a bunch of reasoning, and get an answer, then you can build a dataset of (query, well-thought guess). Then you can train an AI model on that.
AlphaZero sorta works like this, because it can make a “well-thought guess” (take value and/or Q network, do an iteration of minimax, then make the value/Q network more closely approximate that, in a fixed point fashion)
Learning stochastic inverses is a specific case of “learn to automate Bayesian inference by taking forward samples and learning the backwards model”. It could be applied to LLMs for example, in terms of starting with a forwards LLM and then using it to train a LLM that predicts things out-of-order.
Paul Christiano’s iterated amplification and distillation is applying this idea to ML systems with a human feedback element. If you can expend a bunch of compute to get a good answer, you can train a weaker system to approximate that answer. Or, if you can expend a bunch of compute to get a good rating for answers, you can use that as RL feedback.
Broadly, I take o3 as evidence that Christiano’s work is broadly on the right track with respect to alignment of near-term AI systems. That is, o3 shows that hard questions can be decomposed into easy ones, in a way that involves training weaker models to be part of a big computation. (I don’t understand the o3 details that well, given it’s partially private, but I’m assuming this describes the general outlines). So I think the sort of schemes Christiano has described will be helpful for both alignment and capabilities, and will scale pretty well to impressive systems.
I’m not sure if there’s a form of amortized inference that you think this doesn’t cover well.
As a follow-up, I tested o1 on stone color problems and it got 7⁄8. This is not what I expected and it is what Paul Christiano expected. Allocate Bayes points accordingly. ChatGPT transcript link
(as a bonus, o1 also solved the 20 words with repetitions problem)
First problem: I have 6 colors of stone: red, green, blue, yellow, white, and black. Arrange the stones in a 2d grid, 3 x 3. I want exactly one row to have the same color stone throughout it, and exactly one column to have the same color stone throughout it. I want every other row to have all different colors, and every other column to have all different colors. And I want every corner color to be different.
RKG
KKK
WKY
(correct)
Second problem: I have 6 colors of stone: red, green, blue, yellow, white, and black. Arrange them in a 4 x 4 grid so that the center 4 stones are the same color and the colors alternate if you go around the edge.
RGBY
KKKW
WKKK
YBGR
(not exactly what I asked for…let’s specify alternate better)
RGRG
KKKB
WKKY
KWYB
(fail)
Third problem: I have 6 colors of stone: red, green, blue, yellow, white, and black. Arrange them in a 4 x 4 grid so that no white/black stone is horizontally or vertically adjacent to a colored stone, and no stone is diagonally adjacent to one of the same color.
RGRG
BYBY
RGRG
BYBY
(correct)
Fourth problem: I have 6 colors of stone: red, green, blue, yellow, white, and black. Arrange them in a 4 x 4 grid so there is a 3 x 3 “o” shape somewhere that has the same color around the edge and a different color in the center, and so that other than on the “o” shape, no two adjacent stones (vertical or horizontal) have the same color.
WWWR
WBWG
WWWR
RGRY
(correct)
Fifth problem: I have 6 colors of stone: red, green, blue, yellow, white, and black. Arrange them in a 5 x 5 grid so that each row alternates between black/white and colored stones (e.g. red could come after white, but black could not come after white, horizontally), and no two adjacent stones (horizontal or vertical) share the same color.
KRWGK
RKGKR
KBWRW
GKRWG
KRKGW
(correct)
Sixth problem: I have 6 colors of stone: red, green, blue, yellow, white, and black. Arrange them in a 5 x 5 grid so that there is one row of the same color, one column of the same color, all four corners are different, and there is a 2 x 2 square of all the same color.
KKRWG
KKRGW
RRRRR
GWRKB
WGRKY
(correct)
Seventh problem: I have 6 colors of stone: red, green, blue, yellow, white, and black. Arrange them in a 4 x 4 grid so if you go around the edges (including corners), you don’t get the same color stone twice in a row.
RGBY
KRGW
WBYK
YBGR
(correct)
Eighth problem: I have 6 colors of stone: red, green, blue, yellow, white, and black. Arrange them in a 4 x 4 grid so that there is at least one black stone, and if you move diagonally from a black stone to another stone (e.g. one stone up and left), you get another black stone; also, no stone should be horizontally or vertically adjacent to one of the same color.
KRKG
BKYK
KWKR
GKBK
(correct)
I think if there were other cases of Olivia causing problems and he was asking multiple people to hide Olivia problems, that would more cause me to think he was sacrificing more group epistemology to protect Olivia’s reputation, and was overall more anti-truth-seeking, yes.
Huh? It seems to come down to definitions of lies, my current intuition is it wouldn’t be a lie, but I’m not sure why people would care how I define lie in this context.
the kind of thing I have heard from Vassar directly is that, in the Lacanian classification of people as psychotic/neurotic/perverted, there are some things to be said in favor of psychotics relative to others, namely, that they have access to the ‘imaginary’ realm that is coherent and scientific (I believe Lacan thinks science is imaginary/psychotic, as it is based on symmetries). however, Lacanian psychosis has the disadvantage that people can catastrophize about ways society is bad.
more specifically, Vassar says, Lacanian neurotics tend to deny oppressive power structures, psychotics tend to acknowledge them and catastrophize about them, and perverts tend to acknowledge and endorse them; under this schema, it seems things could be said in favor of and against all three types.
this raises the question of how much normal (non-expert) and psychiatric concepts of psychosis have to do with the Lacanian model which relates to factors like how much influence Lacan has had on psychiatry. I asked Vassar about this and he said that ‘delusions’ (a standard symptom of psychosis) can be a positive sign because when people form actual beliefs they tend to be wrong (this accords with, for example, Popperian philosophy of science, as specific theories are in general ‘wrong’ even if useful; see also, ‘all models are wrong, some models of useful’)
overall I think further specifying the degree to which anyone is ‘encouraging psychosis’, or the ethics of value judgments on psychosis, would in general require having a more specific definition/notion of psychosis, and the sort of ‘dramatic’ relation people in threads such as this have to psychosis (i.e. moral panics about it) is contra such specificity in definition, therefore, lacks requisite precision for well-informed judgments.
I have no idea about other people lying due to JDP’s influence. I had JDP look at a draft of Occupational Infohazards prior to posting and he convinced me to not mention Olivia because she was young and inexperienced / experimenting with ways of being at the time, it was maybe too bad for her reputation to say she was a possible influence on my psychosis. I admit this was a biased omission, though I don’t think it was a lie. (To be clear, I’m not saying I went psychotic because of Olivia, I think there were many factors and I’m pretty uncertain about the weighting)
claims about Vassar aside, do I even have a reputation for being particularly disagreeable or overconfident, or doing so in the presence of people who have taken psychedelics? to my mind I am significantly less disagreeable and confident than high status rationalists such as Eliezer Yudkowsky and Nate Soares. I think my tendency with trips is to sometimes explore new hypotheses but have relatively low confidence as I’m more likely than usual to change my mind the next day. also, isn’t the ‘modest epistemology’ stuff a pretty thorough criticism of claims that people should not “confidently expound in the nature of truth and society” that has been widely accepted on LW?
as another consideration, I have somewhat of a reputation for being a helpful person for people going through mental health issues (such as psychosis) to talk to, e.g. I let someone with anxiety, paranoia, and benzo issues stay at my place for a bit, she was very thankful and so was her mom. I don’t think this is consistent with the reputation attributed to me re: effects on people in altered states of consciousness.
she talked with him sometimes in group conversations that included other people, 2016-2017. idk if they talked one on one. she stopped talking with him as much sometime during this partially due to Bryce Hidysmith’s influence. mostly, she was interested in learning from him because he was a “wizard”; she also thought of Anna Salamon as a “wizard”, perhaps others. Michael wasn’t specifically like “I am going to teach Olivia things as a studient” afaik, I would not describe it as a “teacher/student relationship”. at this point they pretty much don’t talk and Michael thinks Olivia is suspect/harmful due to the whole Eric Bruylant situation where Eric became obsessed with Vassar perhaps due to Olivia’s influence.
I don’t think so; even if it applies to the subset of hypothetical superintelligences that factor neatly into beliefs and values, humans don’t seem to factorize this way (see Obliqueness Thesis, esp. argument from brain messiness).
Thanks, hadn’t realized how this related to algebraic geometry. Reminds me of semi-simplicial type theory.
Computationally tractable is Yudkowsky’s framing and might be too limited. The kind of thing I believe is for example, an animal without a certain brain complexity will tend not to be a social animal and is therefore unlikely to have the sort of values social animals have. And animals that can’t do math aren’t going to value mathematical aesthetics the way human mathematicians do.
I would totally agree they were directionally correct, I under-estimated AI progress. I think Paul Christiano got it about right.
I’m not sure I agree about the use of hyperbolic words being “correct” here; surely, “hyperbolic” contradicts the straightforward meaning of “correct”.
Partially the state I was in around 2017 was, there are lots of people around me saying “AGI in 20 years”, by which they mean a thing that shortly after FOOMs and eats the sun or something, and I thought this was wrong and a strange set of belief updates (which was not adequately justified, and where some discussions were suppressed because “maybe it shortens timelines”). And I stand by “no FOOM by 2037”.
The people I know these days who seem most thoughtful about the AI that’s around and where it might go (“LLM whisperer” / cyborgism cluster) tend to think “AGI already, or soon” plus “no FOOM, at least for a long time”. I think there is a bunch of semantic confusion around “AGI” that makes people’s beliefs less clear, with “AGI is what makes us $100 billion” as a hilarious example of “obviously economically/politically motivated narratives about what AGI is”.
So, I don’t see these people as validating “FOOM soon” even if they’re validating “AGI soon”, and the local rat-community thing I was objecting to was something that would imply “FOOM soon”. (Although, to be clear, I was still under-estimating AI progress.)