I really liked this extended passage on math circles from John Psmith’s REVIEW: Math from Three to Seven, by Alexander Zvonkin, it made me wish math circles existed in my home country when I was younger:

in the interviews I’ve read with Soviet mathematicians and scientists, the things that come up over and over again are “mathematical circles,” a practice that originated in the pre-revolutionary Russian Empire and then spread far and wide through the Soviet Union. A mathematical circle is an informal group of teenagers and adults who really enjoy math and want to spend a lot of time thinking and talking about it. They’re a little bit like sports teams, in that they develop their own high-intensity internal culture and camaraderie, and often have a “coach” who is especially talented or famous. But they’re also very unlike sports teams, because they don’t compete with each other or play in leagues or anything like that, and usually any given circle will contain members of widely varying skill levels. Maybe a better analogy is a neighborhood musical ensemble that gets together and jams on a regular basis, but for math.

The most important thing to understand about mathematical circles is that the math they jam on is completely unlike the math you study in school, and also completely unlike the “competition” math that bright kids in the United States sometimes do. Both school math and competition math are primarily comprised of exercises. An exercise is a question concocted by a human being for a didactic purpose. Any bright kid with any amount of genre-savviness can immediately make a few assumptions upon being assigned an exercise. He or she can guess that the exercise is solvable in fewer than five minutes with the appropriate techniques, and that it is related to the material in the current chapter of the book. A clever student can often use psychological techniques to reverse-engineer what the teacher or the designer of the standardized test was trying to get at with the exercise, and answer it through a process of elimination or savvy guessing or pattern matching.

Solving an exercise is like hunting a neutered zoo animal. It may be a low-stress environment for polishing particular aspects of your technique, but it will not help you to survive in the wilderness. For that, you need to see people solving problems. A problem is a question of interest that comes up when somebody is trying to do something real. A problem may not be solvable by you, or by your coach, or by any human being. Even if the problem is solvable, it may require weeks or months of dedicated, painful pursuit. It may not be obvious what techniques are required to solve a problem, they may not be techniques that you know, or it may require a surprising combination of techniques. The problem is mathematical nature red in tooth and claw. There are no guardrails. There are no hints or answers at the back of the book. There is no book. It may eat you.

The bread and butter of the mathematical circle is solving problems together, as a team. There is no time here for exercises; you can do that lame stuff at school. Sometimes the coach picks a problem for you, something just beyond your ability, just the thing you need to hone your edge. But sometimes the whole circle works together on a problem that nobody has the answer to and that challenges the very best members. These problems are the most important, because with them you see great minds, men older and more talented than you, stretched to the breaking point and occasionally beaten. You see them grind and grind and try every possible attack on a problem and sometimes lose anyway. And you see them not run from being defeated, but cheerfully charge in again, because losing is good for you, losing is how you know you’ve picked an opponent worthy of a man. You learn to love things that are hard. And occasionally you win, and when you win it feels like you all win, like humanity wins, because you’re all in it together, all doing something beautiful and dangerous and exemplary of the best qualities that human beings have.

There are also times when everybody is too tired to work on a problem, and in those moments of recuperation, it’s the coach’s job to tell stories of legendary problems of the past and of the mathematicians who slew them. These stories often contain lessons, inspiration, or perspective on how mathematics evolved and got to be the way it is. Human history would look very different, after all, without the brachistochrone problem or the roots of a quintic polynomial problem or the icosahedron problem or the precession of Mercury’s perihelion problem. But other times there’s no hidden lesson, no grand perspective on the human story. They’re just ripping good yarns, and hearing them is a process of initiation into mathematical folklore, because every culture (and mathematics is surely a culture) has shared stories and references and inside jokes, even when they’re purely for fun.

You can start math circles really really young:

This book is the story of one such mathematical circle. But it’s an unusual one because…it’s for preschoolers.

The “coach” of this circle is Alexander Zvonkin, a professional mathematician frustrated that his kids are having all the wonder and life and joy crushed out of them by the grey functionaries at their school. So he starts a circle for his son Dmitry and a few of the neighbors’ kids, most of whom are around three or four years old. That’s young enough that according to Piaget’s experiments there are cognitive modules related to number and volume that simply haven’t come online yet. Fortunately, Zvonkin is familiar with the latest research on developmental psychology, and turns lemons into lemonade by using the kids’ lack of numerical intuition to introduce them to some pretty deep ideas about when two sets have equal cardinality. (If you’re curious, he talks more about these experiments in this journal article.)

At this point I expect you are rolling your eyes, especially if you have experience with three-year-olds. It can be difficult enough to get them to sit still, never mind ponder deep questions about the cardinalities of sets. And what exactly does it look like to pit somebody against a problem who is barely potty-trained? This is where the genius of Zvonkin’s format kicks in — it’s not really a book, it’s a journal, and one that is barely edited. So it’s full of failure after failure, entries like, “today I had a cool idea for a puzzle but everybody just screamed instead and then one of the kids vomited.” And yet, slowly, wondrously, over the four years of the circle’s existence, his patience pays off and the kids start doing really incredible things.

(Sadly I only learned of the existence of math circles well after graduation, a few years ago when I used to spend more time on Quora and noticed that Alon Amit, the most respected writer on math topics and someone who’d done many interesting things in his life, described himself simply as a “mathcircler”.)

Non-simulated beings cannot use them to predict the universe’s future or hypothetical universes’ states without waiting a significant amount of time, comparable to the system itself reaching that state if not much longer. Thereby, such beings will less likely be tempted to create simulated universes...

You’ve done an impressive job of clearly and succinctly explaining the simulation hypothesis, the significance of sim blockers, and the concept of computational irreducibility. I agree with your premise that computational irreducibility makes it harder to use simulations to predict the future, and therefore may result in fewer simulations being created. However, that is only true for simulations that attempt to predict what will happen starting with where things actually are right now.

One useful application of simulations is determining what would happen if a particular set of circumstances should occur in the future. For example, what would happen if there was a sudden worldwide outbreak of a highly contagious virus, or what would happen if a celebrity with no political experience or qualifications were elected president of the United States.

If the starting point of the simulation differs from the present configuration of the universe, it doesn’t matter whether or not the simulation can compute the state of the universe faster than the universe itself reaches that state— because the simulation and the universe are at different starting points. The predictive usefulness of such a counterfactual simulation would depend on the possibility that the real universe may, at some point in the future, approximate the starting point of the simulation.

This sort of simulation could be quite useful so there could be many of them. However, this doesn’t challenge your core premise. Simulations will still be less popular than they would be if they could predict where the universe is going to go FROM HERE faster than it actually does.

Here’s the (obvious) strategy: Apply voluntary attention-control to keep S(getting out of bed) at the center of attention. Don’t let it slip away, no matter what.

Can you explain more precisely how this works mechanistically? What is happening to keep S(getting out of bed) in the center of attention.

8.5.6.1 Aside: The “innate drive to minimize voluntary attention control”

Your hypothesis here doesn’t seem to me to explain why we seem to have limited willpower budget for attention control which gets depleted but which also regenerates after a time. I can see how negative rewards from minimizing voluntary attention control can make us less likely to apply willpower in the future, but why would it regenerate then?

I’ve always regarded the Litany as normative as much as descriptive. If you find yourself flinching away from discovering a truth, you are doing something wrong.

Btw, there’s another simpler possible mechanism, though I don’t know the neuroscience and perhaps Steve’s hypothesis with separate valence assessors and involuntary attention control fits the neuroscience evidence much better and it may also fit observed motivated reasoning better.

But the obvious way to design a mind would be to make it just focus on whatever is most important, aka where most expected utility per necessary resources could be gained.

So we still have a learned value function which assigns how good/​bad something would be, but we also have an estimator of how much the value would increase if we continue thinking (which might e.g. happen because one makes plans for making a somewhat bad situation better), and what gets attended on depends on this estimator, not the value function directly.

Thank you! However, colonialism also has a moral aspect: I conjectured that any system of minds eventually converges to respecting the weaker ones or disrespecting them. If humanity creates an AI that treats mankind as if the AI was an evil explorer and mankind was an alien race that the AI met, then the AI would obviously be misaligned. And if the AI treats mankind as if the AI was a good explorer or a progressor?

Yeah, that was a good response, thank you for your thoughts.

Ok, yes that makes a lot more sense—whilst tarnishing by association increases incentives to point out flaws in your friend, it decreases incentives to point out flaws in your friend’s friend.

And since most of your friends are also your friends’ friends, the aggregate impact is to decrease incentives to point out flaws in your friends as well.

Hmm. There are lots of valuable advocacy targets besides literal regulation; advocates might try to pass laws, win court cases, sustain a boycott, amend a corporate charter, amend a voluntary industry standard at NIST, and so on. I’ll discuss several examples in post #5.

I suppose advocacy might also have some side benefits like teaching us more about politics, helping us bond as a community, or giving us a sense of efficacy, but it’s not obvious to me that any of those are of comparable importance to reducing the likelihood that AI developers release misaligned superintelligence.

Does that answer your question? If not, please let me know more about what you mean and I’ll try to elaborate.

This is fascinating! If there’s nothing else going on with your prompting, this looks like an incredibly hacky mid-inference intervention. My guess would be that openai applied some hasty patch against a sycophancy steering vector and this vector caught both actual sycophantic behaviors and descriptions of sycophantic behaviors in LLMs (I’d guess “sycophancy” as a word isn’t so much the issue as the LLM behavior connotation). Presumably the patch they used activates at a later token in the word “sycophancy” in an AI context. This is incredibly low-tech and unsophisticated—like much worse than the stories of repairing Apollo missions with duct tape. Even a really basic finetuning would not exhibit this behavior (otoh, I suppose stuff like this works for humans, where people will sometimes redirect mid-sentence).

FWIW, I wasn’t able to reconstruct this exact behavior (working in an incognito window with a fresh chatgpt instance), but it did suspiciously avoid talking about sycophancy and when I asked about sycophancy specifically, it got stuck in inference and returned an error

I thought for some time that we would just scale up models and once we reached enough parameters we’d get an AI with a more precise and comprehensive world-model than humans, at which point the AI would be a more advanced general reasoner than humans.

But it seems that we’ve stopped scaling up models in terms of parameters and are instead scaling up RL post-training. Does RL sidestep the need for surpassing (equivalently) the human brain’s neurons and neural connections? Or by scaling up RL on these sub-human (in the sense described) models necessarily just lead to models which are only superhuman in narrow domains, but which are worse general reasoners?

I recognise my ideas here are not well-developed, hoping someone will help steer my thinking in the right direction.

Countersignaling vaguely reminds me of Benign Boundary Violations! Both of these work when you know the other person well enough – which can be nice by making them feel seen.

Cool, yes, I agree. But the reason other insiders don’t like that public criticism is because it reduces their status by association. Your colleagues paid to get a position in a status hierarchy which you are devaluing, and they make you internalize those costs.

The premise of

The “S” in “S(X)” and “S(A)” seems different to me. If I rename the “S” in “S(A)” to “I”, it would make more sense to me:

• A = action of standing up (which gets actually executed if positive valence)

• I(A) = imagined scene of myself standing up

• S(I(A)) = the thought “I am thinking about standing up”

Mind modeling—surprisingly good even out of the box for many famous people who left extensive diaries etc like Leo Tolstoy.

With some caveats also good in my-mind-modeling based on very long prompt. Sometimes it is too good: it extract memories from memory quicker than I do in normal life.

This could also be explained by many real-world prompts being easy to judge as real e.g. because they would have no value as evaluations.

Given just the results, it seems to me difficult to rule out the hypothesis that LLMs currently judge any high-stakes situation, or one involving a significant moral dilemma, as likely to be an evaluation. After all, in today’s environments, this would be a pretty accurate heuristic!

(That said, it already seemed quite likely to me that models can tell when they are in evaluations most of the time, but I’m not sure how much this evidence moves me on that hypothesis vs. the one I suggest above.)

I miss a crucial consideration in the You’re In the Wrong Place part: Successful places for meeting mates in the past (work, school, club...) in the past were not about dating.

As I have said a few times before: Dating apps are broken because of misaligned incentives (see more in the link).

I think the best dating apps are not “dating” apps—you can’t have plausible deniability if the app has “dating” in the name. But every community or forum app automatically becomes a dating app because people can observe many potential mates and interact with them. If you want to build an app where people date successfully, you

• may not call it that!

• you need to offer a way to find communities that share some interests (and hopefully locality) and

• there should be a path to getting closer incrementally (though this is optional—if people meet in person, there will be ways for that to happen organically.

Of course, this will fail if there is only one gender in each forum, but there may be a plausible deniable way to balance gender at least somewhat.

Ah. Thank you for your attempt to get through to this community anyway, in the face of such incompatibility. Enjoy your freedom then, I hope you’ll do better than us.

Do you perceive there are any benefits to AI governance advocacy beside regulation?