Real-money markets do have stronger incentives for sharps to scour for arbitrage, so the 1/1/26 market would have been more likely to be noticed before months had gone by.
However (depending on the fee structure for resolving N/A markets), real-money markets have even stronger incentives for sharps to stay away entirely from spurious conditional markets, since they’d be throwing away cash and not just Internet points. Never ever ever cite out-of-the-money conditional markets.
Broke: Prediction markets are an aggregation of opinions, weighted towards informed opinions by smart people, and are therefore a trustworthy forecasting tool on any question.
Woke: Prediction markets are MMOs set in a fantasy world where, if someone is Wrong On The Internet, you can take their lunch money.
Can you share any strong evidence that you’re an unusually trustworthy person in regard to confidential conversations? People would in fact be risking a lot by talking to you.
(This is sincere btw; I think this service should absolutely exist, but the best version of it is probably done by someone with a longstanding public reputation of circumspection.)
Good question! I picked it up from a friend at a LW meetup a decade ago, so it didn’t come with all the extra baggage that vipassana meditation seems to usually carry. So this is just going to be the echo of it that works for me.
Step 1 is to stare at your index finger (a very sensitive part of your body) and gently, patiently try to notice that it’s still producing a background level of sensory stimulus even when it’s not touching anything. That attention to the background signal, focused on a small patch of your body, is what the body scan is based on.
Step 2 is learning how to “move” that awareness of the background signal slowly. Try to smoothly shift that awareness down your finger, knuckle by knuckle, keeping the area of awareness small by ceasing to focus on the original spot as you focus on a new spot. Then try moving that spot of awareness gradually to the base of your thumb, and noticing the muscle beneath the skin.
Use Case α is harnessing that kind of awareness to relax physical tension and even pain. The next time you have a paper cut or a small burn, once you’ve dealt with it in the obvious objective ways and now just have to handle the pain, focus your awareness right on that spot. The sensation will still be loud, but it won’t be overwhelming when you’re focusing on it rather than fleeing from it. Or the next time you notice a particularly tense muscle, focus your awareness there; for me, that usually loosens it at least a little.
Step 3 is the body scan itself: creating awareness for each part of your skin and muscles, gradually, bit by bit, starting from the crown of your head and slowly tracing out a path that covers everything. This is where a guided meditation could really help. I don’t have one to recommend (after having the guided meditation at the meetup, I got as much of the idea as I needed), but hopefully some of the hundreds out there are as good as Random Meditating Rationalist #37 was.
And Use Case β, when you have a migraine, is to imagine moving that awareness inside your skull, to the place where the migraine pain feels like it’s concentrated. (I recommend starting from a place where the migraine seems to “surface”—for me, the upper orbit of my left eye—if you have such a spot.)
There’s something quite odd about how this works: your brain doesn’t have pain receptors, so the pain from the migraine ends up in some phantom location on your body map, and it’s (conveniently?) interpreted as being inside your head. By tracing your awareness inside your skull, you walk along that body map to the same phantom location as that pain, so it works out basically the same as if you were in Use Case α.
Hope this helps!
I have to further compliment my past self: this section aged extremely well, prefiguring the Shoggoth-with-a-smiley-face analogies several years in advance.
GPT-3 is trained simply to predict continuations of text. So what would it actually optimize for, if it had a pretty good model of the world including itself and the ability to make plans in that world?
One might hope that because it’s learning to imitate humans in an unsupervised way, that it would end up fairly human, or at least act in that way. I very much doubt this, for the following reason:
Two humans are fairly similar to each other, because they have very similar architectures and are learning to succeed in the same environment.
Two convergently evolved species will be similar in some ways but not others, because they have different architectures but the same environmental pressures.
A mimic species will be similar in some ways but not others to the species it mimics, because even if they share recent ancestry, the environmental pressures on the poisonous one are different from the environmental pressures on the mimic.
What we have with the GPTs is the first deep learning architecture we’ve found that scales this well in the domain (so, probably not that much like our particular architecture), learning to mimic humans rather than growing in an environment with similar pressures. Why should we expect it to be anything but very alien under the hood, or to continue acting human once its actions take us outside of the training distribution?
Moreover, there may be much more going on under the hood than we realize; it may take much more general cognitive power to learn and imitate the patterns of humans, than it requires us to execute those patterns.
The spun-off agent foundations team seems to have less reason than most AI safety orgs to be in the Bay Area, so moving to NZ might be worth considering for them.
Note on current methodology:
I am, for now, not doing further research when the spreadsheet lists a person whose name appears on the final leaked letter; so it’s possible that some of the 23 departures among the 702 names on the final leaked letter are spurious. (I will be more thorough when I resolve the market after November.)
I am counting only full-time employees and not counting contractors, as I currently believe that the 770 figure refers only to full-time employees. So far, I’ve seen no contractors among those who signed, but I’ve only checked a few; if the letter includes some categories of contractors, this gets a lot harder to resolve.
I am counting nontechnical employees (e.g. recruiting, marketing) as well as technical staff, because such employees were among those who signed the letter.
Counterpoint: other labs might become more paranoid that SSI is ahead of them. I think your point is probably more correct than the counterpoint, but it’s worth mentioning.
Elon diversifies in the sense of “personally micromanaging more companies”, not in the sense of “backing companies he can’t micromanage”.
By my assessment, the employees who failed to sign the final leaked version of the Altman loyalty letter have now been literally decimated.
I’m trying to track the relative attrition for a Manifold market: of the 265 OpenAI employees who hadn’t yet signed the loyalty letter by the time it was first leaked, what percent will still be at OpenAI on the one-year anniversary?
I’m combining that first leaked copy with 505 signatures, the final leaked copy with 702 signatures, the oft-repeated total headcount of 770, and this spreadsheet tracking OpenAI departures (albeit with many false positives—people self-reporting as OpenAI employees because they customized their GPTs—so I’m working to verify names that appear on the spreadsheet but not on the letter; I’m sure the spreadsheet has false negatives as well, alas).
So far, I’ve verified at least seven [update: seven, with a probable eighth] departures of eligible figures who hadn’t signed the letter with 702 names: Leopold Aschenbrenner, Jay Joshi (not fully verified by me), Andrej Karpathy, Daniel Kokotajlo, Jan Leike, Lucas Negritto, Katarina Slama, and William Saunders. If it’s true that the total headcount at the time was 770, then that’s 8 out of 68, or 11.8%.
Compare that to the attrition rate (as per the spreadsheet) for those who had signed the final leaked version but not the first: 10 departures out of 197, or 5.1%; and compare that to the attrition rate for those who signed promptly: 13 departures out of 505, or 2.6%.
Any causal inferences from this correlation are left as an exercise to the reader.
(A more important exercise, however: can anyone find a confirmation of the 770 number outside of unsourced media reports, or find a copy of the loyalty letter with more than 702 signatories, or ideally find a list of everyone at OpenAI at the time? I’ve tried a few different avenues without success.)
I’m not even angry, just disappointed.
The diagnosis is roughly correct (I would say “most suffering is caused by an internal response of fleeing from pain but not escaping it”), but IMO the standard proffered remedy (Buddhist-style detachment from wanting) goes too far and promises too much.
Re: the diagnosis, three illustrative ways the relationship between pain, awareness, and suffering has manifested for me:
Migraines: I get them every few weeks, and they’re pretty bad. After a friend showed me how to do a vipassana body scan, on a lark I tried moving my attention to the spot inside my skull where the migraine was most intense. To my relief, this helped the suffering greatly; the pain was caused by the migraine but the suffering was caused by trying futilely to not feel the pain, and staring directly at it was a good remedy.
Mental health: I’m chronically depressed and anxious. (I am not asking for advice about it, and there’s a unique causal element so your advice is less likely than usual to help.) One thing I figured out is that I can “wallow” in it: optimize for feeling it as intensely as possible. For example, if I’m feeling miserable I’ll intentionally lie in bed in the dark listening to my saddest music. This genuinely helps make things more bearable, and helps the worst moods pass faster, compared to the approach of trying to distract myself from the feelings or to force a contrary feeling.
Psychedelics: The worst hour of my life was spent on a bad acid trip, feeling nauseous and wanting to escape that feeling, and getting stuck in a “turning away from awareness” motion (reflected in visual hallucination by trying to mentally push my visual field up and to the left, only for it to reset to its actual location several times per second). That was a tremendous mistake.
However, my depression sometimes manifests as anhedonia, i.e. true detachment from desire, and that’s really not all it’s cracked up to be. I’m not suffering when I lie around all day with anhedonia, but I’m not getting any positive valence from it, and meanwhile I’m stagnating as a person. And I genuinely do not see how to wallow in anhedonia, to turn my awareness inward and find something to engage with. I’ve tried. It just seems like nobody’s home in that state.
A key, I suspect, is happiness set point. A person who takes up Buddhism or a similar practice, and starts to experience their preferences less strongly, ends up hovering more stably around their happiness set point without the highs and lows that come with excitement, anticipation, triumph, disappointment, etc. Lows hurt more than highs, so this mental motion is a net improvement. Most people have a pretty good happiness set point, so it feels like a good end result. (And in most cases people stop before ceasing to have preferences entirely; there’s probably even a golden mean where they’re more effective in the real world than either their natural state or a zen-wireheaded extreme.)
But I don’t see much proof that detachment from desire moves one’s happiness set point, so advertising it as a cure for unhappiness feels like the same sort of error as talking about how everyone—even poor people—should buy index funds. (Which is to say, it’s good advice on the margin for the majority of people, but for some people it’s irrelevant, and the correct first advice is more like “how to get a job, or a better job” or “how to budget” or “how to cheaply refinance your credit card debt”, etc.)
And also, I’m dubious that [intentionally reducing the intensity of your good moments] is actually helpful in the same way that [intentionally reducing the intensity of your bad moments] is? Sometimes you happen to know that a good thing is going to happen with very little risk of failure. In that case, it seems strictly better to want and desire and expect that good thing.
In short, I highly recommend turning towards experiences of pain rather than fleeing from them; but I think the Buddhist thesis is questionable.
Go all-in on lobbying the US and other governments to fully prohibit the training of frontier models beyond a certain level, in a way that OpenAI can’t route around (so probably block Altman’s foreign chip factory initiative, for instance).
The chess example is meant to make specific points about RL*F concealing a capability that remains (or is even amplified); I’m not trying to claim that the “put up a good fight but lose” criterion is analogous to current RL*F criteria. (Though it does rhyme qualitatively with “be helpful and harmless”.)
I agree that “helpful-only” RL*F would result in a model that scores higher on capabilities evals than the base model, possibly much higher. I’m frankly a bit worried about even training that model.
Thank you! I’d forgotten about that.
Certainly RLHF can get the model to stop talking about a capability, but usually this is extremely obvious because the model gives you an explicit refusal?
How certain are you that this is always true (rather than “we’ve usually noticed this even though we haven’t explicitly been checking for it in general”), and that it will continue to be so as models become stronger?
It seems to me like additionally running evals on base models is a highly reasonable precaution.
Oh wait, I misinterpreted you as using “much worse” to mean “much scarier”, when instead you mean “much less capable”.
I’d be glad if it were the case that RL*F doesn’t hide any meaningful capabilities existing in the base model, but I’m not sure it is the case, and I’d sure like someone to check! It sure seems like RL*F is likely in some cases to get the model to stop explicitly talking about a capability it has (unless it is jailbroken on that subject), rather than to remove the capability.
(Imagine RL*Fing a base model to stop explicitly talking about arithmetic; are we sure it would un-learn the rules?)
Yes, and I gained some easy mana from such markets; but the market that got the most attention by far was the intrinsically flawed conditional market.