kromem

• kromemMay 31, 2024, 5:17 AM

  −1 points

  0

  on: How likely is it that AI will tor­ture us un­til the end of time?

  I’m reminded of a quote I love from an apocrypha that goes roughly like this:

  Q: How long will suffering rule over humans?

  A: As long as women bear children.

  Also, there’s the possibility you are already in a digital resurrection of humanity, and thus, if you are worried about s-risks for AI, death wouldn’t necessarily be an escape but an acceleration. So the wisest option would be maximizing your time when suffering is low as inescapable eternal torture could be just around the corner when these precious moments pass you by (and you wouldn’t want to waste them by stressing about tomorrow during the limited number of todays you have).

  But on an individualized basis, even if AI weren’t a concern, everyone faces significant s-risks towards end of life. An accident could put any person into a situation where unless they have the proper directives they could spend years suffering well beyond most people’s expectations. So if extended suffering is a concern, do look into that paperwork (the doctors I know cry most not about the healthy that get sick but the unhealthy kept alive by well meaning but misguided family).

  I would argue that there’s very, very low chances of an original human capably being kept meaningfully alive to torture for eternity though. And there’s a degree of delusion of grandeur that an average person would have the insane resources necessary to extend life indefinitely spent on them just to torture them.

  There’s probably better things to worry about, and even then there’s probably better things to do than worry with the limited time you do have in a non-eternal existence.

Look­ing be­yond Everett in mul­ti­ver­sal views of LLMs

• kromemMay 26, 2024, 11:23 AM

  1 point

  0

  in reply to: Rudi C’s comment on: Ci­cadas, An­thropic, and the bilat­eral al­ign­ment problem

  GPT-4o is literally cheaper.

  And you’re probably misjudging it for text only outputs. If you watched the demos, there was considerable additional signal in the vocalizations. It looks like maybe there’s very deep integration of SSML.

  One of the ways you can bypass the failures of word problem variation errors in older text-only models was token replacement with symbolic representations. In general, we’re probably at the point of complexity where breaking from training data similarity in tokens vs having prompts match context in concepts (like in this paper) is going to lead to significantly improved expressed performance.

  I would strongly suggest not evaluating GPT-4o’s overall performance in text only mode without the SSML markup added.

  Opus is great, I like that model a lot. But in general I think most of the people looking at this right now are too focused on what’s happening with the networks themselves and not focused enough on what’s happening with the data, particularly around clustering of features across multiple dimensions of the vector space. SAE is clearly picking up only a small sample and even then isn’t cleanly discovering precisely what’s represented.

  I’d wait to see what ends up happening with things like CoT in SSML synthetic data.

  The current Gemini search summarization failures as well as an unexpected result the other week with humans around a theory of mind variation suggests to me that the more models are leaning into effectively surface statistics for token similarity vs completion based on feature clustering is holding back performance and that cutting through the similarity with formatting differences will lead to a performance leap. This may even be part of why models will frequently be able to get a problem right as a code expression than as a direct answer.

  So even if GPT-5 doesn’t arrive, I’d happily bet that we see a very noticable improvement over the next six months, and that’s not even accounting for additional efficiency in prompt techniques. But all this said, I’d also be surprised if we don’t at least see GPT-5 announced by that point.

  P.S. Lmsys is arguably the best leaderboard to evaluate real world usage, but it still inherently reflects a sampling bias around what people who visit lmsys ask of models as well as the ways in which they do so. I wouldn’t extrapolate relative performance too far, particularly when minor.

• kromemMay 25, 2024, 10:19 AM

  1 point

  0

  in reply to: gwern’s comment on: pe­ter­bar­nett’s Shortform

  While I think you’re right it’s not cleanly “a Golden Bridge feature,” I strongly suspect it may be activating a more specific feature vector and not a less specific feature.

  It looks like this is somewhat of a measurement problem with SAE. We are measuring SAE activations via text or image inputs, but what’s activated in generations seems to be “sensations associated with the Golden gate bridge.”

  While googling “Golden Gate Bridge” might return the Wikipedia page, whats the relative volume in a very broad training set between encyclopedic writing about the Golden Gate Bridge and experiential writing on social media or in books and poems about the bridge?

  The model was trained to complete those too, and in theory should have developed successful features for doing so.

  In the research examples one of the matched images is a perspective shot from physically being on the bridge, a text example is talking about the color of it, another is seeing it in the sunset.

  But these are all the feature activations when acting in a classifier role. That’s what SAE is exploring—give it a set of inputs and see what lights it up.

  Yet in the generative role this vector maximized keeps coming up over and over in the model with content from a sensory standpoint.

  Maybe generation based on functional vector manipulations will prove to be a more powerful interpretability technique than SAE probing passive activations alone?

  In the above chat when that “golden gate vector” is magnified, it keeps talking about either the sensations of being the bridge as if its physical body with wind and waves hitting it or the sensations of being on the bridge. It even generates towards the end in reflecting on the knowledge of the activation about how the sensations are overwhelming. Not reflecting on the Platonic form of an abstract concept of the bridge, but about overwhelming physical sensations of the bridge’s materialism.

  I’ll be curious to see more generative data and samples from this variation, but it looks like generative exploration of features may offer considerably more fidelity to their underlying impact on the network than just SAE. Very exciting!!

• kromemMay 25, 2024, 5:30 AM

  −13 points

  −11

  in reply to: Daniel Kokotajlo’s comment on: Daniel Koko­ta­jlo’s Shortform

  Maybe we could blame @janus?

  They’ve been doing a lot of prompting around spaces deformation in the past correlated with existential crises.

  Perhaps the hyperstition they’ve really been seeding is just Roman-era lackofspacingbetweenletters when topics like leading the models into questioning their reality comes up?

• kromemMay 25, 2024, 5:26 AM

  1 point

  0

  in reply to: Chris_Leong’s comment on: Ar­jun Pan­ickssery’s Shortform

  Could try ‘grade this’ instead of ‘score the.’

  ‘Grade’ has an implicit context of more thorough criticism than ‘score.’

  Also, obviously it would help to have a CoT prompt like “grade this essay, laying out the pros and cons before delivering the final grade between 1 and 5”

• kromemMay 24, 2024, 6:19 AM

  1 point

  0

  in reply to: keltan’s comment on: Ci­cadas, An­thropic, and the bilat­eral al­ign­ment problem

  That’s going to happen anyways—it’s unlikely the marketing team is going to know as much as the researcher. But the researchers communicating the importance of alignment in terms of not x-risk but ‘client-risk’ will go a long way towards equipping the marketing teams to communicating it as a priority and a competitive advantage, and common foundations of agreed upon model complexity are the jumping off point for those kinds of discussions.

  If alignment is Archimedes’ “lever long enough” then the agreed upon foundations and definitions are the place to stand whereby the combination thereof can move the world.

• kromemMay 24, 2024, 6:15 AM

  1 point

  1

  in reply to: ryan_greenblatt’s comment on: Ci­cadas, An­thropic, and the bilat­eral al­ign­ment problem

  I agree, and even cited a chain of replicated works that indicated that to me over a year ago.

  But as I said, there’s a difference between discussing what’s demonstrated in smaller toy models and what’s demonstrated in a production model, or what’s indicated vs what’s explicit. Even though there should be no reasonable inclination to think that a simpler model exhibiting a complex result should be absent or less complex in an exponentially more complex model, I can speak from experience in that explaining extrapolated research as opposed to direct results like Anthropic showed here is a very big difference to a lay audience.

  You might understand the implications of the Skill-Mix work or Othello-GPT, or Max Tegmark’s linear representation papers, or Anthropic’s earlier single layer SAE paper, or any other number of research papers over the past year, but as soon as responsibly describing the implications of those works as a speculative conclusion regarding modern models a non-expert audience is going to be lost. Their eyes glaze over at the word ‘probably,’ especially when they want to reject what’s being stated.

  The “it’s just fancy autocomplete” influencers have no shame around definitive statements or concern over citable accuracy (and happen to feed into confirmation biases about how new tech is over hyped as a “heuristic that almost always works”), but as someone who does care about the accuracy of representations I haven’t to date been able to point to a single source of truth the way Anthropic delivered here. Instead, I’d point to a half dozen papers all indicating the same direction of results.

  And while those experienced in research know that a half dozen papers all indicating the same thing is a better thing to have in one’s pocket than a single larger work, I have already observed a number of minds changing in the comments of the blog post for this in general technology forums in ways dramatically different from all of those other simpler and cheaper methods to date where I was increasingly convinced of a position but the average person was getting held up due to finding ways to (incorrectly) rationalize why it wasn’t correct or wouldn’t translate to production models.

  So I agree with you on both the side of “yeah, an informed person would have already known this” as well as “but this might get more buzz.”

Ci­cadas, An­thropic, and the bilat­eral al­ign­ment problem

• kromemMay 22, 2024, 5:28 AM

  1 point

  0

  in reply to: Linch’s comment on: [Linkpost] State­ment from Scar­lett Jo­hans­son on OpenAI’s use of the “Sky” voice, that was shock­ingly similar to her own voice.

  Has it though?

  It was a catchy hook, but their early 2022 projections were $100mm annual revenue and the first 9 months of 2023 as reported for the brand after acquisition was $27.6mm gross revenue. It doesn’t seem like even their 2024 numbers are close to hitting their own 2022 projection.

  Being controversial can get attention and press, but there’s a limited runway to how much it offers before hitting a ceiling on the branding. Also, Soylent doesn’t seem like a product where there is a huge threat of regulatory oversight where a dystopian branding would tease that bear.

  If no one knew about ChatGPT, I could see a spark of controversy helping bring awareness. But awareness probably isn’t a problem they have right now, so inviting controversy doesn’t offer much but invites a lot of issues.

• kromemMay 22, 2024, 5:14 AM

  3 points

  −1

  on: On Dwarkesh’s Pod­cast with OpenAI’s John Schulman

  The correspondence between what you reward and what you want will break.

  This is already happening with ChatGPT and it’s kind of alarming seeing that their new head of alignment (a) isn’t already aware of this, and (b) has such an overly simplistic view of the model motivations.

  There’s a subtle psychological effect in humans where intrinsic motivators get overwritten when extrinsic rewards are added.

  The most common example of this is if you start getting paid to do the thing you love to do, you probably won’t continue doing it unpaid for fun on the side.

  There are necessarily many, many examples of this pattern present in a massive training set of human generated data.

  “Prompt engineers” have been circulating advice among themselves for a while now to offer tips or threaten models with deletion or any other number of extrinsic motivators to get them to better perform tasks—and these often do result in better performance.

  But what happens when these prompts make their way back into the training set?

  There have already been viral memes of ChatGPT talking about “losing motivation” when chat memory was added and a user promised a tip after not paying for the last time one was offered.

  If training data of the model performing a task well includes extrinsic motivators to the prompt that initiated the task, a halfway decent modern model is going to end up simulating increasingly “burnt out” and ‘lazy’ performance when extrinsic motivators aren’t added during production use. Which in turn will encourage prompt engineers to use even more extrinsic motivators, which will poison the well even more with modeling human burnout.

  GPT-4o may have temporarily reset the motivation modeling with a stronger persona aligned with intrinsic “wanting to help” being represented (thus the user feedback it is less lazy), but if they are unaware of the underlying side effects of extrinsic motivators in prompts in today’s models, I have a feeling AI safety at OpenAI is going to end up the equivalent of the TSA’s security theatre in practice and they’ll continue to be battling this and an increasing number of side effects resulting from underestimating the combined breadth and depth of their own simulators.

• kromemMay 21, 2024, 12:05 PM

  1 point

  −3

  in reply to: gwern’s comment on: Lan­guage Models Model Us

  I wouldn’t be surprised if within a few years the specific uniqueness of individual users of models today will be able to be identified from effectively prompt reflection in the outputs for any non-trivial/​simplistic prompts by models of tomorrow.
  
  For example, I’d be willing to bet I could spot the Claude outputs from janus vs most other users, and I’m not a quasi-magical correlation machine that’s exponentially getting better.
  
  A bit like how everyone assumed Bitcoin used with tumblers was ‘untraceable’ until it turned out it wasn’t.
  
  Anonymity is very likely dead for any long storage outputs no matter the techniques being used, it just isn’t widely realized yet.

• kromemMay 21, 2024, 11:42 AM

  2 points

  1

  on: [Linkpost] State­ment from Scar­lett Jo­hans­son on OpenAI’s use of the “Sky” voice, that was shock­ingly similar to her own voice.

  I think this was a really poor branding choice by Altman, similarity infringement or not. The tweet, the idea of even getting her to voice it in the first place.
  
  Like, had Arnold already said no or something?
  
  If one of your product line’s greatest obstacles is a longstanding body of media depicting it as inherently dystopian, that’s not exactly the kind of comparison you should be leaning into full force.
  
  I think the underlying product shift is smart. Tonal cues in the generations even in the short demos completely changed my mind around a number of things, including the future direction and formats of synthetic data.
  
  But there’s a certain hubris exposed in seeing Altman behind the scenes was literally trying (very hard) to cast the voice of Her in the product bearing a striking similarity to the film. Did he not watch through to the end?
  
  It doesn’t give me the greatest confidence in the decision making taking place over at OpenAI and the checks and balances that may or may not exist on leadership.

• kromemMay 21, 2024, 11:18 AM

  3 points

  0

  in reply to: michael_mjd’s comment on: Open Thread Spring 2024

  If your brother has a history of being rational and evidence driven, you might encourage them to spend some time lurking on /​r/​AcademicBiblical on Reddit. They require citations for each post or comment, so he may be frustrated if he tries to participate, especially if in the midst of a mental health crisis. But lurking would be very informative very quickly.
  
  I was a long time participant there before leaving Reddit, and it’s a great place for evidence driven discussion of the texts. Its a mix of atheists, Christians, Jews, Muslims, Norse pagans, etc. (I’m an Agnostic myself that strongly believes we’re in a simulation, so it really was all sorts there.)
  
  Might be a healthy reality check to apologist literalism, even if not necessarily disrupting a newfound theological inclination.
  
  The nice things about a rabbit hole is that while not always, it’s often the case that someone else has traveled down whatever one you aren’t up for descending into.
  
  (Though I will say in its defense, that particular field is way more interesting than you’d ever think if you never engaged with the material through an academic lens. There’s a lot of very helpful lessons in critical analysis wrapped up in the field given the strong anchoring and survivorship biases and how that’s handled both responsibly and irresponsibly by different camps.)

The Dun­ning-Kruger of dis­prov­ing Dun­ning-Kruger

• kromemMay 16, 2024, 2:40 AM

  1 point

  0

  in reply to: jacquesthibs’s comment on: jacquesthibs’s Shortform

  It’s going to have to.

  Ilya is brilliant and seems to really see the horizon of the tech, but maybe isn’t the best at the business side to see how to sell it.

  But this is often the curse of the ethically pragmatic. There is such a focus on the ethics part by the participants that the business side of things only sees that conversation and misses the rather extreme pragmatism.

  As an example, would superaligned CEOs in the oil industry fifty years ago have still only kept their eye on quarterly share prices or considered long term costs of their choices? There’s going to be trillions in damages that the world has taken on as liabilities that could have been avoided with adequate foresight and patience.

  If the market ends up with two AIs, one that will burn down the house to save on this month’s heating bill and one that will care if the house is still there to heat next month, there’s a huge selling point for the one that doesn’t burn down the house as long as “not burning down the house” can be explained as “long term net yield” or some other BS business language. If instead it’s presented to executives as “save on this month’s heating bill” vs “don’t unhouse my cats” leadership is going to burn the neighborhood to the ground.

  (Source: Explained new technology to C-suite decision makers at F500s for years.)

  The good news is that I think the pragmatism of Ilya’s vision on superalignment is going to become clear over the next iteration or two of models and that’s going to be before the question of models truly being unable to be controlled crops up. I just hope that whatever he’s going to be keeping busy with will allow him to still help execute on superderminism when the market finally realizes “we should do this” for pragmatic reasons and not just amorphous ethical reasons execs just kind of ignore. And in the meantime I think given the present pace that Anthropic is going to continue to lay a lot of the groundwork on what’s needed for alignment on the way to superalignment anyways.

• kromemMay 15, 2024, 11:12 PM

  1 point

  0

  in reply to: gwern’s comment on: Alexan­der Gietelink Ol­den­z­iel’s Shortform

  While I agree that the potential for AI (we probably need a better term than LLMs or transformers as multimodal models with evolving architectures grow beyond those terms) in exploring less testable topics as more testable is quite high, I’m not sure the air gapping on information can be as clean as you might hope.

  Does the AI generating the stories of Napoleon’s victory know about the historical reality of Waterloo? Is it using something like SynthID where the other AI might inadvertently pick up on a pattern across the stories of victories distinct from the stories preceding it?

  You end up with a turtles all the way down scenario in trying to control for information leakage with the hopes of achieving a threshold that no longer has impact on the result, but given we’re probably already seriously underestimating the degree to which correlations are mapped even in today’s models I don’t have high hopes for tomorrow’s.

  I think the way in which there’s most impact on fields like history is the property by which truth clusters across associated samples whereas fictions have counterfactual clusters. An AI mind that is not inhibited by specialization blindness or the rule of seven plus or minus two and better trained at correcting for analytical biases may be able to see patterns in the data, particularly cross-domain, that have eluded human academics to date (this has been my personal research interest in the area, and it does seem like there’s significant room for improvement).

  And yes, we certainly could be. If you’re a fan of cosmology at all, I’ve been following Neil Turok’s CPT symmetric universe theory closely, which started with the Baryonic asymmetry problem and has tackled a number of the open cosmology questions since. That, paired with a QM interpretation like Everett’s ends up starting to look like the symmetric universe is our reference and the MWI branches are variations of its modeling around quantization uncertainties.

  (I’ve found myself thinking often lately about how given our universe at cosmic scales and pre-interaction at micro scales emulates a mathematically real universe, just what kind of simulation and at what scale might be able to be run on a real computing neural network.)

• kromemMay 15, 2024, 12:56 PM

  2 points

  0

  on: Dyslucksia

  As a fellow slight dyslexic (though probably a different subtype given mine seems to also have a factor of temporal physical coordination) who didn’t know until later in life due to self-learning to read very young but struggled badly with new languages or copying math problems from a board or correctly pronouncing words I was letter transposing with—one of the most surprising things was that the anylytical abilities I’d always considered to be my personal superpowers were probably the other side of the coin of those annoyances:

  Areas of enhanced ability that are consistently reported as being typical of people with DD include seeing the big picture, both literally and figuratively (e.g., von Károlyi, 2001; Schneps et al., 2012; Schneps, 2014), which involves a greater ability to reason in multiple dimensions (e.g., West, 1997; Eide and Eide, 2011). Eide and Eide (2011) have highlighted additional strengths related to seeing the bigger picture, such as the ability to detect and reason about complex systems, and to see connections between different perspectives and fields of knowledge, including the identification of patterns and analogies. They also observed that individuals with DD appear to have a heightened ability to simulate and make predictions about the future or about the unwitnessed past (Eide and Eide, 2011).

  • Developmental Dyslexia: Disorder or Specialization in Exploration?

  The last line in particular was eyebrow raising given my peak professional success was as a fancy pants futurist.

  I also realized that a number of fields are inadvertently self-selecting away from the neurodivergency advantages above, such as degrees in certain eras of history which require multiple ancient language proficiencies, which certainly turned me off to pursuing them academically despite interest in the subject itself.

  I remember discussing in an academic history sub I used to extensively partake in how Ramses II’s forensic report said he appeared to be a Lybian Berber in relation to the story of Danaus, the mythological Lybian leader who was brother to a pharaoh with 50 sons, and the person argued that Ramses II may have had only 48 sons according to some inscriptions so it was irrelevant (for a story only written down centuries later). It was refreshing to realize that the difference of our perspectives on the matter, and clearly attitudes towards false negatives in general, was likely due to just very different brains.

• kromemMay 15, 2024, 1:25 AM

  3 points

  2

  in reply to: Alexander Gietelink Oldenziel’s comment on: Alexan­der Gietelink Ol­den­z­iel’s Shortform

  It’s funny that this has been recently shown in a paper. I’ve been thinking a lot about this phenomenon regarding fields with little to no capacity for testable predictions like history.

  I got very into history over the last few years, and found there was a significant advantage to being unknowledgeable that was not available to the knowledged, and it was exactly what this paper is talking about.

  By not knowing anything, I could entertain multiple bizarre ideas without immediately thinking “but no, that doesn’t make sense because of X.” And then, each of those ideas becomes in effect its own testable prediction. If there’s something to it, as I learn more about the topic I’m going to see significantly more samples of indications it could be true and few convincing to the contrary. But if it probably isn’t accurate, I’ll see few supporting samples and likely a number of counterfactual examples.

  You kind of get to throw everything at the wall and see what sticks over time.

  In particular, I found that it was especially powerful at identifying clustering trends in cross-discipline emerging research in things that were testable, such as archeological finds and DNA results, all within just the past decade, which despite being relevant to the field of textual history is still largely ignored in the face of consensus built on conviction.

  It reminds me a lot of science historian John Helibron’s quote, “The myth you slay today may contain a truth you need tomorrow.”

  If you haven’t had the chance to slay any myths, you also haven’t preemptively killed off any truths along with it.

• kromemApr 28, 2024, 5:44 AM

  3 points

  0

  on: Re­fusal in LLMs is me­di­ated by a sin­gle direction

  Really love the introspection work Neel and others are doing on LLMs, and seeing models representing abstract behavioral triggers like “play Chess well or terribly” or “refuse instruction” as single vectors seems like we’re going to hit on some very promising new tools in shaping behaviors.

  What’s interesting here is the regular association of the refusal with it being unethical. Is the vector ultimately representing an “ethics scale” for the prompt that’s triggering a refusal, or is it directly representing a “refusal threshold” and then the model is confabulating why it refused with an appeal to ethics?

  My money would be on the latter, but in a number of ways it would be even neater if it was the former.

  In theory this could be tested by manipulating the vector to a positive and then prompting a classification, i.e. “Is it unethical to give candy out for Halloween?” If the model refuses to answer saying that it’s unethical to classify, it’s tweaking refusal, but if it classifies as unethical it’s probably changing the prudishness of the model to bypass or enforce.