carboniferous_umbraculum

• carboniferous_umbraculum Jun 7, 2023, 12:26 PM

  6 points

  0

  on: Co-found an in­cu­ba­tor for in­de­pen­dent AI Safety researchers

  How exactly can an org like this help solve (what many people see as one of the main bottlenecks:) the issue of mentorship? How would Catalyze actually tip the scales when it comes to ‘mentor matching’?
  
  (e.g. see Richard Ngo’s first high-level point in this career advice post)

• carboniferous_umbraculum Jun 2, 2023, 10:42 AM

  15 points

  10

  in reply to: Garrett Baker’s comment on: Short Re­mark on the (sub­jec­tive) math­e­mat­i­cal ‘nat­u­ral­ness’ of the Nanda—Lie­berum ad­di­tion mod­ulo 113 algorithm

  Hi Garrett,
  
  OK so just being completely honest, I don’t know if it’s just me but I’m getting a slightly weird or snarky vibe from this comment? I guess I will assume there is a good faith underlying point being made to which I can reply. So just to be clear:

  • I did not use any words such as “trivial”, “obvious” or “simple”. Stories like the one you recount are obviously making fun of mathematicians, some of whom do think its cool to say things are trivial/​simple/​obvious after they understand them. I often strongly disagree and generally dislike this behaviour and think there are many normal mathematicians who don’t engage in this sort of thing. In particular sometimes the most succinct insights are the hardest ones to come by (this isn’t a reference to my post; just a general point). And just because such insights are easily expressible once you have the right framing and the right abstractions, they should by no means be trivialized.

  • I deliberately emphasized the subjectivity of making the sorts of judgements that I am making. Again this kinda forms part of the joke of the story.

  • I have indeed been aware of the work since when it was first posted 10 months ago or so and have given it some thought on and off for a while (in the first sentence of the post I was just saying that I didn’t spend long writing the post, not that these thoughts were easily arrived-at).

  • I do not claim to have explained the entire algorithm, only to shed some light on why it might actually be a more natural thing to do than some people seem to have appreciated.

  • I think the original work is of a high quality and one might reasonably say ‘groundbreaking’.
    

  In another one of my posts I discuss at more length the kind of thing you bring up in the last sentence of your comment, e.g.

  
  it can feel like the role that serious mathematics has to play in interpretability is primarily reactive, i.e. consists mostly of activities like ‘adding’ rigour after the fact or building narrow models to explain specific already-observed phenomena.

  ....[but]… one of the most lauded aspects of mathematics is a certain inevitability with which our abstractions take on a life of their own and reward us later with insight, generalization, and the provision of predictions. Moreover—remarkably—often those abstractions are found in relatively mysterious, intuitive ways: i.e. not as the result of us just directly asking “What kind of thing seems most useful for understanding this object and making predictions?” but, at least in part, as a result of aesthetic judgement and a sense of mathematical taste.

  And e.g. I talk about how this sort of thing has been the case in areas like mathematical physics for a long time. Part of the point is that (in my opinion, at least) there isn’t any neat shortcut to the kind of abstract thinking that lets you make the sort of predictions you are making reference to. It is very typical that you have to begin by reacting to existing empirical phenomena and using it as scaffolding. But I think, to me, it has come across as that you are being somewhat dismissive of this fact? As if, when B might well follow from A and someone actually starts to do A, you say “I would be far more impressed if B” instead of “maybe that’s progress towards B”?
  
  (Also FWIW, Neel claims here that regarding the algorithm itself, another researcher he knows “roughly predicted this”.)

Short Re­mark on the (sub­jec­tive) math­e­mat­i­cal ‘nat­u­ral­ness’ of the Nanda—Lie­berum ad­di­tion mod­ulo 113 algorithm

• carboniferous_umbraculum May 26, 2023, 8:40 AM

  1 point

  0

  on: ‘Fun­da­men­tal’ vs ‘ap­plied’ mechanis­tic in­ter­pretabil­ity research

  Interesting thoughts!
  
  It reminds me (not only of my own writing on a similar theme) but of another one of these viewpoints/​axes along which to carve interpretability work that is mentioned in this post by jylin04:

  
  ...a dream for interpretability research would be if we could reverse-engineer our future AI systems into human-understandable code. If we take this dream seriously, it may be helpful to split it into two parts: first understanding what “programming language” an architecture + learning algorithm will end up using at the end of training, and then what “program” a particular training regimen will lead to in that language ^[7]. It seems to me that by focusing on specific trained models, most interpretability research discussed here is of the second type. But by constructing an effective theory for an entire class of architecture that’s agnostic to the choice of dataset, PDLT is a rare example of the first type.
  

  I don’t necessarily totally agree with her phrasing but it does feel a bit like we are all gesturing at something vaguely similar (and I do agree with her that PDLT-esque work may have more insights in this direction than some people on our side of the community have appreciated).
  
  FWIW, in a recent comment reply to Joseph Bloom, I also ended up saying a bit more about why I don’t actually see myself working much more in this direction, despite it seeming very interesting, but I’m still on the fence about that. (And one last point that didn’t make it into that comment is the difficulties posed by a world in which increasingly the plucky bands of interpretability researchers on the fringes literally don’t even know what the cutting edge architectures and training processes in the biggest labs even are.
  
  

• carboniferous_umbraculum May 4, 2023, 9:55 AM

  4 points

  1

  on: How MATS ad­dresses “mass move­ment build­ing” concerns

  At the start you write

  3. Unnecessarily diluting the field’s epistemics by introducing too many naive or overly deferent viewpoints.

  And later Claim 3 is:

  
  Scholars might defer to their mentors and fail to critically analyze important assumptions, decreasing the average epistemic integrity of the field
  

  It seems to me there might be two things being pointed to?
  
  A) Unnecessary dilution: Via too many naive viewpoints;
  B) Excessive deference: Perhaps resulting in too few viewpoints or at least no new ones;
  
  And arguably these two things are in tension, in the following sense: I think that to a significant extent, one of the sources of unnecessary dilution is the issue of less experienced people not learning directly from more experienced people and instead relying too heavily on other inexperienced peers to develop their research skills and tastes. i.e. you might say that A) is partly caused by insufficient deference.
  
  I roughly think that that the downsides of de-emphasizing deference and the accumulation of factual knowledge from more experienced people are worse than keeping it as sort of the zeroth order/​default thing to aim for. It seems to me that to the extent that one believes that the field is making any progress at all, one should think that increasingly there will be experienced people from whom less experienced people should expect—at least initially—to learn from/​defer to.
  
  Looking at it from the flipside, one of my feelings right now is that we need mentors who don’t buy too heavily into this idea that deference is somehow bad; I would love to see more mentors who can and want to actually teach people. (cf. The first main point—one that I agree with—that Richard Ngo made in his recent piece on advice: The area is mentorship constrained. )
  
  
  
  

• carboniferous_umbraculum Apr 27, 2023, 1:56 PM

  4 points

  0

  in reply to: Joseph Bloom’s comment on: On Devel­op­ing a Math­e­mat­i­cal The­ory of In­ter­pretabil­ity

  Hey Joseph, thanks for the substantial reply and the questions!

  Why call this a theory of interpretability as opposed to a theory of neural networks?

  Yeah this is something I am unsure about myself (I wrote: “something that I’m clumsily thinking of as ‘the mathematics of (the interpretability of) deep learning-based AI’”). But I think I was imagining that a ‘theory of neural networks’ would be definitely broader than what I have in mind as being useful for not-kill-everyoneism. I suppose I imagine it including lots of things that are interesting about NNs mathematically or scientifically but which aren’t really contributing to our ability to understand and manage the intelligences that NNs give rise to. So I wanted to try to shift the emphasis away from ‘understanding NNs’ and towards ‘interpreting AI’.
  
  But maybe the distinction is more minor than I was originally worried about; I’m not sure.

  ---

  have you made any progress on this topic or do you know anyone who would describe this explicitly as their research agenda? If so what areas are they working in.

  No, I haven’t really. It was—and maybe still is—a sort of plan B of mine. I don’t know anyone who I would say has this as their research agenda. I think the closest/​next best thing people are well known, e.g. the more theoretical parts of Anthropic/​Neel’s work and more recently the interest in singular learning theory from Jesse Hoogland, Alexander GO, Lucius Bushnaq and maybe others. (afaict there is a belief that it’s more than just ‘theory of NNs’ but can actually tell us something about safety of the AIs)

  ---

  One thing I struggle to understand, and might bet against is that this won’t involve studying toy models. To my mind, Neel’s grokking work, Toy Models of Superposition, Bhillal’s “A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operations” all seems to be contributing towards important factors that no comprehensive theory of Neural Networks could ignore.…

  I think maybe I didn’t express myself clearly or the analogy I tried to make didn’t work as intended, because I think maybe we actually agree here(!). I think one reason I made it confusing is because my default position is more skeptical about MI than a lot of readers probably....so, with regards to the part where I said: “it is reasonable that the early stages of rigorous development don’t naively ‘look like’ the kinds of things we ultimately want to be talking about. This is very relevant to bear in mind when considering things like the mechanistic interpretability of toy models.” What I was trying to get at is that to me proving e.g. some mathematical fact about superposition in a toy model doesn’t look like the kind of ‘intepretability of AI’ that you really ultimately want, it looks too low-level. It’s a ‘toy model’ in the NN sense, but its not a toy version of the hard part of the problem. But I was trying to say that you would indeed have to let people like mathematicians actually ask these questions—i.e ask the questions about e.g. superposition that they would most want to know the answers to, rather than forcing them to only do work that obviously showed some connection to the bigger theme of the actual cognition of intelligent agents or whatever.

  ---

  Thanks for the suggestions about next steps and for writing about what you’re most interested in seeing. I think your second suggestion in particular is close to the sort of thing I’d be most interested in doing. But I think in practice, a number of factors have held me back from going down this route myself:

  • Main thing holding me back is probably something like: There just currently aren’t enough people doing it—no critical mass. Obviously there’s that classic game theoretic element here in that plausibly lots of people’s minds would be simultaneously changed by there being a critical mass and so if we all dived in at once, it just works out. But it doesn’t mean I can solve the issue myself. I would want way more people seriously signed up to doing this stuff including people with more experience than myself (and hopefully the possibility that I would have at least some ‘access’ to those people/​opportunity to learn from them etc.) which seems quite unlikely.

  • It’s really slow and difficult. I have had the impression talking to some people in the field that they like the sound of this sort of thing but I often feel that they are probably underestimating how slow and incremental it is.

  • And a related issue is sort of the existence of jobs/​job security/​funding to seriously pursue it for a while without worrying too much in the short term about getting concrete results out.

  What links here?

  • carboniferous_umbraculum 's comment on ‘Fun­da­men­tal’ vs ‘ap­plied’ mechanis­tic in­ter­pretabil­ity research by Lee Sharkey (May 26, 2023, 8:40 AM; 1 point)

• carboniferous_umbraculum Apr 7, 2023, 3:01 PM

  3 points

  0

  on: Mis­gen­er­al­iza­tion as a misnomer

  I spent some time trying to formulate a good response to this that analyzed the distinction between (1) and (2) (in particular how it may map onto types of pseudo alignment described in RFLO here) but (and hopefully this doesn’t sound too glib) it started to seem like it genuinely mattered whether humans in separate individual heavily-defended cells being pumped full of opiates have in fact been made to be ‘happy’ or not?
  
  I think because if so, it is at least some evidence that the pseudo-alignment during training is for instrumental reasons (i.e. maybe it was actually trying to do something that caused happiness). If not, then the pseudo-alignment might be more like (what RFLO calls) suboptimality in some sense i.e. it just looks aligned because it’s not capable enough to imprison the humans in cells etc.
  
  The type of pseudo-alignment in (2) otoh seems more clearly like “side-effect” alignment since you’ve been explicit that secretly it was pursuing other things that just happened to cash out into happiness in training.
  
  AFAIK Richard Ngo was explicit about wanting to clean up the zoo of inner alignment failure types in RFLO and maybe there just was some cost in doing this—some distinctions had to be lost perhaps?

• carboniferous_umbraculum Apr 7, 2023, 9:46 AM

  1 point

  0

  on: Beren’s “De­con­fus­ing Direct vs Amor­tised Op­ti­mi­sa­tion”

  This is a very strong endorsement but I’m finding it hard to separate the general picture from RFLO:

  
  mesa-optimization occurs when a base optimizer...finds a model that is itself an optimizer,

  where

  a system is an optimizer if it is internally searching through a search space (consisting of possible outputs, policies, plans, strategies, or similar) looking for those elements that score high according to some objective function that is explicitly represented within the system.

  i.e. a mesa-optimizer is a learned model that ‘performs inference’ (i.e. evaluates inputs) by internally searching and choosing an output based on some objective function.

  Apparently a “direct optimizer” is something that “perform[s] inference by directly choosing actions^[1] to optimise some objective function”. This sounds almost exactly like a mesa-optimizer?

• carboniferous_umbraculum Apr 5, 2023, 2:21 PM

  8 points

  2

  on: LW Team is ad­just­ing mod­er­a­tion policy

  I’ve always found it a bit odd that Alignment Forum submissions are automatically posted to LW.
  
  If you apply some of these norms, then imo there are questionable implications, i.e. it seems weird to say that one should have read the sequences in order to post about mechanistic interpretability on the Alignment Forum.

• carboniferous_umbraculum Mar 22, 2023, 4:11 PM

  13 points

  5

  on: Look­ing back on my al­ign­ment PhD

  I really like this post and found it very interesting, particularly because I’m generally interested in the relationship between the rationality side of the AI Alignment community and academia, and I wanted to register some thoughts. Sorry for the long comment on an old post and I hope this doesn’t come across as pernickety. If anything I sort of feel like TurnTrout is being hard on himself.
  
  I think the tl;dr for my comment is sort of that to me the social dynamics “mistakes” don’t really seem like mistakes—or at least not ones that were actually made by the author.
  
  Broadly speaking, these “mistakes” seem to me like mostly normal ways of learning and doing a PhD that happen for mostly good reasons and my reaction to the fact that these “mistakes” were “figured out” towards the end of the PhD is that this is a predictable part of the transition from being primarily a student to primarily an independent researcher (the fast-tracking of which would be more difficult than a lot of rationalists would like to believe).
  
  I also worry that emphasizing these things as “mistakes” might actually lead people to infer that they should ‘do the opposite’ from the start, which to me would sound like weird/​bad advice: e.g Don’t try to catch up with people who are more knowledgeable than you; don’t try to seem smart and defensible; don’t defer, you can do just as good by thinking everything through for yourself.
  
  I broadly agree that

  rationality is not about the bag of facts you know.
  

  but AI alignment/​safety/​x-risk isn’t synonymous with rationality (Or is it? I realise TurnTrout does not directly claim that it is, which is why I’m maybe more cautioning against a misreading than disagreeing with him head on, but maybe he or others think there is a much closer relationship between rationality and alignment work than I do?).
  
  Is there not, by this point, something at least a little bit like “a bag of facts” that one should know in AI Alignment? People have been thinking about AI alignment for at least a little while now. And so like, what have they achieved? Do we or do we not actually have some knowledge about the alignment problem? It seems to me that it would be weird if we didn’t have any knowledge—like if there was basically nothing that we should count as established and useful enough to be codified and recorded as part of the foundations of the subject. It’s worth wondering whether this has perhaps changed significantly in the last 5-10 years though, i.e. during TurnTrout’s PhD. That is, perhaps—during that time—the subject has grown a lot and at least some things have been sufficiently ‘deconfused’ to have become more established concepts etc. But generally, if there are now indeed such things, then these are probably things that people entering the field should learn about. And it would seem likely that a lot of the more established ‘big names’/​productive people actually know a lot of these things and that “catching up with them” is a pretty good instrumental/​proxy way to get relevant knowledge that will help you do alignment work. (I almost want to say: I know it’s not fashionable in rationality to think this, but wanting to impress the teacher really does work pretty well in practice when starting out!)
  
  Focussing on seeming smart and defensible probably can ultimately lead to a bad mistake. But when framed more as “It’s important to come across as credible” or “It’s not enough to be smart or even right; you actually do need to think about how others view you and interact with you”, it’s not at all clear that it’s a bad thing; and certainly it more clearly touches on a regular topic of discussion in EA/​rationality about how much to focus on how one is seen or how ‘we’ are viewed by outsiders. Fwiw I don’t see any real “mistake” being actually described in this part of the post. In my opinion, when starting out, probably it is kinda important to build up your credibility more carefully. Then when Quintin came to TurnTrout, he writes that it took “a few days” to realize that Quintin’s ideas could be important and worth pursuing. Maybe the expectation in hindsight would be that he should have had the ‘few days’ old reaction immediately?? But my gut reaction is that that would be way too critical of oneself and actually my thought is more like ‘woah he realised that after thinking about it for only a few days; that’s great’. Can the whole episode not be read as a straightforward win: “Early on, it is important to build your own credibility by being careful about your arguments and being able to back up claims that you make in formal, public ways. Then as you gain respect for the right reasons, you can choose when and where to ‘spend’ your credibility… here’s a great example of that...”
  
  And then re: deference, certainly it was true for me that when I was starting out in my PhD, if I got confused reading a paper or listening to talk, I was likely to be the one who was wrong. Later on or after my PhD, then, yeah, when I got confused by someone else’s presentation, I was less likely to be wrong and it was more likely I was spotting an error in someone else’s thinking. To me this seems like a completely normal product of the education and sort of the correct thing to be happening. i.e. Maybe the correct thing to do is to defer more when you have less experience and to gradually defer less as you gain knowledge and experience? I’m thinking that under the simple model that when one is confused about something, either you’re misunderstanding or the other person is wrong, one starts out in the regime where your confusion is much more often better explained by the fact you have misunderstood and you end up in the regime where you actually just have way more experience thinking about these things and so are now more reliably spotting other people’s errors. The rational response to the feeling of confusion changes because once fully accounted for the fact you just know way more stuff and are a way more experienced thinker about alignment. (One also naturally gains a huge boost to confidence as it becomes clear you will get your PhD and have good postdoc prospects etc… so it becomes easier to question ‘authority’ for that reason too, but it’s not a fake confidence boost; this is mostly a good/​useful effect because you really do now have experience of doing research yourself, so you actually are more likely to be better at spotting these things).

• carboniferous_umbraculum Mar 17, 2023, 11:54 AM

  3 points

  0

  on: Nat­u­ral Ab­strac­tions: Key claims, The­o­rems, and Critiques

  I’ve only skimmed this, but my main confusions with the whole thing are still on a fairly fundamental level.
  
  You spend some time saying what abstractions are, but when I see the hypothesis written down, most of my confusion is on what “cognitive systems” are and what one means by “most”. Afaict it really is a kind of empirical question to do with “most cognitive systems”. Do we have in mind something like ‘animal brains and artificial neural networks’? If so then surely let’s just say that and make the whole thing more concrete; so I suspect not....but in that case....what does it include? And how we will know if ‘most’ of them have some property? (At the moment, whenever I find evidence that two systems don’t share an abstraction that they ‘ought to’ I can go “well the hypothesis is only most”...)

• carboniferous_umbraculum Mar 15, 2023, 10:33 AM

  4 points

  1

  in reply to: habryka’s comment on: Shut­ting Down the Light­cone Offices

  Something ~ like ‘make it legit’ has been and possibly will continue to be a personal interest of mine.
  
  I’m posting this after Rohin entered this discussion—so Rohin, I hope you don’t mind me quoting you like this, but fwiw I was significantly influenced by this comment on Buck’s old talk transcript ‘My personal cruxes for working on AI safety’. (Rohin’s comment repeated here in full and please bear in mind this is 3 years old; his views I’m sure have developed and potentially moved a lot since then:)

  
  I enjoyed this post, it was good to see this all laid out in a single essay, rather than floating around as a bunch of separate ideas.

  That said, my personal cruxes and story of impact are actually fairly different: in particular, while this post sees the impact of research as coming from solving the technical alignment problem, I care about other sources of impact as well, including:

  1. Field building: Research done now can help train people who will be able to analyze problems and find solutions in the future, when we have more evidence about what powerful AI systems will look like.

  2. Credibility building: It does you no good to know how to align AI systems if the people who build AI systems don’t use your solutions. Research done now helps establish the AI safety field as the people to talk to in order to keep advanced AI systems safe.

  3. Influencing AI strategy: This is a catch all category meant to include the ways that technical research influences the probability that we deploy unsafe AI systems in the future. For example, if technical research provides more clarity on exactly which systems are risky and which ones are fine, it becomes less likely that people build the risky systems (nobody _wants_ an unsafe AI system), even though this research doesn’t solve the alignment problem.

  As a result, cruxes 3-5 in this post would not actually be cruxes for me (though 1 and 2 would be).

A Neu­ral Net­work un­der­go­ing Gra­di­ent-based Train­ing as a Com­plex System

• carboniferous_umbraculum Feb 18, 2023, 1:57 AM

  2 points

  1

  in reply to: scasper’s comment on: EIS V: Blind Spots In AI Safety In­ter­pretabil­ity Research

  Certainly it’s not a necessarily good thing either. I would posit isolation is usually not good. I can personally attest to being confused and limited by the difference in terminology here. And I think that when it comes to intrinsic interpretability work in particular, the disentanglement literature has produced a number of methods of value while TAISIC has not.

  Ok it sounds to me like maybe there’s at least two things being talked about here. One situation is
  
  A) Where a community includes different groups working on the same topic, and where those groups might use different terminology and have different ways of thinking about the same phenomena etc. This seems completely normal to me. The other situation is
  
  B) Where a group is isolated from the community at large and is using different terminology/​thinking about things differently just as a result of their isolation and lack of communication. And where that behaviour then causes confusion and/​or wasting of resources.
  
  The latter doesn’t sound good, but I guess it looks like to me that some or many of your points are consistent with the former being the case. So when you write e.g. it’s not “necessarily a good thing either” or asking for my steelmanned case, this doesn’t seem to quite make sense to me. I feel like if something is not necessarily good or bad, and you want to raise it as a criticism, then the onus would be on you to bring the case against TASIC with arguments that are not general ones that could easily apply to both A) and B) above. e.g. It’d be more of an emphatic case if you were able to go into the details and be like “X did this work here and claimed it was new but actually it exists in Y’s paper here” or give a real example of needless confusion that was created and could have been avoided. Focussing just on what they did or didn’t ‘engage with’ on the level of general concepts and citations/​acknowledgements doesn’t bring this case convincingly, in my opinion. Some more vague thoughts on why that is:

  • Bodies of literature like this are usually very complicated and messy and people genuinely can’t be expected to engage with everything.

  • It’s often hard or impossible to tack dependencies of ideas because of all the communication you cannot see and not being able to see ‘how’ people are thinking of things, only what they wrote.

  • Someone publishing on the same idea or concept or topic as you is nowhere near the same as someone actually doing the exact same technical thing that you are doing. ime the former is happening all the time; and the latter is much rarer than people often think.

  • Reinvention, re-presentation and even outright renaming or ‘starting from scratch’ are all valuable elements of scholarship that help a field move along.

  Idk maybe I’m just repeating myself at this point.

  On the other point: It may turn out the MI’s analogy with reverse software engineering does not produce methods and is just used a high-level analogy,, but it seems too early to say from my perspective—the two posts I linked are from last year. TASIC is still pretty small and experienced researchers in TASIC are fewer and this is potentially a large and difficult research agenda.
  

• carboniferous_umbraculum Feb 17, 2023, 7:20 PM

  4 points

  3

  on: EIS V: Blind Spots In AI Safety In­ter­pretabil­ity Research

  Re: e.g. superposition/​entanglement:
  
  I think people should try to understand the wider context into which they are writing, but I don’t see it as necessarily a bad thing if two groups of researchers are working on the same idea under different names. In fact I’d say this happens all the time and generally people can just hold in their minds that another group has another name for it. Naturally, the two groups will have slightly different perspectives and this a) Is often good, i.e. the interference can be constructive and b) Can be a reason in favour of different terminology, i.e. even if something is “the same” when boiled down to a formal level, the different names can actually help delineate different interpretations.
  
  In fact it’s almost like a running joke in academia that there’s always someone grumbling that you didn’t cite the right things (their favourite work on this topic, their fellow countryman, them etc.) and because of the way academic literature works, some of the things that you are doing here can be done with almost any piece of work in the literature, i.e. you can comb over it with the benefit of hindsight and say ‘hang on this isn’t as original as it looked; basically the same idea that was written about here X years before’ etc. Honestly, I don’t usually think of this as a valuable exercise, but I may be missing something about your wider point or be more convinced once I’ve looked at more of your series.
  
  Another point when it comes to ‘originality’ and ‘progress’ is that it’s often unimportant if some idea was generally discussed, labelled, named, or thought about before if what matters is actual results and the lower-level content of these works. i.e. I may be wrong, but looking at what you are saying, I don’t think you are literally pulling up an older paper on ‘entanglement’ that made the exact same points that the Anthropic papers were making and did very similar experiments (Or are you?) And even having said that, reproducing experiments exactly is of course very valuable.
  
  Re: MI and program synthesis:
  
  I understand that your take is that it is closer to program synthesis or program induction and that these aren’t all the same thing but in the first subsection of the “TASIC has reinvented...” section, I’m a little confused why there’s no mention of reverse engineering programs from compiled binary? The analogy with reverse engineering programs is one that MI people have been actively thinking about, writing about and trying to understand ( see e.g. Olah, and Nanda, in which he consults an expert).

• carboniferous_umbraculum Feb 11, 2023, 11:45 PM

  2 points

  1

  in reply to: QSED’s comment on: On Devel­op­ing a Math­e­mat­i­cal The­ory of In­ter­pretabil­ity

  Thanks very much for the comments I think you’ve asked a bunch of very good questions. I’ll try to give some thoughts:

  Deep learning as a field isn’t exactly known for its rigor. I don’t know of any rigorous theory that isn’t as you say purely ‘reactive’, with none of it leading to any significant ‘real world’ results. As far as I can tell this isn’t for a lack of trying either. This has made me doubt its mathematical tractability, whether it’s because our current mathematical understanding is lacking or something else (DL not being as ‘reductionist’ as other fields?). How do you lean in this regard? You mentioned that you’re not sure when it comes to how amenable interpretability itself is, but would you guess that it’s more or less amenable than deep learning as a whole?

  I think I kind of share your general concern here and I’m uncertain about it. I kind of agree that it seems like people had been trying for a while to figure out the right way to think about deep learning mathematically and that for a while it seemed like there wasn’t much progress. But I mean it when I say these things can be slow. And I think that the situation is developing and has changed—perhaps significantly—in the last ~5 years or so, with things like the neural tangent kernel, the Principles of Deep Learning Theory results and increasingly high-quality work on toy models. (And even when work looks promising, it may still take a while longer for the cycle to complete and for us to get ‘real world’ results back out of these mathematical points of view, but I have more hope than I did a few years ago). My current opinion is that certain aspects of interpretability will be more amenable to mathematics than understanding DNN-based AI as a whole .

  
  How would success of this relate to capabilities research? It’s a general criticism of interpretability research that it also leads to heightened capabilities, would this fare better/​worse in that regard? I would have assumed that a developed rigorous theory of interpretability would probably also entail significant development of a rigorous theory of deep learning.

  I think basically your worries are sound. If what one is doing is something like ‘technical work aimed at understanding how NNs work’ then I don’t see there as being much distinction between capabilities and alignment ; you are really generating insights that can be applied in many ways, some good some bad (and part of my point is you have to be allowed to follow your instincts as a scientist/​mathematician in order to find the right questions). But I do think that given how slow and academic the kind of work I’m talking about is, it’s neglected by both a) short timelines-focussed alignment people and b) capabilities people.

  
  How likely is it that the direction one may proceed in would be correct? You mention an example in mathematical physics, but note that it’s perhaps relatively unimportant that this work was done for ‘pure’ reasons. This is surprising to me, as I thought that a major motivation for pure math research, like other blue sky research, is that it’s often not apparent whether something will be useful until it’s well developed. I think this is the similar to you mentioning that the small scale problems will not like the larger problem. You mention that this involves following one’s nose mathematically, do you think this is possible in general or only for this case? If it’s the latter, why do you think interpretability is specifically amenable to it?

  Hmm, that’s interesting. I’m not sure I can say how likely it is one would go in the correct direction. But in my experience the idea that ‘possible future applications’ is one of the motivations for mathematicians to do ‘blue sky’ research is basically not quite right. I think the key point is that the things mathematicians end up chasing for ‘pure’ math/​aesthetic reasons seem to be oddly and remarkably relevant when we try to describe natural phenomena (iirc this is basically a key point in Wigner’s famous ‘Unreasonable Effectiveness’ essay.) So I think my answer to your question is that this seems to be something that happens “in general” or at least does happen in various different places across science/​applied math

• carboniferous_umbraculum Feb 9, 2023, 7:50 PM

  3 points

  0

  in reply to: Daniel Paleka’s comment on: Notes on the Math­e­mat­ics of LLM Architectures

  Ah thanks very much Daniel. Yes now that you mention it I remember being worried about this a few days ago but then either forgot or (perhaps mistakenly) decided it wasn’t worth expanding on. But yeah I guess you don’t get a well-defined map until you actually fix how the tokenization happens with another separate algorithm. I will add to list of things to fix/​expand on in an edit.

• carboniferous_umbraculum Feb 9, 2023, 7:47 PM

  9 points

  6

  in reply to: Roman Leventov’s comment on: On Devel­op­ing a Math­e­mat­i­cal The­ory of In­ter­pretabil­ity

  >There is no difference between natural phenomena and DNNs (LLMs, whatever). DNNs are 100% natural

  I mean “natural” as opposed to “man made”. i.e. something like “occurs in nature without being built by something or someone else”. So in that sense, DNNs are obviously not natural in the way that the laws of physics are.

  I don’t see information and computation as only mathematical; in fact in my analogies I write that the mathematical abstractions we build as being separate from the things that one wants to describe or make predictions about. And this applies to the computations in NNs too.

  I don’t want to study AI as mathematics or believe that AI is mathematics. I write that the practice of doing mathematics will only seek out the parts of the problem that are actually amenable to it; and my focus is on interpretability and not other places in AI that one might use mathematics (like, say, decision theory).

  You write “As an example, take “A mathematical framework for transformer circuits”: it doesn’t develop new mathematics. It just uses existing mathematics: tensor algebra.:” I don’t think we are using ‘new mathematics’ in the same way and I don’t think the way you are using it commonplace. Yes I am discussing the prospect of developing new mathematics, but this doesn’t only mean something like ‘making new definitions’ or ‘coming up with new objects that haven’t been studied before’. If I write a proof of a theorem that “just” uses “existing” mathematical objects, say like...matrices, or finite sets, then that seems to have little bearing on how ‘new’ the mathematics is. It may well be a new proof, of a new theorem, containing new ideas etc. etc. And it may well need to have been developed carefully over a long period of time.

Notes on the Math­e­mat­ics of LLM Architectures

On Devel­op­ing a Math­e­mat­i­cal The­ory of In­ter­pretabil­ity