DPiepgrass

Karma: 473

Worried that typical commenters at LW care way less than I expected about good epistemic practice. Hoping I’m wrong.

Software developer and EA with interests including programming language design, international auxiliary languages, rationalism, climate science and the psychology of its denial.

Looking for someone similar to myself to be my new best friend:

❖ Close friendship, preferably sharing a house ❖ Rationalist-appreciating epistemology; a love of accuracy and precision to the extent it is useful or important (but not excessively pedantic) ❖ Geeky, curious, and interested in improving the world ❖ Liberal/humanist values, such as a dislike of extreme inequality based on minor or irrelevant differences in starting points, and a like for ideas that may lead to solving such inequality. (OTOH, minor inequalities are certainly necessary and acceptable, and a high floor is clearly better than a low ceiling: an “equality” in which all are impoverished would be very bad) ❖ A love of freedom ❖ Utilitarian/consequentialist-leaning; preferably negative utilitarian ❖ High openness to experience: tolerance of ambiguity, low dogmatism, unconventionality, and again, intellectual curiosity ❖ I’m a nudist and would like someone who can participate at least sometimes ❖ Agnostic, atheist, or at least feeling doubts

Calgary ACX Meetup

DPiepgrass4 Oct 2024 21:49 UTC

4 points

0 comments1 min readLW link

DPiepgrass 4 Oct 2024 19:54 UTC
1 point
0
in reply to: Seth Herd’s comment on: DPiepgrass’s Shortform
Well, yeah, it bothers me that the “bayesian” part of rationalism doesn’t seem very bayesian―otherwise we’d be having a lot of discussions about where priors come from, how to best accomplish the necessary mental arithmetic, how to go about counting evidence and dealing with ambiguous counts (if my friends Alice and Bob both tell me X, it could be two pieces of evidence for X or just one depending on what generated the claims; how should I count evidence by default, and are there things I should be doing to find the underlying evidence?)
So―vulnerable in the current culture, but rationalists should strive to be the opposite of the “gishy” dark-epistemic people I have on my mind. Having many reasons to think X isn’t necessarily a sin, but dark-epistemic people gather many reasons and have many sins, which are a good guide of what not to do.

DPiepgrass 4 Oct 2024 19:33 UTC
1 point
0
in reply to: quila’s comment on: DPiepgrass’s Shortform
TBH, a central object of interest to me is people using Dark Epistemics. People with a bad case of DE typically have “gishiness” as a central characteristic, and use all kinds of fallacies of which motte-and-bailey (hidden or not) is just one. I describe them together just because I haven’t seen LW articles on them before. If I were specifically naming major DE syndromes, I might propose the “backstop of conspiracy” (a sense that whatever the evidence at hand doesn’t explain is probably still explained by some kind of conspiracy) and projection (a tendency to loudly complain that one’s political enemies have whatever negative characteristics you yourself, or your political heroes, are currently exhibiting). Such things seem very effective at protecting the person’s beliefs from challenge. I think there’s also a social element (“my friends all believe the same thing”) but this is kept well-hidden. EDIT: other telltale signs include refusing to acknowledge that one got anything wrong or made any mistake, no matter how small; refusing to acknowledge that the ‘opponent’ is right about anything, no matter how minor; an allergy to detail (refusing to look at the details of any subtopic); and shifting the playfield repeatedly (changing the topic when one appears to be losing the argument).

DPiepgrass 30 Sep 2024 14:48 UTC
29 points
4
on: DPiepgrass’s Shortform
The Hidden-Motte-And-Bailey fallacy: belief in a Bailey inspires someone to invent a Motte and write an article about it. The opinion piece describes the Motte exclusively with no mention of the Bailey. Others read it and nod along happily because it supports their cherished Bailey, and finally they share it with others in an effort to help promote the Bailey.
Example: Christian philosopher describes new argument for the existence of a higher-order-infinity God which bears no resemblance to any Abrahamic God, and which no one before the 20th century had ever conceived of.
Maybe the Motte is strong, maybe it isn’t, but it feels very strong when combined with the Gish Fallacy: the feeling of safety some people (apparently) get by collecting large numbers of claims and arguments, whether or not they routinely toss them out as Gish Gallops at anyone who disagrees. The Gish Fallacy seems to be the opposite of the mathematician’s mindset, for mathematicians know that a single flaw can destroy proofs of any length. While the mathematician is satisfied by a single a short and succinct proof or disproof, the Gish mindset wishes to read a thousand different descriptions of three dozen arguments, with another thousand pithy rejections of their counterarguments, so they’re thoroughly prepared to dismiss the evil arguments of the enemies of truth―or, if the enemy made a good point, there are still 999 articles supporting their view, and more where that came from!
Example: my dad

DPiepgrass 28 Sep 2024 19:28 UTC
1 point
0
in reply to: Eliezer Yudkowsky’s comment on: Argument Screens Off Authority
I know this comment was 17 years ago, but nearly half of modern US politics in 2024 is strongly influenced by the idea “don’t trust experts”. I have listened to a lot of cranks (ex), and the popular ones are quite good at what they do, rhetorically, so most experts have little chance against them in a debate. Plus, if the topic is something important, the public was already primed to believe one side or the other by whatever media they happen to consume, and if the debate winner would otherwise seem close (to a layman) then such preconceptions will dominate perceptions about who won.
One thing these winning cranks take advantage is that the audience (like the cranks themselves!) tend to believe they are “knowledgeable enough to understand all the technical arguments” when they’re not. Also, most experts have much worse teaching skills than these cranks (who are popular for a reason) so the expert tends to provide worse explanations, while also overlooking key counterarguments they could make against the crank’s case.
So: in today’s environment, I see a large fraction of the population as giving too little credence to authority in cases where they only partially evaluate the arguments. Related SSC. Like proofs, arguments can be completely wrong due to a single flaw. A partial evaluation will not necessarily uncover the flaw, but pop cranks are often good at pointing out things that appear as flaws to the uneducated (perhaps because they have a shorter inferential distance to the audience than the experts do, but also because they have more practice in public debate). And needless to say, people who are undecided don’t know that they’re listening to a crank, and would be offended if you claimed that the one who they thought won the debate was a crank.

DPiepgrass 22 Aug 2024 23:03 UTC
−1 points
−2
in reply to: Rob Bensinger’s comment on: MIRI announces new “Death With Dignity” strategy
I actually think Yudkowsky’s biggest problem may be that he is not talking about his models. In his most prominent posts about AGI doom, such as this and the List of Lethalities, he needs to provide a complete model that clearly and convincingly leads to doom (hopefully without the extreme rhetoric) in order to justify the extreme rhetoric. Why does attempted, but imperfect, alignment lead universally to doom in all likely AGI designs*, when we lack familiarity with the relevant mind design space, or with how long it will take to escalate a given design from AGI to ASI?
* I know his claim isn’t quite this expansive, but his rhetorical style encourages an expansive interpretation.
I’m baffled he gives so little effort to explaining his model. In List of Lethalities he spends a few paragraphs of preamble to cover some essential elements of concern (-3, −2, −1), then offers a few potentially-reasonable-but-minimally-supported assertions, before spending much of the rest of the article prattling off the various ways AGI can kill everyone. Personally I felt like he just skipped over a lot of the important topics, and so didn’t bother to read it to the end.
I think there is probably some time after the first AGI or quasi-AGI arrives, but before the most genocide-prone AGI arrives, in which alignment work can still be done. Eliezer’s rhetorical approach confusingly chooses to burn bridges with this world, as he and MIRI (and probably, by association, rationalists) will be regarded as a laughing stock when that world arrives. Various techbros including AI researchers will be saying “well, AGI came and we’re all still alive, yet there’s EY still reciting his doomer nonsense”. EY will uselessly protest “I didn’t say AGI would necessarily kill everyone right away” while the techbros retweet old EY quotes that kinda sound like that’s what he’s saying.
Edit: for whoever disagreed & downvoted: what for? You know there are e/accs on Twitter telling everyone that the idea of x-risk is based on Yudkowsky being “king of his tribe”, and surely you know that this is not how LessWrong is supposed to work. The risk isn’t supposed to be based on EY’s say-so; a complete and convincing model is needed. If, on the other hand, you disagreed that his communication is incomplete and unconvincing, it should not offend you that not everyone agrees. Like, holy shit: you think humanity will cause apocalypse because it’s not listening to EY, but how dare somebody suggest that EY needs better communication. I wrote this comment because I think it’s very important; what are you here for?
What links here?
- TAG's comment on james oofou’s Shortform by james oofou (14 Sep 2024 10:42 UTC; 3 points)

DPiepgrass 22 Aug 2024 19:19 UTC
6 points
0
in reply to: DPiepgrass’s comment on: LLM Generality is a Timeline Crux
P.S. if I’m wrong about the timeline―if it takes >15 years―my guess for how I’m wrong is (1) a major downturn in AGI/AI research investment and (2) executive misallocation of resources. I’ve been thinking that the brightest minds of the AI world are working on AGI, but maybe they’re just paid a lot because there are too few minds to go around. And when I think of my favorite MS developer tools, they have greatly improved over the years, but there are also fixable things that haven’t been fixed in 20 years, and good ideas they’ve never tried, and MS has created a surprising number of badly designed libraries (not to mention products) over the years. And I know people close to Google have a variety of their own pet peeves about Google.
Are AGI companies like this? Do they burn mountains cash to pay otherwise average engineers who happen to have AI skills? Do they tend to ignore promising research directions because the results are uncertain, or because results won’t materialize in the next year, or because they don’t need a supercomputer or aren’t based mainly on transformers? Are they bad at creating tools that would’ve made the company more efficient? Certainly I expect some companies to be like that.
As for (1), I’m no great fan of copyright law, but today’s companies are probably built on a foundation of rampant piracy, and litigation might kill investment. Or, investors may be scared away by a persistent lack of discoveries to increase reliability / curtail hallucinations.

DPiepgrass 22 Aug 2024 5:03 UTC
1 point
0
in reply to: Violet Hour’s comment on: LLM Generality is a Timeline Crux
Doesn’t the problem have no solution without a spare block?
Worth noting that LLMs don’t see a nicely formatted numeric list, they see a linear sequence of tokens, e.g. I can replace all my newlines with something else and Copilot still gets it:
brief testing doesn’t show worse completions than when there are newlines. (and in the version with newlines this particular completion is oddly incomplete.)
Anyone know how LLMs tend to behave on text that is ambiguous―or unambiguous but “hard to parse”? I wonder if they “see” a superposition of meanings “mixed together” and produce a response that “sounds good for the mixture”.

DPiepgrass 22 Aug 2024 4:16 UTC
1 point
0
in reply to: Nathan Helm-Burger’s comment on: LLM Generality is a Timeline Crux
I’m having trouble discerning a difference between our opinions, as I expect a “kind-of AGI” to come out of LLM tech, given enough investment. Re: code assistants, I’m generally disappointed with Github Copilot. It’s not unusual that I’m like “wow, good job”, but bad completions are commonplace, especially when I ask a question in the sidebar (which should use a bigger LLM). Its (very hallucinatory) response typically demonstrates that it doesn’t understand our (relatively small) codebase very well, to the point where I only occasionally bother asking. (I keep wondering “did no one at GitHub think to generate an outline of the app that could fit in the context window?”)

DPiepgrass 22 Aug 2024 3:19 UTC
3 points
0
on: You don’t know how bad most things are nor precisely how they’re bad.
A title like “some people can notice more imperfections than you (and they get irked)” would be more accurate and less clickbaity, though when written like that it it sounds kind of obvious.

DPiepgrass 22 Aug 2024 2:23 UTC
1 point
0
in reply to: MondSemmel’s comment on: DPiepgrass’s Shortform
Do you mean the “send us a message” popup at bottom-right?

DPiepgrass 20 Aug 2024 2:35 UTC
4 points
1
in reply to: Martin Randall’s comment on: DPiepgrass’s Shortform
Yikes! Apparently I said “strong disagree” when I meant “strong downvote”. Fixed. Sorry. Disagree votes generally don’t bother me either, they just make me curious what the disagreer disagrees about.

DPiepgrass 19 Aug 2024 22:31 UTC
4 points
0
in reply to: habryka’s comment on: DPiepgrass’s Shortform
Shoot, I forgot that high-karma users have a “small-strength” of 2, so I can’t tell if it was strong-downvoted or not. I mistakenly assumed it was a newish user. Edit: P.S. I might feel better if karma was hidden on my own new comments, whether or not they are hidden on others, though it would then be harder to guess at the vote distribution, making the information even more useless than usual if it survives the hiding-period. Still seems like a net win for the emotional benefits.

DPiepgrass 19 Aug 2024 22:21 UTC
3 points
−3
on: DPiepgrass’s Shortform
I wrote a long comment and someone took the “strong downvote, no disagreement, no reply” tack again. Poppy-cutters^[1] seem way more common at LessWrong than I could’ve predicted, and I’d like to see statistics to see how common they are, and to see whether my experience here is normal or not.
1. ^
  Meaning users who just seem to want to make others feel bad about their opinion, especially unconstructively. Interesting statistics might include the distributions of users who strong-downvote a positive score into a negative one (as compared to their other voting behaviors), who downvote more than they disagree, who downvote more than they upvote, who strong-downvote more than they downvote, and/or strong-downvote more often than they reply. Also: is there an ⁸⁰⁄₂₀ rule like “20% of the people do 80% of the downvotes”? Is the distribution different for “power users” than “casual users”?

DPiepgrass 19 Aug 2024 20:56 UTC
13 points
2
on: LLM Generality is a Timeline Crux
Given that I think LLMs don’t generalize, I was surprised how compelling Aschenbrenner’s case sounded when I read it (well, the first half of it. I’m short on time...). He seemed to have taken all the same evidence I knew about it, and arranged it into a very different framing. But I also felt like he underweighted criticism from the likes of Gary Marcus. To me, the illusion of LLMs being “smart” has been broken for a year or so.
To the extent LLMs appear to build world models, I think what you’re seeing is a bunch of disorganized neurons and connections that, when probed with a systematic method, can be mapped onto things that we know a world model ought to contain. A couple of important questions are
1. the way that such a world model was formed and
2. how easily we can easily figure out how to form those models better/differently^[1].
I think LLMs get “world models” (which don’t in fact cover the whole world) in a way that is quite unlike the way intelligent humans form their own world models―and more like how unintelligent or confused humans do the same.
The way I see it, LLMs learn in much the same way a struggling D student learns (if I understand correctly how such a student learns), and the reason LLMs sometimes perform like an A student is because they have extra advantages that regular D students do not: unlimited attention span and ultrafast, ultra-precise processing backed by an extremely large set of training data. So why do D students perform badly, even with “lots” of studying? I think it’s either because they are not trying to build mental models, or because they don’t really follow what their teachers are saying. Either way, this leads them to fall back on secondary “pattern-matching” learning mode which doesn’t depend on a world model.
If, when learning in this mode, you see enough patterns, you will learn an implicit world model. The implicit model is a proper world model in terms of predictive power, but
1. It requires much more training data to predict as well as a human system-2 can, which explains why D students perform worse than A students on the same amount of training data―and this is one of the reasons why LLMs need so much more training data than humans do in order to perform at an A level (other reasons: less compute per token, fewer total synapses, no ability to “mentally” generate training data, inability to autonomously choose what to train on). The way you should learn is to first develop an explicit worldmodel via system-2 thinking, then use system-2 to mentally generate training data which (along with external data) feeds into system-1. LLMs cannot do this.
2. Such a model tends to be harder to explain in words than an explicit world model, because the predictions are coming from system-1 without much involvement from system-2, and so much of the model is not consciously visible to the student, nor is it properly connected to its linguistic form, so the D student relies more on “feeling around” the system-1 model via queries (e.g. to figure out whether “citation” is a noun, you can do things like ask your system-1 whether “the citation” is a valid phrase―human language skills tend to always develop as pure system-1 initially, so a good linguistics course teaches you explicitly to perform these queries to extract information, whereas if you have a mostly-system-2 understanding of a language, you can use that to decide whether a phrase is correct with system-2, without an intuition about whether it’s correct. My system-1 for Spanish is badly underdeveloped, so I lean on my superior system-2/analytical understanding of grammar).
  
  When an LLM cites a correct definition of something as if it were a textbook, then immediately afterward fails to apply that definition to the question you ask, I think that indicates the LLM doesn’t really have a world model with respect to that question, but I would go further and say that even if it has a good world model, it cannot express its world-model in words, it can only express the textbook definitions it has seen and then apply its implicit world-model, which may or may not match what it said verbally.
So if you just keep training it on more unique data, eventually it “gets it”, but I think it “gets it” the way a D student does, implicitly not explicitly. With enough experience, the D student can be competent, but never as good as similarly-experienced A students.
A corollary of the above is that I think the amount of compute required for AGI is wildly overestimated, if not by Aschenbrenner himself then by less nuanced versions of his style of thinking (e.g. Sam Altman). And much of the danger of AGI follows from this. On a meta level, my own opinions on AGI are mostly not formed via “training data”, since I have not read/seen that many articles and videos about AGI alignment (compared to any actual alignment researcher). No coincidence, then, that I was always an A to A- student, and the one time I got a C- in a technical course was when I couldn’t figure out WTF the professor was talking about. I still learned, that’s why I got a C-, but I learned in a way that seemed unnatural to me, but which incorporated some of the “brute force” that an LLM would use. I’m all about mental models and evidence; LLMs are about neither.
Aschenbrenner did help firm up my sense that current LLM tech leads to “quasi-AGI”: a competent humanlike digital assistant, probably one that can do some AI research autonomously. It appears that the AI industry (or maybe just OpenAI) is on an evolutionary approach of “let’s just tweak LLMs and our processes around them”. This may lead (via human ingenuity or chance discovery) to system-2s with explicit worldmodels, but without some breakthough, it just leads to relatively safe quasi-AGIs, the sort that probably won’t generate groundbreaking new cancer-fighting ideas but might do a good job testing ideas for curing cancer that are “obvious” or human-generated or both.
Although LLMs badly suck at reasoning, my AGI timelines are still kinda short―roughly 1 to 15 years for “real” AGI, with quasi-AGI in 2 to 6 years―mainly because so much funding is going into this, and because only one researcher needs to figure out the secret, and because so much research is being shared publicly, and because there should be many ways to do AGI, and because quasi-AGI (if invented first) might help create real AGI. Even the AGI safety people^[2] might be the ones to invent AGI, for how else will they do effective safety research? FWIW my prediction is that quasi-AGI is consists of a transformer architecture with quite a large number of (conventional software) tricks and tweaks bolted on to it, while real AGI consists of transformer architecture plus a smaller number of tricks and tweaks, plus a second breakthrough of the same magnitude as transformer architecture itself (or a pair of ideas that work so well together that combining them counts as a breakthrough).
EDIT: if anyone thinks I’m on to something here, let me know your thoughts as to whether I should redact the post lest changing minds in this regard is itself hazardous. My thinking for now, though, is that presenting ideas to a safety-conscious audience might well be better than safetyists nodding along to a mental model that I think is, if not incorrect, then poorly framed.
1. ^
  I don’t follow ML research, so let me know if you know of proposed solutions already.
2. ^
  why are we calling it “AI safety”? I think this term generates a lot of “the real danger of AI is bias/disinformation/etc” responses, which should decrease if we make the actual topic clear

DPiepgrass 19 Aug 2024 19:13 UTC
20 points
9
on: DPiepgrass’s Shortform
I have a feature request for LessWrong. It’s the same as my feature request for every site: you should be able to search within a user (i.e. visit a user page and begin a search from there). This should be easy to do technically; you just have to add the author’s name as one of the words in the search index. And in terms of UI, I think you could just link to https://www.lesswrong.com/search?query=@author:username.
Preferably do it in such a way that a normal post cannot do the same, e.g. if “foo authored this post” is placed in the index as @author:foo, if a normal post contains the text @author:foo then perhaps the index only ends up with @author (or author) and foo, while the full string is not in the index (or, if it is in the index, can only be found by searching with quotes a la Google: "@author:foo"
P.S. we really should be able to search by date range, too. I’m looking for something posted by Yudkowsky at some point before 2010… and surely this is not an unusual thing to want to search for.

DPiepgrass’s Shortforms

DPiepgrass19 Aug 2024 19:13 UTC

4 points

15 comments1 min readLW link

DPiepgrass 19 Aug 2024 18:59 UTC
1 point
0
on: Feature Wish List for LessWrong
Wow, is this the newest feature-request thread? Well, it’s the newest I easily find given that LessWrong search has very poor granularity when it comes to choosing a time frame within which to search...
My feature request for LessWrong is the same as my feature request for every site: you should be able to search within a user. This is easy to do technically; you just have to add the author’s name as one of the words in the search index.
Preferably do it in such a way that a normal post cannot do the same, e.g. you might put ”foo authored this post” in the index as @author:foo but if a normal post contains the text “@author:foo” then perhaps the index only ends up with @author (or author) and foo, while the full string is not in the index (or, if it is in the index, can only be found by searching with quotes a la Google: "@author:foo"

DPiepgrass 6 Aug 2024 20:30 UTC
2 points
3
in reply to: habryka’s comment on: Towards more cooperative AI safety strategies
I have definitely taken actions within the bounds of what seems reasonable that have aimed at getting the EA community to shut down or disappear (and will probably continue to do so).
Wow, what a striking thing for you to say without further explanation.
Personally, I’m a fan of EA. Also am an EA―signed the GWWC/10% pledge and all that. So, I wonder what you mean.

Calgary ACX meetup, Aug. 4

DPiepgrass1 Aug 2024 14:14 UTC

4 points

0 comments1 min readLW link

DPiepgrass

Cal­gary ACX Meetup

DPiep­grass’s Shortforms

Cal­gary ACX meetup, Aug. 4

Calgary ACX Meetup

DPiepgrass’s Shortforms

Calgary ACX meetup, Aug. 4