I’m an independent researcher currently working on a sequence of posts about consciousness. You can send me anonymous feedback here: https://www.admonymous.co/rafaelharth. If it’s about a post, you can add [q] or [nq] at the end if you want me to quote or not quote it in the comment section.
Rafael Harth
Karma: 4,913
No I definitely think thought assessment has more to it than just attention. In fact I think you could argue that LLMs’ attention equivalent is already more powerful/accurate than human attention.
The only evidence I can provide at this point is the similarity of LLMs to humans who don’t pay attention (as first observed in Sarah’s post that I linked in the text). If you want to reject the post based on the lack of evidence for this claim, I think that’s fair.
I think we have specialized architectures for consciously assessing thoughts, whereas LLMs do the equivalent of rattling off the first thing that comes to mind, and reasoning models do the equivalent of repeatedly feeding back what comes to mind into the input (and rattling off the first thing that comes to mind for that input).
Trump says a lot of stuff that he doesn’t do, the set of specific things that presidents don’t do is larger than the set of things they do, and tariffs didn’t even seem like they’d be super popular with his base if in fact they were implemented. So “~nothing is gonna happen wrt tariffs” seemed like the default outcome with not enough evidence to assume otherwise.
I was also not paying a lot of attention to what he was saying. After the election ended, I made a conscious decision to tune out of politics to protect my mental health. So it was a low information take—but I don’t know if paying more attention would have changed my prediction. I still don’t think I actually know why Trump is doing the tariffs, especially to such an extreme extent..
I’ve also noticed this assumption. I myself don’t have it, at all. My first thought has always been something like “If we actually get AGI then preventing terrible outcomes will probably require drastic actions and if anything I have less faith in the US government to take those”. Which is a pretty different approach from just assuming that AGI being developed by government will automatically lead to a world with values of government . But this a very uncertain take and it wouldn’t surprise me if someone smart could change my mind pretty quickly.
Yeah.
These are very poor odds, to the point that they seem to indicate a bullish rather than a bearish position on AI.
If you think the odds of something are , but lots of other people think they are with , then the rational action is not to offer bets at a point close to ; it’s to find the closest number to possible. Why would you bet at 1:5 odds if you have reason to believe that some people would be happy to bet at 1:7 odds?
You could make an argument that this type of thinking is too mercenary/materialistic or whatever, but then critique should be about that. In any case the inference that offering a bet close to indicates beliefs close to is just not accurate.
I’m glad METR did this work, and I think their approach is sane and we should keep adding data points to this plot.
It sounds like you also think the current points on the plot are accurate? I would strongly dispute this, for all the reasons discussed here and here. I think you can find sets of tasks where the points fit on an exponential curve, but I don’t think AI can do 1 hour worth of thinking on all, or even most, practically relevant questions.
In the last few months, GPT models have undergone a clear shift toward more casual language. They now often close a post by asking a question. I strongly dislike this from both a ‘what will this do to the public’s perception of LLMs’ and ‘how is my personal experience as a customer’ perspective. Maybe this is the reason to finally take Gemini seriously.
I unfortunately don’t think this proves anything relevant. The example just shows that there was one question where the market was very uncertain. This neither tells us how certain the market is in general (that depends on its confidence on other policy questions), nor how good this particular estimate was (that, I would argue, depends on how far along the information chart it was, which is not measurable—but even putting my pet framework aside, it seems intuitively clear “it was 56% and then it happened” doesn’t tell you how much information the market utilized).
The point is that even if voters did everything right and checked prediction markets as part of their decision making algorithm, it wouldn’t help.
This depends on the first point, which again requires looking at a range of policy markets, not just one. And actually, I personally didn’t expect Trump to do any tarrifs at all (was 100% wrong there), so for me, the market would have updated me significantly into the right direction.
Thanks. I’ve submitted my own post on the ‘change our mind form’, though I’m not expecting a bounty. I’d instead be interested in making a much bigger bet (bigger than Cole’s 100 USD), gonna think about what resolution criterion is best.
I might be misunderstanding how this works, but I don’t think I’m gonna win the virtue of The Void anytime soon. Or at all.
Yeah, valid correction.
If people downvoted because they thought the argument wasn’t useful, fine—but then why did no one say that? Why not critique the focus or offer a counter? What actually happened was silence, followed by downvotes. That’s not rational filtering. That’s emotional rejection.
Yeah, I do not endorse the reaction. The situation pattern-matches to other cases where someone new writes things that are so confusing and all over the place that making them ditch the community (which is often the result of excessive downvoting) is arguably a good thing. But I don’t think this was the case here. Your essays look to me to be coherent (and also probably correct). I hadn’t seen any of them before this post but I wouldn’t have downvoted. My model is that most people are not super strategic about this kind of thing and just go “talking politics → bad” without really thinking through whether demotivating the author is good in this case.
So if I understand you correctly: you didn’t read the essay, and you’re explaining that other people who also didn’t read the essay dismissed it as “political” because they didn’t read it.
Yes—from looking at it, it seems like it’s something I agree with (or if not, disagree for reasons that I’m almost certain won’t be addressed in the text), so I didn’t see a reason to read. I mean reading is a time investment, you have to give me a reason to invest that time, that’s how it works. But I thought the (lack of) reaction was unjustified, so I wanted to give you a better model of what happened, which also doesn’t take too much time.
Most people say capitalism makes alignment harder. I’m saying it makes alignment structurally impossible.
The point isn’t to attack capitalism. It’s to explain how a system optimised for competition inevitably builds the thing that kills us.
I mean that’s all fine, but those are nuances which only become relevant after people read, so it doesn’t really change the dynamic I’ve outlined. You have to give people a reason to read first, and then put more nuances into the text. Idk if this helps but I’ve learned this lesson the hard way by spending a ridiculous amount of time on a huge post that was almost entirely ignored (this was several years ago).
(It seems like you got some reactions now fwiw, hope this may make you reconsider leaving.)
I think you probably don’t have the right model of what motivated the reception. “AGI will lead to human extinction and will be built because of capitalism” seems to me like a pretty mainstream position on LessWrong. In fact I strongly suspect this is exactly what Eliezer Yudkowsky believes. The extinction part has been well-articulated, and the capitalism part is what I would have assumed is the unspoken background assumption. Like, yeah, if we didn’t have a capitalist system, then the entire point about profit motives, pride, and race dynamics wouldn’t apply. So… yeah, I don’t think this idea is very controversial on LW (reddit is a different story).
I think the reason that your posts got rejected is that the focus doesn’t seem useful. Getting rid of capitalism isn’t tractable, so what is gained by focusing on this part of the causal chain? I think that’s the part your missing. And because this site is very anti-[political content], you need a very good reason to focus on politics. So I’d guess that what happened is that people saw the argument, thought it was political and not-useful, and consequently downvoted.
Sorry, but isn’t this written by an LLM? Especially since milan’s other comments ([1], [2], [3]) are clearly in a different style, the emotional component goes from 9⁄10 to 0⁄10 with no middle ground.
I find this extremely offensive (and I’m kinda hard to offend I think), especially since I’ve ‘cooperated’ with milan’s wish to point to specific sections in the other comment. LLMs in posts is one thing, but in comments, yuck. It’s like, you’re not worthy of me even taking the time to respond to you.
The guidelines don’t differentiate between posts and comments but this violates them regardless (and actually the post does as well) since it very much does have the stereotypical writing style of an AI assistant, and the comment also seems copy-pasted without a human element at all.
A rough guideline is that if you are using AI for writing assistance, you should spend a minimum of 1 minute per 50 words (enough to read the content several times and perform significant edits), you should not include any information that you can’t verify, haven’t verified, or don’t understand, and you should not use the stereotypical writing style of an AI assistant.
The sentence you quoted is a typo, it’s is meant to say that formal languages are extremely impractical.
Here’s one section that strikes me as very bad
At its heart, we face a dilemma that captures the paradox of a universe so intricately composed, so profoundly mesmerizing, that the very medium on which its poem is written—matter itself—appears to have absorbed the essence of the verse it bears. And that poem, unmistakably, is you—or more precisely, every version of you that has ever been, or ever will be.
I know what this is trying to do but invoking mythical language when discussing consciousness is very bad practice since it appeals to an emotional response. Also it’s hard to read.
Similar things are true for lots of other sections here, very unnecessarily poetic language. I guess you can say that this is policing tone, but I think it’s valid to police tone if the tone is manipulative (on top of just making it harder and more time intensive to read.
Since you asked for a section that’s explicitly nonsense rather than just bad, I think this one deserves the label:
We can encode mathematical truths into natural language, yet we cannot fully encode human concepts—such as irony, ambiguity, or emotional nuance—into formal language. Therefore: Natural language is at least as expressive as formal language.
First of all, if you can’t encode something, it could just be that the thing is not well-defined, rather than that the system is insufficiently powerful
Second, the way this is written (unless the claim is further justified elsewhere) implies that the inability to encode human concepts in formal languages is self-evident, presumably because no one has managed it so far. This is completely untrue; formal[^1] languages are extremely impractical, which is why mathematicians don’t write any real proofs in them. If a human concept like irony could be encoded, it would be extremely long and way way beyond the ability of any human to write down. So even if it were theoretically possible, we almost certainly wouldn’t have done it yet, which means that it not having been done yet is negligible evidence of it being impossible.
[1]: typo corrected from “natural”
I agree that this sounds not very valuable; sounds like a repackaging of illusionism without adding anything. I’m surprised about the votes (didn’t vote myself).
Another quick way to get at my skepticism for LLM capability: the ability do differentiate good from bad ideas. ImE the only way LLMs can do this (or at least seem to do this) is the prevailing views are different depending on context—like if there’s an answer popular among lay people and a better answer popular among academics, and your tone makes it clear that you belong in the second category, then it will give you the second one, and that can make it seem like it can tell good and bad takes apart. But this clearly doesn’t count. If the most popular view is the same across contexts it’ll echo that view no matter how dumb it is. I’d guess most people here have rolled their eyes before at GPT regurgitating bad popular takes.
If there’s an update to LLMs where you can ask it a question and it’ll just outright tell you that the prevailing view on this is dumb (and it is in fact dumb) then I shall be properly terrified. Of course this is another non-objective benchmark, but as I said in my post, I think being not-objective may just be a property that all the actually important milestones have.