I sometimes wonder about this. This post does pose the question, but I don’t think it gives an analysis that could make me change my mind on anything, it’s too shallow and not adversarial.
rotatingpaguro
Karma: 623
I read part of the paper. That there’s a cultural difference north-south about honesty and willingness to break the rules matches my experience on the ground.
I find this intellectually stimulating, but it does not look useful in practice, because with repeated i.i.d. data the information in the data is much higher than the prior if the prior is diffuse/universal/ignorance.
Italians over time sorted themselves geographically by honesty, which is both weird and damn cool, and also makes a lot of sense. There are multiple equilibria, so let everyone find the one that suits them. We need to use this more in logic puzzles. In one Italian villa everyone tells the truth, in the other…
I can’t get access to the paper, anyone has a tip on this?
I agree with whay you say about how to maximize what you get out of an interview. I also agree about that discussion vs. debate distinction you make, and I wasn’t specifically trying to go there when I used the word “debate”, I was just sloppy with words.
I guess you agree that it is friction to create a social norm that you should do a read up of the other person material before engaging in public. I expect less discussions would happen. There is not a clear threshold at how much you should be prepared.
I guess we disagree about how much value do we lose due to eliminating discussions that could have happaned, vs. how much value we gain by eliminating some lower quality discussions.
Another angle I have in mind that sidesteps this direct compromise, is that maybe what we value out of such discussions is not just doing an optimal play in terms of information transmitted between the parties. A public discussion has many different viewers. In the case at hand, I expect many people get more out of the discussion if they can see Wolfram think through the thing for the first time in real time, rather than having two informed people start discussing finer points in medias res.
I see your proposed condition for meaningful debate as bureaucracy that adds friction rather than value.
I somewhat disagree with Tenobrus’ commentary about Wolfram.
I watched the full podcast, and my impression was that Wolfram uses a “scientific hat”, of which he is well aware of, which comes with a certain ritual and method for looking at new things and learning them. Wolfram is doing the ritual of understanding what Yudkowsky says, which involves picking at the details of everything.
Wolfram often recognizes that maybe he feels like agreeing with something, but “scientifically” he has a duty to pick it apart. I think this has to be understood as a learning process rather than as a state of belief.
So, should the restrictions on gambling be based on feedback loop length? Should sport betting be broadly legal when about the far enough future?
current inference scaling methods tend to be tied to CoT and the like, which are quite transparent
Aschenbrenner in Situational Awareness predicts illegible chains of thought are going to prevail because they are more efficient. I know of one developer claiming to do this (https://platonicresearch.com/) but I guess there must be many.
Related, I have a vague understanding on how product safety certification works in EU, and there are multiple private companies doing the certification in every state.
Half-informed take on “the SNPs explain a small part of the genetic variance”: maybe the regression methods are bad?
Not sure if I missed something because I read quickly, but: all these are purely correlational studies, without causal inference, right?
OpenAI is recklessly scaling AI. Besides accelerating “progress” toward mass extinction, it causes increasing harms. Many communities are now speaking up. In my circles only, I count seven new books critiquing AI corps. It’s what happens when you scrape everyone’s personal data to train inscrutable models (computed by polluting data centers) used to cheaply automate out professionals and spread disinformation and deepfakes.
Could you justify that it causes increasing harms? My intuition is that OpenAI is currently net-positive without taking into account future risks. It’s just an intuition, however, I have not spent time thinking about it and writing down numbers.
(I agree it’s net-negative overall.)
Ok, that. China seems less interventionist, and to use more soft power. The US is more willing to go to war. But is that because the US is more powerful than China, or because Chinese culture is intrinsically more peaceful? If China made the killer robots first, would they say “MUA-HA-HA actually we always wanted to shoot people for no good reason like in yankee movies! Go and kill!”
Since politics is a default-no on lesswrong, I’ll try to muddle the waters by making a distracting unserious figurative narration.
Americans maybe have more of a culture of “if I die in a shooting conflict, I die honorably, guns for everyone”. Instead China is more about harmony&homogenity, “The CCP is proud to announce that in 2025 the Harmonious Agreement Quinquennal Plan in concluded successfully; all disagreements are no more, and everyone is officially friends”. When the Chinese send Uighurs to the adult equivalent of school, Americans freak out: “What? Mandated school? Without the option of shooting back?”
My doubt is mostly contingent on not having first-hand experience of China, while I have of the US. I really don’t trust narratives from outside. In particular I don’t trust narratives from Americans right now! My own impression of the US changed substantially by going there in person, and I even am from an allied country with broad US cultural influence.
[Alert: political content]
About the US vs. China argument: have any proponent made a case that the Americans are the good guys here?
My vague perspective as someone not in China neither in the US, is that the US is overall more violent and reckless than China. My personal cultural preference is for US, but when I think about the future of humanity, I try to set aside what I like for myself.
So far the US is screaming “US or China!” while creating the problem in the first place all along. It could be true that if China developed AGI it would be worse, but that should be argued.
I bet there is some more serious non-selfish analysis of why China developing AGI is worse than US developing AGI, I just have never encountered it, would be glad if someone surfaced it to me.
I agree it’s not a flaw in the grand scheme of things. It’s a flaw for using it for consensus for reasoning.
I start with a very low prior of AGI doom (for the purpose of this discussion, assume I defer to consensus).
You link to a prediction market (Manifold’s “Will AI wipe out humanity before the year 2100”, curretly at 13%).
Problems I see with using it for this question, in random order:
It ends in 2100 so the incentive is effectively about what people will believe a few years from now, not about the question. It is a Keynesian beauty contest. (Better than nothing.)
Even with the stated question, you win only if it resolves NO, so it is strategically correct to bet NO.
It is dynamically inconsistent, if you think that humans have power over the outcome and that such markets influence what humans do about it. Illustrative story: “The market says P(doom)=1%, ok I can relax and not work on AI safety” ⇒ everyone says that ⇒ the market says P(doom)=99% because no AI safety work ⇒ “AAAAH SOMEONE DO SOMETHING” ⇒ marker P(doom)=1% ⇒ …
This type of issue is a huge effective blocker for people with my level of skills. I find myself excited to write actual code that does the things, but the thought of having to set everything up to get to that point fills with dread – I just know that the AI is going to get something stupid wrong, and everything’s going to be screwed up, and it’s going to be hours trying to figure it out and so on, and maybe I’ll just work on something else. Sigh. At some point I need to power through.
Reminds me of this 2009 kalzumeus quote:
I want to quote a real customer of mine, who captures the B2C mindset about installing software very eloquently: “Before I download yet another program to my poor old computer, could you let me know if I can…” Painful experience has taught this woman that downloading software to her computer is a risky activity. Your website, in addition to making this process totally painless, needs to establish to her up-front the benefits of using your software and the safety of doing so. (Communicating safety could be an entire article in itself.)
Ah, sorry for being so cursory.
A common trope about mathematicians vs. other math users is that mathematicians are paranoid persnickety truth-seekers, they want everything to be exactly correct down to every detail. Thus engineers and physicists often perceive mathematicians as a sort of fact-checker caste.
As you say, in some sense mathematicians deal with made-up stuff and engineers with real stuff. But from the engineer’s point of view, they deal with mathematicians when writing math, not when screwing bolts, and so perceive mathematicians as “the annoying people who want everything to be perfectly correct”.
Example: I write “E[E[X|Y]] = E[X]” in a paper, and the mathematician pops up complaining “What’s the measure space? Is it sigma-finite? You have to declare if your random variables are square-integrable. Are X and Y measureable in the same space?” and my reply would be “come on we know it’s true I don’t care about writing it properly”.So to me and many people in STEM your analogy has the opposite vibe, which defeats the purpose of an analogy.
I think there’s one important factor missing: if you really used evals for regulation, then they would be gamed. I trust more the eval when the company is not actually at stake on it. If it was, there would be a natural tendence for evals to slide towards empty box-checking.