What if you believe in DKRUDT, the “Dick Kick’em rewards you” decision theory?
Seriously though, Newcomb’s setup is not adversarial in the same way, the predictor rewards or punishes you for actions, not beliefs. Your internal reasoning does not matter, as long as you end up one-boxing you walk away with more money.
As someone who gets rate-limited due to downvoted comments occasionally, I can see a rather strong correlation between the style/tone and the response. It is very much possible to express controversial, contrarian or even outright silly views without being downvoted. The rule of thumb is to be respectful and charitable to your opponents and, well, read the room. The more your view diverges from the local mainstream, the better, tighter and clearer your argument must be.
It is standard practice when writing scientific papers to first demonstrate intimate familiarity with the views you argue against later. This is what you would want from someone else writing a refutation of your argument, right? So give them the same courtesy. You can cut corners if your post or comment is close to the mainstream, but it is not a great habit. If you want to learn how to do it right, read ACX, Scott is a master of this art.
it is a common pitfall to blame the society/subculture for the negative reaction you get. It is almost never a constructive way to proceed.
If you are in doubt as to how your post would be received, and you want to get through to more people, consider reaching out to someone familiar with this site to review your draft. Or to anyone, really. The bar for writing a post or a comment with non-negative expected karma is pretty low.
it’s not a probability question. Blood tests exist.
Your potential contribution to timeline shortening as an undergrad is probably negligible on the margins, unless you are in the top 1% of all applicants, or even then. The field is crowded by very intelligent, well paid and very determined people who do this for a living. So whatever considerations you have, they need not be influenced by the AGI killeveryoneism.
That’s a fair point, different directions have different landmines. Culture war issues are tempting and most people seem to have strong yet poorly informed opinions that are not obviously poorly informed. I think that the original collection by Rob Bensinger The Library of Scott Alexandria is really good to start with, it is very light on political and culture war topics and grasps the essence of rational thinking, without going into the esoteric and irrelevant topics like quantum mechanics.
I find it hard to not assign whatever issues or problems we have to how the Universe works.
Indeed, the Universe just is, everything else is an emergent concept for the tiny embedded agents in it (bacteria, ants and humans). That includes sugar gradient, scent of food and laws of physics, respectively.
One of these emergent concepts is personal responsibility. Societies that do not have it do not last as long.
Step away from this place for a time and read Scott Alexander first. https://www.lesswrong.com/posts/vwqLfDfsHmiavFAGP/the-library-of-scott-alexandria is a good start, then continue on his blogs directly. He links back to this site where appropriate. You might get inured to some of the common pitfalls newbies here are prone to.
I agree with your analysis of the current situation. However, the technological issues arise when trying to correct it without severe unintended consequences, and that is not related to profit. You can’t transplant a house easily. You cannot easily feed only those who go hungry without affecting the economy (food banks help to some degree). There are people in need of companionship that cannot find it, even though there is a companion that would match somewhere out there. There are potential technological solutions to all those that are way outside our abilities (teleportation! replication! telepathy!) that would solve these issues. You can also probably find a few examples where what looks like profit-based incentive is in fact a technological deficiency.
I guess my point is the standard one: in many ways even poor people live a lot better now than royalty 300 years ago.
I’d assume that people themselves would define what they need, within the limits of what is possible given the technology of the time.
You might be thinking about it in a wrong way. Societal structures follow capabilities, not wants. If you try to push for “each person works and is paid according to their abilities and needs” too early, you end up with communist dystopias. If we are lucky, the AGI age will improve our capabilities enough where “to everyone according to their needs” may become feasible, aligning the incentives with well-being rather than with profit. So, to answer your questions:
It is currently impossible to “align the incentives” without causing widespread suffering.
It is undesirable if you do not want to cause suffering.
It is ineffective to try to align the incentives away from profit if your goal is making them aligned with “human well being”.
That said, there are incremental steps that are possible to take without making things worse, and they are discussed quite often by Scott Alexander and Zvi, as well as by others in the rationalist diaspora. So read them.
There is no a priori reason to believe that world has to be learnable. But if it were not, then we wouldn’t exist, nor would (most?) animals. The existing world, thus, is learnable. The human sensorium and motor system are necessarily adapted to that learnable structure, whatever it is.
I have a post or a post draft somewhere discussing this issue. The world indeed just is. It does not have to be internally predictable to an arbitrary degree of accuracy, but it needs to be somewhat internally predictable in order for the long-lived patterns we identify as “agents” or “life” to exist. Internal predictability (i.e. that an incredibly tiny part of the universe that is a human (or a bacterium) can infer enough about the world to not immediately poof away) is not something that should be a given in general. Further, even if some coarse-grained internal predictability can be found in such a world, there is no guaranteed that it can be extended to arbitrarily fine accuracy. it might well be the case in our world that at some point we hit the limit of internal predictability and from then on things will just look random for us. Who knows, maybe we hit it already, and the outstanding issues in the Standard Model of Cosmology and/or the Standard Model of Particle Physic, and/or maybe some of the Millennium prize problems, and/or the nature of consciousness are simply unknowable. I hope this is not the case, but I do not see any good argument that says “yep, we can push much further”, other than “it worked so far, if in fits and starts”.
I guess that is one way to say it. But the statement is stronger than that, I think. They do not care about the box or about anything else. They react to stimuli, then go silent again.
Yeah, I get that. But to look into the future one must take stock of the past and present and reevaluate models that gave wrong predictions. I am yet to see this happening.
I do not know if we will or will not build something recognizable agentic any time soon. I am simply pointing out that currently there is a sizable gap that people did not predict back then. Given that we still have no good model what constitutes values or drives (definitely not a utility function, since LLMs have plenty of that), I am very much uncertain about the future, and I would hesitate to unequivocally state that “AGI isn’t just a technology”. So far it most definitely is “just a technology”, despite the original expectations to the contrary by the alignment people.
Maybe we understand agency differently. You give LLMs tools to use, they will use it. But there is no discernable drive or “want” to change the world to their liking. Not saying that it won’t show up some day, it’s just conspicuously lagging the capabilities.
I’ve been following the “tools want to become agents” argument since Holden Karnofsky raised the topic a long time ago, and I was almost convinced by the logic, but the LLMs show a very surprising lack of agency, and, as far as I can tell, this gap between apparent intelligence and apparent agency was never predicted or expected by the alignment theorists. I would trust their cautions more if they had a model that makes good predictions.
This seems to assume that AI has its own drives and values. So far this has not been the case.
Interesting! It seems like something like that should be a canonical reference for “let’s enter a problem” e.g. smoking lesion, then “select decision theory”, and out pops the answer. Of course, formalizing the problem seems like the hard part.
How well can you predict what will be good in 100 years? For perspective, given the contemporary zeitgeist of 1923, rather than knowing all we know now, what would have been considered a great policy to enact then that would take effect in 2023?