JBlack

Karma: 2,242

JBlack Apr 23, 2025, 1:28 AM
3 points
0
on: Intuition in AI
Yet our AI systems, even the most advanced, focus almost exclusively on logical, step-by-step reasoning.
This is absolutely false.
We design them to explain every decision, show their work and follow clear patterns of deduction.
We are trying to design them to be able to explain their decisions and follow clear patterns of deduction, but we are still largely failing. In practice they often arrive at an answer in a flash (whether correct or incorrect), and this was almost universal for earlier models without the more recent development of “chain of thought”.
Even in “reasoning” models there is plenty of evidence that they often still do have an answer largely determined before starting any “chain of thought” tokens and then make up reasons for it, sometimes including lies.

JBlack Apr 23, 2025, 1:05 AM
4 points
2
on: Experimental testing: can I treat myself as a random sample?
Yes, you can use yourself as a random sample but at best only within a reference class of “people who use themselves as a random sample for this question in a sufficiently similar context to you”. That might be a population of 1.
For example, suppose someone without symptoms has just found out that they have genes for a disease that always progresses to serious illness. They have a mathematics degree and want to use their statistical knowledge to estimate how long they have before becoming debilitated.
They are not a random sample from the reference class of people who have these genes. They are from people who have the genes and didn’t show symptoms before finding that out and did so during adulthood (almost certainly) and live in a time and place and with sufficient capacity to earn a mathematics degree and of suitable mindset to ask themselves this question and so on.
Any of these may be relevant information for estimating the distribution, especially if the usual age of onset is in childhood or the disease also reduces intellectual capacity or affects personality in general.
Relating back to the original doomsday problem: suppose that in the reference class of all civilizations, most discover some principle that conclusively resolves the Doomsday problem not long after formulating it (within a few hundred years or so). It doesn’t really matter what that resolution happens to be, there are plenty of possibilities.
If that is the case, then most people who even bother to ask the Doomsday question without already knowing the answer are those in that narrow window of time where their civilization is sophisticated enough to ask the question without being sophisticated enough to answer it, regardless of how long those civilizations might last or how many people exist after resolving the question.
To the extent that the Doomsday reasoning is valid at all (which it may not be), all that it provides is an estimate of time until most people stop asking the Doomsday question in a similar context to yours. Destruction of the species is not required for that. Even it becoming unfashionable is enough.

JBlack Apr 20, 2025, 8:54 AM
2 points
0
in reply to: Forged Invariant’s comment on: strawberry calm’s Shortform
Yes, player 2 loses with extremely low probability even for a 1-bit hash (on the order of 2^-256). For a more commonly used hash, or for 2^24 searches on their second-last move, they reduce their probability of loss by a huge factor more.

JBlack Apr 13, 2025, 1:22 AM
1 point
0
in reply to: Czynski’s comment on: Czynski’s Shortform
This paragraph also misses the possibility of constructing a LLM and/or training methodology such that it will learn certain functions, or can’t learn certain functions. There is also a conflation of “reliable” with “provable” on top of that.
Perhaps there is some provision made elsewhere in the text that addresses these objections. Nonetheless, I am not going to search. I found that the abstract smells enough like bullshit to do something else.

JBlack Apr 13, 2025, 1:14 AM
2 points
0
in reply to: Tapatakt’s comment on: Weird Random Newcomb Problem
I’ll try to make it clearer:
Suppose b “knows” that Omega runs this experiment for all programs b. Then the optimal behaviour for a competent b (by a ridiculously small margin) is to 1-box.
Suppose b suspects that box-choosing programs are slightly less likely to be run if they 1-box on equal inputs. Then the optimal behaviour for b is to 2-box, because the average extra payoff for 1-boxing on equal inputs is utterly insignificant while the average penalty for not being chosen to run is very much greater. Anything that affects probability of being run as box-chooser with probability greater than 1000/|P| (which is on the order of 1/10^10^10^10^100) matters far more than what the program actually does.
In the original Newcombe problem, you know that you are going to get money based on your decision. In this problem, a running program does not know this. It doesn’t know whether it’s a or b or both, and every method for selecting a box-chooser is a different problem with different optimal strategies.

JBlack Apr 12, 2025, 6:19 AM
3 points
0
on: Weird Random Newcomb Problem
As a function of M, |P| is very likely to be exponential and so it will take O(M) symbols to specify a member of P. Under many encodings, there isn’t one that can even check whether the inputs are equal before running out of time.
That aside, why are you assuming that program b “wants” anything? Essentially all of P won’t be programs that have any sort of “want”. If it is a precondition of the problem that b is such a program, what selection procedure is assumed between those that do “want” money from this scenario? Note that being selected for running is also a precondition for getting any money at all, so this selection procedure is critically important—far more so than anything the program might output!

JBlack Apr 9, 2025, 8:25 AM
5 points
3
in reply to: Akira Pyinya’s comment on: (Some) Humans do choose 5 dollars in the 5-and-10 problem
That is nothing like the 5-and-10 problem. I am no longer interested in what you consider to be evidence.

JBlack Apr 8, 2025, 3:21 AM
2 points
0
on: (Some) Humans do choose 5 dollars in the 5-and-10 problem
Evidence for the claim in the title? Or for anything else in the post?

JBlack Apr 7, 2025, 1:13 AM
5 points
2
on: Introduction to Representing Sentences as Logical Statements
It’s interesting (and perhaps a bit sad) that a relatively lengthy post on representing sentences as logical statements doesn’t make any reference to the constructed language Lojban in which the entire grammar and semantics is designed around expressing sentences as logical statements.

JBlack Apr 2, 2025, 2:01 AM
2 points
0
in reply to: Greenless Mirror’s comment on: A Fraction of Global Market Capitalization as the Best Currency
Going into all the ways in which civilization—and its markets—fails to be rational seems way beyond the scope of a few comments. I will just say that GDP does absolutely fail to capture a huge range of value.
However, to address “share prices are set by the latest trade” you need to consider why a trade is made. In principle, prices are based on the value to the participants, somewhere between the value to the buyer and value to the seller. A seller who needs cash soon (to meet some other obligation or opportunity) may accept a lower price to attract a buyer more quickly. In our hypothetical and simplified one-trade-per-day scenario, that seller may accept up to 20% less than the previous day’s trade price, though they find a buyer at only 5% less. So the company’s market cap drops 5% even though 99.9999% of the investors and potential investors still value it exactly the same as yesterday.
This scales up since there are many highly correlated and often very short-term factors that influence desirability of shares vs cash vs bonds vs commodities vs …etc. It’s not just “what do I think this is worth to me”, but also “what do I think that other people think that the market price will be tomorrow” and so on, and this can result in self-fulfilling predictions over surprisingly long time spans.

JBlack Apr 1, 2025, 1:28 AM
3 points
0
on: Why does Claude Speak Byzantine Music Notation?
Could it be generalizing from T E X T L I K E T H I S and/or mojibake UTF-16 interpreted as UTF-8 with every second character being zero? It’s still a bit more of a stretch from there to generalize to ignoring two intervening constant characters, though.

JBlack Apr 1, 2025, 12:53 AM
2 points
0
in reply to: Greenless Mirror’s comment on: A Fraction of Global Market Capitalization as the Best Currency
Market cap is a marginal measure of desirability of shares in the entity represented. It mostly measures the expectations of the most flighty investors over short timescales. If a company issues a billion shares but only one of those is traded in any given day, the price of that single share agreed between the single seller and the single buyer entirely determines the market capitalization of that company.
In practice there is usually a lot more volume, but the principle remains. Almost all shares of any given entity are not traded over the timescales that determine share (or index) prices and hence market capitalization. In addition, market price has very little to do with the value of an entity’s assets. Such assets may be instrumental in generating profits, but the relation is weak and very far from linear.
GDP, too, does not measure what you appear to think it measures.

JBlack Mar 25, 2025, 1:42 AM
72 points
58
on: Policy for LLM Writing on LessWrong
I was very interested to see the section “Posts by AI Agents”, as the first policy I’ve seen anywhere acknowledging that AI agents may be both capable of reading the content of policy terms and acting based on them.

JBlack Mar 5, 2025, 3:11 AM
2 points
0
in reply to: Davey Morse’s comment on: Davey Morse’s Shortform
Why not both?
Human design will determine the course of AGI development, and if we do the right things then whether it goes well is fully and completely up to us. Naturally at the moment we don’t know what the right things are or even how to find them.
If we don’t do the right things (as seems likely), then the kinds of AGI which survive will be the kind which evolve to survive. That’s still largely up to us at first, but increasingly less up to us.

JBlack Feb 26, 2025, 1:35 AM
3 points
1
in reply to: Daniel V’s comment on: Economics Roundup #5
The fun thing is that the actual profile of wages earned can be absolutely identical and yet end up with incredibly different results for personal wage changes. For example:
In year 1, A earns $1/hr, B $2, C $3, D $4, and E $5.
In year 2, A earns $2/hr, B $3, C $4, D $5, and E $1.
A, B, C, and D personally all increased their income by substantial amounts and may vote accordingly. E lost a lot more than any of the others gained, but doesn’t get more votes because of that. 80% of voters saw their income increase. What’s more, this process can repeat endlessly.
If in year 2, A instead earns $5/hr, B $1, C $2, D $3, and E $4 then 80% of voters will be rather unhappy at the change despite the income distribution still being identical.

JBlack Feb 25, 2025, 3:17 AM
3 points
0
on: Local Trust
In the rain forecaster example, it appears that the agent (“you”) is more of an expert on Alice’s calibration than Alice is. Is this intended?

JBlack Feb 18, 2025, 2:26 AM
13 points
5
on: Born on Third Base: The Case for Inheriting Nothing and Building Everything
In practice, a lot of property is transferred into family trusts, and appointed family members exercise decision making over those assets according to the rules of that trust. A 100% death tax would simply ensure that essentially all property is managed in this manner for the adequately wealthy, and only impact families too disadvantaged to use this sort of structure. If you don’t personally own anything of note at the time of your death, your taxes will be minimal.
You would also need a 100% gift tax, essentially prohibiting all gifts between private citizens. You bought your child something, or (worse!) gave them money to buy it themselves? That’s clearly an attempt to get around inheritance tax and must be prevented.
There are also huge numbers of private businesses, for which this sort of tax would be nothing but an enormous micromanaging nationalization scheme with predictable disastrous results.
This does not work, at all.

JBlack 17 Feb 2025 2:01 UTC
3 points
0
in reply to: dx26’s comment on: dx26′s Shortform
I think one argument is that optimizing for IGF basically gives humans two jobs: survive, and have kids.
Animal skulls are evidence that the “survive” part can be difficult. We’ve nailed that one, though. Very few humans in developed countries die before reaching an age suitable for having kids. I doubt that there are any other animal species that come close to us in that metric. Almost all of us have “don’t die” ingrained pretty deeply.
It’s looking like we are moving toward failing pretty heavily on the second “have kids” job though, and you would think that would be the easier one.
So if there’s a 50% failure rate on preserving outer optimizer values within the inner optimizer, that’s actually pretty terrible.

JBlack 14 Feb 2025 2:33 UTC
2 points
0
in reply to: Anthony DiGiovanni’s comment on: Notes on Occam via Solomonoff vs. hierarchical Bayes
It doesn’t completely avoid the problem of priors, just the problem of arbitrarily fixing a specific type of update rule on fixed priors such as in Solomonoff induction. You can’t afford this if you’re a bounded agent, and a Solomonoff inductor can only get away with it since it has not just unbounded resources but actually infinite computational power in any given time period.
A bounded agent needs needs to be able to evaluate alternative priors, update rules, and heuristics in addition to the evidence and predictions themselves, or it won’t even approximate bounded rationality. While this is a more complicated scenario than the Solomonoff updater in some senses, it is philosophically simpler since we can view it more like a “bootstrap” process and ask what sort of bootstrapping might “generally” do well rather than taking anything as fixed.
I suspect that heuristics that score highly involve “universal” but finite systems (such as Turing machines or other mathematical structures capable of representing their own rules), and a “simple and not too costly” evaluation heuristic (not just simplicity).
There would be “degenerate” distributions of universe rules that would be exceptions, so there is still a problem of describing what sort of distributions I’m thinking of as being “degenerate”, and naturally this whole sort of statement is too vague to prove any such thing even if such proofs weren’t famously difficult (and plausibly impossible to prove even if not false).

JBlack 13 Feb 2025 1:59 UTC
2 points
0
on: Notes on Occam via Solomonoff vs. hierarchical Bayes
One thing that seems worth exploring from a conceptual point of view is doing away with priors altogether, and working more directly with metrics such as “what are the most expected-value rewarding actions that a bounded agent can make given the evidence so far”. I suspect that from this point of view it doesn’t much matter whether you use a computational basis such as Turing machines, something more abstract, or even something more concrete such as energy required to assemble and run a predicting machine.
From a computing point of view not all simple models will require the fewest resources available to actually follow it through and compare against other models, so in that respect this differs from Solomonoff induction (which assumes a system of infinite computing power). However, my expectation is that there usually would be some model only a bit more complex than the simplest that makes similar predictions at lower cost. A heuristic that examines models in a simplest-first order (but discarding ones that look expensive to evaluate) may well end up being close to optimal in trading off prediction accuracy against whatever costs there are of evaluating multiple models. There are exponentially fewer simple models to evaluate.