Forged Invariant

Karma: 100

Forged Invariant 20 Jan 2025 2:10 UTC
14 points
14
in reply to: No77e’s comment on: meemi’s Shortform
In addition to the object level reasons mentioned by plex, misleading people about the nature of a benchmark is a problem because it is dishonest. Having an agreement to keep this secret indicates that the deception was more likely intentional on OpenAI’s part.

Forged Invariant 4 Jan 2025 5:24 UTC
3 points
4
on: Preference Inversion
Based on the quote from Kirkpatrick, It looks like a clear example of preference falsification, but I do not see any reason to believe that it is internalized preference falsification. Did I miss how the submissive apes were internalizing the preference to not mate? The sentence “This is an easy to understand example of an important general fact about humans: we can be threatened into internalized preference falsification, i.e. preference inversion.” makes me think that you intended it as an example of primates internalizing a preference falsification. It feels like the quote is only evidence that primates will be deceptive.
On rereading, the claim that “This is an easy to understand example of an important general fact about humans: we can be threatened into internalized preference falsification, i.e. preference inversion” seems reasonably supported by the next paragraph about male vs female arousal in humans. Maybe I just attached the claim to the wrong evidence.

Forged Invariant 9 Jun 2023 16:24 UTC
2 points
0
in reply to: Garrett Baker’s comment on: The Base Rate Times, news through prediction markets
As an example of how Manifold reacted to a (crude) attempt at manipulation:
Dr P (a Manifold user) would create and bet yes on markets for “Will Trump be president on [some date]?” for various dates where there was no plausible way trump would be president. Other users quickly noticed and set up limit orders to capture this source of free money. Eventually Dr. P’s bets were cancelled out quickly enough that they had little to no effect on the probability, and it became hard to find one of those bets profit from. Eventually Dr P gave up and their account became inactive. (There was some uncertainty about what would happen if Dr P misresolved the markets. Today I would expect false resolutions to be reversed. Various derivative/insurance markets were set up.)

Forged Invariant 9 Jun 2023 16:07 UTC
3 points
0
in reply to: Sune’s comment on: The Base Rate Times, news through prediction markets
One thing that I have seen on manifold is markets that will resolve at a random time, with a distribution such that at any time, their expected duration (from the current day, conditional on not having already resolved) is 6 months. They do not seem particularly common, and are not quite equivalent to a market with a deadline exactly 6 months in the future. (I can’t seem to find the market.)

Forged Invariant 2 Oct 2022 20:42 UTC
2 points
1
in reply to: Martin Randall’s comment on: Petrov Day Retrospective: 2022
The timing evidence is thus hostile evidence and updating on it correctly requires superintelligence.
What do you mean by this? It seems trivially false that updating on hostile evidence requires superintelligence; for example poker players will still use their opponent’s bets as evidence about their cards, even though these bets are frequently trying to mislead them in some way.
The evidence being from someone who went against the collective desire does mean that confidently taking it at face value is incorrect, but not that we can’t update on it.

Forged Invariant 24 Sep 2022 6:21 UTC
27 points
18
in reply to: Ruby’s comment on: LW Petrov Day 2022 (Monday, 9/26)
The LW staff are necessary to take down the site. If we assume that there are multiple users that are willing to press the button, then the (shapely-attributed) blame for taking the site down mostly falls on the LW staff, rather than whoever happens to press the button first.
According to http://shapleyvalue.com/?example=8 if there were 6 people who were willing to push the button, the LW team would deserve 85% of the blame. (Here I am considering the people who take actions that act to facilitate bringing down the site as part of the coalition.)
I am not quite sure how to take into account all the people who choose not to take down the website and thus delay, and there is some value in running the Petrov day event, so the above does not take everything into account.
Tweaking some values in the website to model this, where value = 7 if either LW and/or all the other users refuse to shut down the site, and 7-i where i is the highest numbered player that shuts down the site (higher meaning they shut things down sooner), I get these values:
The Shapley value of player 1(Low Karma button pusher) is: −0.023809523809524
The Shapley value of player 2 is: −0.057142857142857
The Shapley value of player 3 is: −0.10714285714286
The Shapley value of player 4 is: −0.19047619047619
The Shapley value of player 5 is: −0.35714285714286
The Shapley value of player 6(High karma button pusher) is: −0.85714285714286
The Shapley value of player 7(LW team) is: −4.4071428571429
(All the values are negative, since this assigns no value to running the experiment or to keeping the site online despite running the experiment and for simplicity’s sake measures things in site uptime, and not shutting down the site achieves that.)

Forged Invariant 21 Sep 2022 4:23 UTC
12 points
4
in reply to: YimbyGeorge’s comment on: Gene drives: why the wait?
Here is an example of something that comes close from “The Selfish Gene”:
One of the best-known segregation distorters is the so-called t gene in mice. When a mouse has two t genes it either dies young or is sterile, t is therefore said to be lethal in the homozygous state. If a male mouse has only one t gene it will be a normal, healthy mouse except in one remarkable respect. If you examine such a male’s sperms you will find that up to 95 per cent of them contain the t gene, only 5 per cent the normal allele. This is obviously a gross distortion of the 50 per cent ratio that we expect. Whenever, in a wild population, a t allele happens to arise by mutation, it immediately spreads like a brushfire. How could it not, when it has such a huge unfair advantage in the meiotic lottery? It spreads so fast that, pretty soon, large numbers of individuals in the population inherit the t gene in double dose (that is, from both their parents). These individuals die or are sterile, and before long the whole local population is likely to be driven extinct. There is some evidence that wild populations of mice have, in the past, gone extinct through epidemics of t genes.
Not all segregation distorters have such destructive side-effects as t. Nevertheless, most of them have at least some adverse consequences.
From the discussion of human-engineered gene drives, they would only cause sterility in one sex, which would help avoid the gene dying off as quickly.

Forged Invariant 15 Jun 2022 5:18 UTC
5 points
0
on: Why all the fuss about recursive self-improvement?
I had not thought of self-play as a form of recursive self-improvement, but now that you point it out, it seems like a great fit. Thank you.
I had been assuming (without articulating the assumption) that any recursive self improvement would be improving things at an architectural level, and rather complex (I had pondered improvement of modular components, but the idea was still to improve the whole model). After your example, this assumption seems obviously incorrect.
Alpha-go was improving its training environment, but not any other part of the training process.

Forged Invariant 25 Nov 2021 4:23 UTC
1 point
in reply to: Jsevillamol’s comment on: A Bayesian Aggregation Paradox
The left hand side of the example is deliberately making the mistake described in your article, as a way to build intuition on why it is a mistake.
(Adding instead of averaging in the update summaries was an unintended mistake)
Thanks for explaining how to summarize updates, it took me a bit to see why averaging works.

Forged Invariant 23 Nov 2021 4:28 UTC
6 points
on: A Bayesian Aggregation Paradox
Seeing the equations, it was hard to intuitively grasp why updates work this way. This example made things more intuitive for me:
If an event can have 3 outcomes, and we encounter strong evidence against outcomes B and C, then the update looks like this:
$⎛ ⎜ ⎝ \begin{matrix} 111 \end{matrix} ⎞ ⎟ ⎠      Prior \times ⎛ ⎜ ⎝ \begin{matrix} 1 0.01 1 \end{matrix} ⎞ ⎟ ⎠      Refute B \times ⎛ ⎜ ⎝ \begin{matrix} 11 0.01 \end{matrix} ⎞ ⎟ ⎠      Refute C \neq (\begin{matrix} 12 \end{matrix})      Pooled prior \times (\begin{matrix} 1 1.01 \end{matrix})      Refute B \times (\begin{matrix} 1 1.01 \end{matrix})      Refute C$
The information about what hypotheses are in the running is important, and pooling the updates can make the evidence look much weaker than it is.

Forged Invariant 22 Oct 2021 7:22 UTC
13 points
in reply to: Vanessa Kosoy’s comment on: Petrov Day Retrospective: 2021
I found the postmortem over-focuses on what went wrong or was sub-optimal. I would like to point out that I found the event fun, despite being a lurker with no code.

Forged Invariant 22 Oct 2021 7:20 UTC
4 points
on: Petrov Day Retrospective: 2021
There were some reports of people seeing a frozen countdown on the button, that disappeared when the page was refreshed. Was this an intentional false alarm? I had assumed that was the case, as a false alarm with some evidence that it was false echoes some parts of Petrov’s situation nicely.

Forged Invariant 26 Sep 2021 21:44 UTC
1 point
in reply to: Peter Wildeford’s comment on: Petrov Day 2021: Mutually Assured Destruction?
Just be aware that other users have already noticed messages which could be deliberate false alarms: https://www.lesswrong.com/posts/EW8yZYcu3Kff2qShS/petrov-day-2021-mutually-assured-destruction?commentId=JbsutYRotfPDLNskK

Forged Invariant 16 Aug 2021 7:46 UTC
2 points
in reply to: PatrickDFarley’s comment on: Covid 8/12: The Worst Is Over
I had not noticed my own Gel-Mann amnesia when reading that bit, and therefore find your response quite convincing. I had thought that Ziv’s answer to (D) made sense due to the FDA being over-cautious about approving things, but both the scope of the precedent and the kinds/directions of errors had not registered with me.

Forged Invariant 22 Jul 2021 7:33 UTC
14 points
in reply to: Tomás B.’s comment on: The shoot-the-moon strategy
One possible strategy would be to make AI more dangerous as quickly as possible, in the hopes it produces a strong reaction and addition of safety protocols. Doing this with existing tools so that it is not an AGI makes it survivable. This reminds me a bit of Robert Miles facial recognition and blinding laser robot. (Which of course is never used to actually cause harm.)

Forged Invariant 9 Jul 2021 6:45 UTC
3 points
in reply to: Gerald Monroe’s comment on: Potential Bottlenecks to Taking Over The World
If the AGI can simply double it’s cognitive throughput, it can just repeat the action “sleuth to find an under-priced stock” as needed. This does not exhaust the order book until the entire market is operating at AGI-comparable efficiency, at which point the AGI probably controls a large (or majority) share of the trading volume.
Also, the other players would have limited ability to imitate the AGI’s tactics, so its edge would last until they left the market.

Forged Invariant 21 Jun 2021 6:38 UTC
5 points
on: Why did the UK switch to a 12 week dosing schedule for COVID-19 vaccines?
A hypothesis I had was that the US was sticking to an exact formula due to higher vaccine hesitancy, in order to “play it safe” and give less for anti-vaxers to criticize. After looking at a small handful of countries, I think this is not a significant cause of the difference in responses.
If this were true I would expect countries that have higher vaccine hesitancy to be less likely to do first doses first.
Checking [this data](https://www.thelancet.com/cms/10.1016/S0140-6736(20)31558-0/attachment/720358f5-8df0-405b-b06f-7734cf542a58/mmc1.pdf) which was near the top of search results, and using eyeballed values of strongly agree to “I think vaccines are safe” as the measure:
Canada: 75%, Yes FDF (March 3)
US: 66%, No FDF
Mexico: 60%, Yes FDF (Jan 22)
UK: 50%, Yes FDF (Jan 4)
Germany: 50%, Yes FDF (March 5)
Obviously a really small sample and I am being loose with the data, but it does not support this hypothesis, with no obvious correspondence between vaccine-confidence and when FDF started. I chose the countries in question off the top of my head.
Dates and sources were found by searching online, I have not carefully checked them.
https://www.statista.com/statistics/1195560/coronavirus-covid-19-vaccinations-number-germany/ This graph looks like there is about a 3-week lag in 2nd doses.
https://www.thetimes.co.uk/article/germany-follows-uk-by-delaying-second-dose-of-covid-vaccine-mk65kkh9w March 05, Germany starts FDF.
https://abcnews.go.com/Health/wireStory/mexico-russias-sputnik-shortages-limited-2nd-doses-77617433 Mexico does first doses first due to supply issues with the Sputnik vaccine; the first dose can be produced faster. The article does not mention the save more lives argument.
https://www.nasdaq.com/articles/mexico-may-delay-second-vaccine-doses-and-allow-private-orders-to-tame-raging-pandemic The tone of this piece seems to suggest FDF out of desperation.

Forged Invariant 21 Jun 2021 5:24 UTC
1 point
in reply to: Orual’s comment on: Why did the UK switch to a 12 week dosing schedule for COVID-19 vaccines?
From my understanding of the Canada situation, it may have been motivated by less access to vaccines initially. The US did very well in terms of getting lots of vaccines soon (https://ourworldindata.org/covid-vaccinations) while Canada took about 4 months after the US to really get going. Canada may have been more desperate to prevent Covid (or have their numbers stop lagging the US), and thus been less risk-adverse.
This argument does not work for the UK, as they have been ahead of the US the whole time.
https://www.msn.com/en-ca/news/canada/vaccine-panel-says-canada-can-delay-second-dose-of-covid-19-vaccine-if-shortage/ar-BB1cIJaG This article cites the decision being partly justified by limited supplies and how bad things were.

Forged Invariant 17 Jun 2021 20:13 UTC
1 point
in reply to: JonasMoss’s comment on: Open problem: how can we quantify player alignment in 2x2 normal-form games?
I like how this proposal makes explicit the player strategies, and how they are incorporated into the calculation. I also think that the edge case where the agents actions have no effect on the result
I think that this proposal making alignment symmetric might be undesirable. Taking the prisoner’s dilemma as an example, if s = always cooperate and r = always defect, then I would say s is perfectly aligned with r, and r is not at all aligned with s.
The result of 0 alignment for the Nash equilibrium of PD seems correct.
I think this should be the alignment matrix for pure-strategy, single-shot PD:
$a = [\begin{matrix} 1, 1 & 1, 0 0, 1 & 0, 0 \end{matrix}]$
Here the first of each ordered pair represents A’s alignment with B. (assuming we use the [0,1] interval)
I think in this case the alignments are simple, because A can choose to either maximize or to minimize B’s utility.

Forged Invariant 17 Jun 2021 6:15 UTC
1 point
in reply to: Templarrr’s comment on: Open problem: how can we quantify player alignment in 2x2 normal-form games?
1/1 0/0
⁰⁄₀ 0.8/-1
I have put the preferred state for each player in bold. I think by your rule this works out to 50% aligned. However, the Nash equilibrium is both players choosing the ¹⁄₁ result, which seems perfectly aligned (intuitively).
¹⁄₀.5 ⁰⁄₀
⁰⁄₀ 0.5/1
In this game, all preferred states are shared, yet there is a Nash equilibrium where each player plays the move that can get them 1 point ²⁄₃ of the time, and the other move ¹⁄₃ of the time. I think it would be incorrect to call this 100% aligned.
(These examples were not obvious to me, and tracking them down helped me appreciate the question more. Thank you.)