harfe

• harfe 15 Nov 2024 11:09 UTC

  7 points

  2

  in reply to: Garrett Baker’s comment on: D0TheMath’s Shortform

  insofar as the simplest & best internal logical-induction market traders have strong beliefs on the subject, they may very well be picking up on something metaphysically fundamental. Its simply the simplest explanation consistent with the facts.

  Theorem 4.6.2 in logical induction says that the “probability” of independent statements does not converge to $1$ or $0$, but to something in-between. So even if a mathematician says that some independent statement feels true (eg some objects are “really out there”), logical induction will tell him to feel uncertain about that.

• harfe 14 Nov 2024 12:27 UTC

  6 points

  0

  in reply to: johnswentworth’s comment on: john­swent­worth’s Shortform

  A related comment from lukeprog (who works at OP) was posted on the EA Forum. It includes:

  However, at present, it remains the case that most of the individuals in the current field of AI governance and policy (whether we fund them or not) are personally left-of-center and have more left-of-center policy networks. Therefore, we think AI policy work that engages conservative audiences is especially urgent and neglected, and we regularly recommend right-of-center funding opportunities in this category to several funders.

• harfe 13 Nov 2024 19:16 UTC

  4 points

  0

  in reply to: notfnofn’s comment on: Flip­ping Out: The Cos­mic Coin­flip Thought Ex­per­i­ment Is Bad Philosophy

  it’s for the sake of maximizing long-term expected value.

  Kelly betting does not maximize long-term expected value in all situations. For example, if some bets are offered only once (or even a finite amount), then you can get better long-term expected utility by sometimes accepting bets with a potential “0”-Utility outcome.

• harfe 11 Nov 2024 12:26 UTC

  4 points

  0

  in reply to: Remmelt’s comment on: If we had known the at­mo­sphere would ignite

  This is maybe not the central point, but I note that your definition of “alignment” doesn’t precisely capture what I understand “alignment” or a good outcome from AI to be:

  ‘AGI’ continuing to exist

  AGI could be very catastrophic even when it stops existing a year later.

  eventually

  If AGI makes earth uninhabitable in a trillion years, that could be a good outcome nonetheless.

  ranges that existing humans could survive under

  I don’t know whether that covers “humans can survive on mars with a space-suit”, but even then, if humans evolve/​change to handle situations that they currently do not survive under, that could be part of an acceptable outcome.

• harfe 11 Nov 2024 11:15 UTC

  3 points

  0

  in reply to: flandry39’s comment on: If we had known the at­mo­sphere would ignite

  it is the case that most algorithms (as a subset in the hyperspace of all possible algorithms) are already in their maximally most simplified form. Even tiny changes to an algorithm could convert it from ‘simplifiable’ to ‘non-simplifiable’.

  This seems wrong to me: For any given algorithm you can find many equivalent but non-simplified algorithms with the same behavior, by adding a statement to the algorithm that does not affect the rest of the algorithm (e.g. adding a line such as foobar1234 = 123 in the middle of a python program)). In fact, I would claim that the majority python programs on github are not in their “maximally most simplified form”. Maybe you can cite the supposed theorem that claims that most (with a clearly defined “most”) algorithms are maximally simplified?

• harfe 5 Nov 2024 12:02 UTC

  1 point

  2

  in reply to: Remmelt’s comment on: If we had known the at­mo­sphere would ignite

  This is not a formal definition.

  Your English sentence has no apparent connection to mathematical objects, which would be necessary for a rigorous and formal definition.

• harfe 23 Oct 2024 14:42 UTC

  3 points

  0

  in reply to: Lucius Bushnaq’s comment on: Lu­cius Bush­naq’s Shortform

  I think you are broadly right.

  So we’re automatically giving $x_1$ ca. $2^{1000}$ higher probability – even before applying the length penalty $\frac{1}{2^{|p|}}$.

  But note that under the Solomonoff prior, you will get another $2^{-2000-|G|}$ penalty for these programs with DEADCODE. So with this consideration, the weight changes from $2^{-1000}$ (for normal $p_1$) to $2^{-1000}(1+2^{-|G|})$ (normal $p_1$ plus $2^{1000}$ DEADCODE versions of $p_1$), which is not a huge change.

  For your case of “uniform probability until ${10}^{90}$” I think you are right about exponential decay.

• harfe 9 Oct 2024 21:03 UTC

  2 points

  0

  in reply to: exmateriae’s comment on: How many lan­guages should ex­ist?

  That point is basically already in the post:

  large language models can help document and teach endangered languages, providing learning tools for younger generations and facilitating the transmission of knowledge. However, this potential will only be realized if we prioritize the integration of all languages into AI training data.

• harfe 15 Aug 2024 15:32 UTC

  1 point

  2

  on: An­nounc­ing the $200k EA Com­mu­nity Choice

  I have doubts that the claim about “theoretically optimal” apply to this case.

  Now, you have not provided a precise notion of optimality, so the below example might not apply if you come up with another notion of optimality or assume that voters collude with each other, or use a certain decision theory, or make other assumptions… Also there are some complications because the optimal strategy for each player depends on the strategy of the other players. A typical choice in these cases is to look at Nash-equilibria.

  Consider three charities A,B,C and two players X,Y who can donate $100 each. Player X has utilities $1$, $0.9$, $0$ for the charities A,B,C. Player Y has utilities $0$, $0.9$, $1$ for the charities A,B,C.

  The optimal (as in most overall utility) outcome would be to give everything to charity B. This would require that both players donate everything to charity B. However, this is not a Nash-equilibrium, as player X has an incentive to defect by giving to A instead of B and getting more utility.

  This specific issue is like the prisoners dilemma and could be solved with other assumptions/​decision theories.

  The difference between this scenario and the claims in the literature might be that public goods is not the same as charity, or that the players cannot decide to keep the funds for themselves. But I am not sure about the precise reasons.

  Now, I do not have an alternative distribution mechanism ready, so please do not interpret this argument as serious criticism of the overall initiative.

• harfe 8 Aug 2024 11:23 UTC

  4 points

  0

  in reply to: rsaarelm’s comment on: It’s time for a self-re­pro­duc­ing machine

  There is also Project Quine, which is a newer attempt to build a self-replicating 3D printer

• harfe 29 Jul 2024 18:00 UTC

  3 points

  4

  on: New Blog Post Against AI Doom

  This was already referenced here: https://​​www.lesswrong.com/​​posts/​​MW6tivBkwSe9amdCw/​​ai-existential-risk-probabilities-are-too-unreliable-to

  I think it would be better to comment there instead of here.

• harfe 20 Jun 2024 16:06 UTC

  8 points

  0

  on: Ilya Sutskever cre­ated a new AGI startup

  One thing I find positive about SSI is their intent to not have products before superintelligence (note that I am not arguing here that the whole endeavor is net-positive). Not building intermediate products lessens the impact on race dynamics. I think it would be preferable if all the other AGI labs had a similar policy (funnily, while typing this comment, I got a notification about Claude 3.5 Sonnet… ). The policy not to have any product can also give them cover to focus on safety research that is relevant for superintelligence, instead of doing some shallow control of the output of LLMs.

  To reduce bad impacts from SSI, it would be desirable that SSI also

  • have a clearly stated policy to not publish their capabilities insights,

  • take security sufficiently seriously to be able to defend against nation-state actors that try to steal their insights.

• harfe 19 Jun 2024 18:37 UTC

  4 points

  1

  in reply to: Chris_Leong’s comment on: Ilya Sutskever cre­ated a new AGI startup

  It does not appear paywalled to me. The link that @mesaoptimizer posted is an archive, not the original bloomberg.com article.

Ilya Sutskever cre­ated a new AGI startup

• harfe 12 May 2024 10:57 UTC

  3 points

  0

  on: New in­tro text­book on AIXI

  I haven’t watched it yet, but there is also a recent technical discussion/​podcast episode about AIXI and relatedd topics with Marcus Hutter: https://​​www.youtube.com/​​watch?v=7TgOwMW_rnk

• harfe 3 May 2024 21:55 UTC

  3 points

  1

  on: Max­i­mal Lot­tery-Lot­ter­ies Exist

  It suffices to show that the Smith lotteries that the above result establishes are the only lotteries that can be part of maximal lottery-lotteries are also subject to the partition-of-unity condition.

  I fail to understand this sentence. Here are some questions about this sentence:

  • what are Smith lotteries? Ctrl+f only finds lottery-Smith lottery-lotteries, do you mean these? Or do you mean lotteries that are smith?

  • which result do you mean by “above result”?

  • What does it mean for a lottery to be part of maximal lottery-lotteries?

  • does “also subject to the partition-of-unity” refer to the smith lotteries or to the lotteries that are part of maximal lottery-lotteries? (it also feels like there is a word missing somewhere)

  • Why would this suffice?

  • Is this part also supposed to imply the existence of maximal lottery-lotteries? If so, why?

• harfe 19 Apr 2024 20:19 UTC

  1 point

  0

  on: What is the best way to talk about prob­a­bil­ities you ex­pect to change with ev­i­dence/​ex­per­i­ments?

  A lot of the probabilities we talk about are probabilities we expect to change with evidence. If we flip a coin, our p(heads) changes after we observe the result of the flipped coin. My p(rain today) changes after I look into the sky and see clouds. In my view, there is nothing special in that regard for your p(doom). Uncertainty is in the mind, not in reality.

  However, how you expect your p(doom) to change depending on facts or observation is useful information and it can be useful to convey that information. Some options that come to mind:

  1. describe a model: If your p(doom) estimate is the result of a model consisting of other variables, just describing this model is useful information about your state of knowledge, even if that model is only approximate. This seems to come closest to your actual situation.

  2. describe your probability distribution of your p(doom) in 1 year (or another time frame): You could say that you think there is a 25% chance that your p(doom) in 1 year is between 10% and 30%. Or give other information about that distribution. Note: your current p(doom) should be the mean of your p(doom) in 1 year.

  3. describe your probability distribution of your p(doom) after a hypothetical month of working on a better p(doom) estimate: You could say that if you were to work hard for a month on investigating p(doom), you think there is a 25% chance that your p(doom) after that month is between 10% and 30%. This is similar to 2., but imo a bit more informative. Again, your p(doom) should be the mean of your p(doom) after a hypothetical month of investigation, even if you don’t actually do that investigation.

• harfe 10 Feb 2024 16:47 UTC

  1 point

  in reply to: Yitz’s comment on: Yitz’s Shortform

  This sounds like https://​​www.super-linear.org/​​trumanprize. It seems like it is run by Nonlinear and not FTX.

• harfe 10 Jan 2024 19:25 UTC

  1 point

  on: Ba­sic In­framea­sure Theory

  I think Proposition 1 is false as stated because the resulting functional $f^{+}:M^{\pm }\left(X\right)\oplus R\to R$ is not always continuous (wrt the KR-metric). The function $f:[0,1]\to [0,1]$, $x\mapsto x^{1/3}$ with $X=[0,1]$ should be a counterexample. However, the non-continuous functional $f^{+}$ should still be continuous on the set of sa-measures.

  Another thing: the space of measures $M^{\pm }\left(X\right)$ is claimed to be a Banach space with the KR-norm (in the notation section). Afaik this is not true, while the space is a Banach space with the TV-norm, with the KR-metric/​norm it should not be complete and is merely a normed vector space. Also the claim (in “Basic concepts”) that $M^{\pm }\left(X\right)$ is the dual space of $C\left(X\right)$ is only true if equipped with TV-norm, not with KR-metric.

  Another nitpick: in Theorem 5, the type of $h$ in the assumption is probably meant to be $C\left(X\right)\to R\cup \{-\infty \}$, instead of $C\left(X\right)\to R$.

• harfe 9 Jan 2024 16:21 UTC

  LW: 3 AF: 3

  0

  AF

  on: The Learn­ing-The­o­retic Agenda: Sta­tus 2023

  Regarding direction 17: There might be some potential drawbacks to ADAM. I think its possible that some very agentic programs have relatively low $g$ score. This is due to explicit optimization algorithms being low complexity.

  (Disclaimer: the following argument is not a proof, and appeals to some heuristics/​etc. We fix $M=M_0$ for these considerations too.) Consider an utility function $\hat{U}$. Further, consider a computable approximation of the optimal policy (AIXI that explicitly optimizes for $\hat{U}$) and has an approximation parameter n (this could be AIXI-tl, plus some approximation of $\hat{U}$; higher $n$ is better approximation). We will call this approximation of the optimal policy ${\pi }_n^{\hat{U}}$. This approximation algorithm has complexity $K\left({\pi }_n^{\hat{U}}\right)=C+K\left(\hat{U}\right)+K\left(n\right)$, where $C>0$ is a constant needed to describe the general algorithm (this should not be too large).

  We can get better approximation by using a quickly growing function, such as the Ackermann function with $n=A(k,k)$. Then we have $K\left({\pi }_{A(k,k)}^{\hat{U}}\right)=C+K\left(\hat{U}\right)+K\left(A\right(k,k\left)\right)\le C+K\left(\hat{U}\right)+\log \left(k\right)$.

  What is the $g$ score of this policy? We have $g\left({\pi }_{A(k,k)}^{\hat{U}}\right)={max}_U\left({min}_{{\pi }^{\prime }:\dots }K\right({\pi }^{\prime })-K(U\left)\right)$. Let $\overline{U}$ be maximal in this expression. If $K\left(\overline{U}\right)\ge K\left(\hat{U}\right)-C$, then $$g\left({\pi }_{A(k,k)}^{\hat{U}}\right)=\underset{{\pi }^{\prime }:E_{{\zeta }_{M_0}{\pi }^{\prime }}\left(\overline{U}\right)\ge E_{{\zeta }_{M_0}{\pi }_{A(k,k)}^{\hat{U}}}\left(\overline{U}\right)}{min}K\left({\pi }^{\prime }\right)-K\left(\overline{U}\right)\le K\left({\pi }_{A(k,k)}^{\hat{U}}\right)-K\left(\hat{U}\right)+C\le 2C\log \left(k\right)$$.

  For the other case, let us assume that if $K\left(\overline{U}\right)<K\left(\hat{U}\right)-C$, the policy ${\pi }_{A(k,k)}^{\overline{U}}$ is at least as good at maximizing $\overline{U}$ than ${\pi }_{A(k,k)}^{\hat{U})}$. Then, we have $$g\left({\pi }_{A(k,k)}^{\hat{U}}\right)=\underset{{\pi }^{\prime }:E_{{\zeta }_{M_0}{\pi }^{\prime }}\left(\overline{U}\right)\ge E_{{\zeta }_{M_0}{\pi }_{A(k,k)}^{\hat{U}}}\left(\overline{U}\right)}{min}K\left({\pi }^{\prime }\right)-K\left(\overline{U}\right)\le K\left({\pi }_{A(k,k)}^{\overline{U}}\right)-K\left(\overline{U}\right))\le C+\log (k)$$.

  I don’t think that the assumption ($({\pi }_{A(k,k)}^{\overline{U}}$ maximizes $barU$ better than $({\pi }_{A(k,k)}^{\hat{U}}$) is true for all $\hat{U}$ and $k$, but plausibly we can select $\hat{U}$ such that this is the case (exceptions, if they exist, would be a bit weird, and if ADAM working well due to these weird exceptions feels a bit disappointing to me). A thing that is not captured by approximations such as AIXI-tl are programs that halt but have insane runtime (longer than $A(k,k)$). Again, it would feel weird to me if ADAM sort of works because of low-complexity extremely-long-running halting programs.

  To summarize, maybe there exist policies ${\pi }_{A(k,k)}^{\hat{U}}$ which strongly optimize a non-trivial utility function $\hat{U}$ with approximation parameter $n=A(k,k)$, but where $g\left({\pi }_{A(k,k)}^{\hat{U}}\right)\le 2C+\log \left(k\right)$ is relatively small.