daozaich

Karma: 264

daozaich 14 Jun 2022 0:38 UTC
1 point
on: How much does cybersecurity reduce AI risk?
I doubt your optimism on the level of security that is realistically achievable. Don’t get me wrong: The software industry has made huge progress (at large costs!) in terms of security. Where before, most stuff popped a shell if you looked at it funny, it is now a large effort for many targets.
Further progress will be made.
If we extrapolate this progress—we will optimistically reach a point where impactful reliable 0day is out of reach for most hobbyists and criminals, and the domain of natsec of great powers.
But I don’t see how raising this waterline will help for AI risk in particular?
As in: godlike superintelligence is game over anyway. AI that is comparably good at exploitation as the rest of humanity taken together, is beyond what is realistically defendable against, in terms of wide-spread deployed security level. An AI that doesn’t reach that level without human assistance is probably not lethal anyways.
On the other hand, one could imagine pivotal acts by humans with limited-but-substantial AI assistance that rely on the lack of wide-spread security.
Pricing human + weakish AI collaborations out of the world-domination-via-hacking game might actually make matters worse, in so far as weakish non-independent AI might be easier to keep aligned.
A somewhat dystopian wholesale surveillance of almost every word written and said by humans, combined with AI that is good enough at text comprehension and energy efficient enough to pervasively and correctly identify scary-looking research and flag it to human operators for intervention is plausibly pivotal and alignable, and makes for much better cyberpunk novels than burning GPUs anyway (mentally paging cstross, I want my Gibson homage in form of a “Turing Police”/laundry-verse crossover).
Also, good that you mentioned rowhammer. Rowhammer and the DRAM industries half-baked pitiful response are humankinds capitulation in terms of “making at least some systems actually watertight”.

daozaich 19 Feb 2022 0:17 UTC
6 points
on: Implications of automated ontology identification
The fixed point problem is worse than you think. Take the Hungarian astrology example, with an initial easy set with both a length limitation (e.g. < 100k characters) and simplicity limitation.
Now I propose a very simple improvement scheme: If the article ends in a whitespace character, then try to classify the shortened article with last character removed.
This gives you an infinite sequence of better and better decision boundaries (each time, a couple of new cases are solved—the ones that are of lenth 100k + $N$, end in at least $N$ whitespace, and are in the easy set once the whitespace has been stripped). This nicely converges to the classifier that trims all trailing whitespace and then asks its initial classifier.
What I’m trying to say here is: The space of cases to consider can be large in many dimensions. The countable limit of a sequence of extensions needs not be a fixed point of the magical improvement oracle.
Generally, I’d go into a different direction: Instead of arguing about iterated improvement, argue that of course you cannot correctly extrapolate all decision problems from a limited amount of labeled easy cases and limited context. The style of counter-example is to construct two settings (“models” in the lingo of logic) A and B with same labeled easy set (and context made available to the classifier), where the correct answer for some datapoint x differs in both settings. Hence, safe extrapolation must always conservatively answer NO to x, and cannot be expected to answer all queries correctly from limited training data (typical YES / NO / MAYBE split).
I think the discussion about the fixed point or limit iterative improvement does not lead to the actually relevant argument that extrapolation cannot conjure information out of nowhere?
You could cut it out completely without weakening the argument against certain types of automated ontology identification being impossible.

daozaich 10 Jul 2018 14:26 UTC
1 point
on: Math: Textbooks and the DTP pipeline
The Definition-Theorem-Proof style is just a way of compressing communication. In reality, heuristic / proof-outline comes first; then, you do some work to fill the technical gaps and match to the existing canon, in order to improve readability and conform to academic standards.

Imho, this is also the proper way of reading maths papers / books: Zoom in on the meat. Once you understood the core argument, it is often unnecessary too read definitions or theorems at all (Definition: Whatever is needed for the core argument to work. Theorem: Whatever the core argument shows). Due to the perennial mismatch between historic definitions and theorems and the specific core arguments this also leaves you with stronger results than are stated in the paper / book, which is quite important: You are standing on the shoulders of giants, but the giants could not foresee where you want to go.

daozaich 8 Jul 2018 0:33 UTC
3 points
on: RFC: Mental phenomena in AGI alignment

This paints a bleak picture for the possibility of aligning mindless AGI since behavioral methods of alignment are likely to result in divergence from human values and algorithmic methods are too complex for us to succeed at implementing.

To me it appears like the terms cancel out: Assuming we are able to overcome the difficulties of more symbolic AI design, the prospect of aligning such an AI seem less hard.

In other words, the main risk is wasting effort on alignment strategies that turn out to be mismatched to the eventually implemented AI.

daozaich 3 Jul 2018 18:58 UTC
1 point
on: What will we do with the free energy?
The negative prices are a failure of the market / regulation, they don’t actually mean that you have free energy.

That being said, the question for the most economical opportunistic use of intermittent energy makes sense.

daozaich 3 Jul 2018 18:52 UTC
19 points
in reply to: Mitchell_Porter’s comment on: Why it took so long to do the Fermi calculation right?
No. It boils down to the following fact: If you take given estimates on the distribution of parameter values at face value, then:

(1) The expected number of observable alien civilizations is medium-large (2) If you consider the distribution of the number of alien civs, you get a large probability of zero, and a small probability of “very very many aliens”, that integrates up to the medium-large expectation value.

Previous discussions computed (1) and falsely observed a conflict with astronomical observations, and totally failed to compute (2) from their own input data. This is unquestionably an embarrassing failure of the field.

daozaich 27 Jun 2018 18:03 UTC
3 points
on: Logical uncertainty and Mathematical uncertainty
What is logical induction’s take on probabilistic algorithms? That should be the easiest test-case.

Say, before “PRIME is in P”, we had perfectly fine probabilistic algorithms for checking primality. A good theory of mathematical logic with uncertainty should permit us to use such an algorithm, without random oracle, for things you place as “logical uncertainty”. As far as I understood, the typical mathematician’s take is to just ignore this foundational issue and do what’s right (channeling Thurston: Mathematicians are in the business of producing human understanding, not formal proofs).

daozaich 8 Jun 2018 22:16 UTC
11 points
on: Monty Hall in the Wild

It’s excellent news! Your boss is a lot more likely to complain about some minor detail if you’re doing great on everything else, like actually getting the work done with your team.

Unfortunately this way of thinking has a huge, giant failure mode: It allows you to rationalize away critique about points you consider irrelevant, but that are important to your interlocutor. Sometimes people / institutions consider it really important that you hand in your expense sheets correctly or turn up in time for work, and finishing your project in time with brilliant results is not a replacement for “professional demeanor”. This was not a cheap lesson for me; people did tell me, but I kinda shrugged it off with this kind of glib attitude.

daozaich 8 Jun 2018 20:29 UTC
9 points
0
on: Editor Mini-Guide
Is there a way of getting “pure markdown” (no wysiwyg at all) including Latex? Alternatively, a hotkey-less version of the editor (give me buttons/menus for all functionality)?

I’m asking because my browser (chromium) eats the hotkeys, and latex (testing: $\Sigma$ ) appears not to be parsed from markdown. I would be happy with any syntax you choose. For example \Sigma; alternatively the github classic of using backticks appears still unused here.

edit: huh, backticks are in use and html-tags gets eaten.

daozaich 8 Jun 2018 19:45 UTC
3 points
on: Beyond Astronomical Waste
Isn’t all this massively dependent on how your utility $U$ scales with the total number $N$ of well-spent computations (e.g. one-bit computes)?

That is, I’m asking for a gut feeling here: What are your relative utilities for $10^{100}$, $10^{110}$, $10^{120}$, $10^{130}$ universes?

Say, $U(0)=0$, $U(10^100)=1$ (gauge fixing); instant pain-free end-of-universe is zero utility, and a successful colonization of the entire universe with a suboptimal black hole-farming near heat-death is unit utility.

Now, per definitionem, the utility $U(N)$ of a $N$-computation outcome is the inverse of the probability $p$ at which you become indifferent to the following gamble: Immediate end-of-the-world at probability $(1-p)$ vs an upgrade of computational world-size to $N$ at propability $p$.

I would personally guess that $U(10^{130})< 2 $; i.e. this upgrade would probably not be worth a 50% risk of extinction. This is massively sublinear scaling.

daozaich 2 Jun 2018 13:41 UTC
3 points
on: Into the Kiln: Insights from Tao’s ‘Analysis I’
What was initially counterintuitive is that even though $a_{n} \to 0$ , the series doesn’t converge.
This becomes much less counterintuitive if you instead ask: How would you construct a sequence $a_{n} \to 0$ with divergent series?
Obviously, take a divergent series, e.g. $\sum_{n} 1$ , and then split the $n$ th term $a_{n} = 1$ into $n \times \frac{1}{n}$ .

daozaich 1 Jun 2018 21:04 UTC
2 points
on: Understanding is translation
FWIW, looking at an actual compiler, we see zero jumps (using a conditional move instead):
```
julia> function test(n)
          i=0
          while i<n
              i += 1
          end
          return i
          end
test (generic function with 1 method)

julia> @code_native test(10)
    .text
    Filename: REPL\[26\]
pushq %rbp
movq %rsp, %rbp
    Source line: 3
xorl %eax, %eax
testq %rdi, %rdi
cmovnsq %rdi, %rax
    Source line: 6
popq %rbp
retq
nop
```
edit: Sorry for the formatting. I don’t understand how source-code markup is supposed to work now?

edit2: Thanks, the markup works now!

edit3: So, to tie this into your greater point:

If you don’t ask “how would I code this in assembly” but rather “how should my compiler reason about this code”, then it is clear that the loop can be obviously eliminated: You place a phi-node at the end of the loop, and a tiny bit of inductive reasoning makes the loop body obviously dead code if n is an integer type. Slightly more magical (meaning I’m not a compiler expert) is the fact that the compiler (LLVM) can completely eliminate the following loop (replacing it with an explicit formula):
```
julia> function sumN6(lim)
   s=0
   i=0
   while i<lim
       i+=1
       s+= i*i*i*i*i*i
   end
   return s
   end
```

daozaich 25 May 2018 22:37 UTC
4 points
in reply to: jessicata’s comment on: Decision theory and zero-sum game theory, NP and PSPACE
“what move should open with in reversi” would be considered as an admissible decision-theory problem by many people. Or in other words: Your argument that EU maximization is in NP only holds for utility functions that permit computation in P of expected utility given your actions. That’s not quite true in the real world.

daozaich 5 May 2018 17:26 UTC
1 point
in reply to: RamblinDash’s comment on: Moral frameworks and the Harris/Klein debate
This, so much.
So, in the spirit of learning from other’s mistakes (even better than learning from my own): I thought Ezra made his point very clear.
So, all of you people who missed Ezra’s point (confounded data, outside view) on first reading:
How could Ezra have made clearer what he was arguing, short of adopting LW jargon? What can we learn from this debacle of a discussion?
Edit: tried to make my comment less inflammatory.

daozaich 24 Apr 2018 16:30 UTC
1 point
in reply to: Stuart_Armstrong’s comment on: Weird question: could we see distant aliens?
>I was imagining a sort of staged rocket, where you ejected the casing of the previous rockets as you slow, so that the mass of the rocket was always a small fraction of the mass of the fuel.
Of course, but your very last stage is still a rocket with a reactor. And if you cannot build a rocket with 30g motor+reactor weight, then you cannot go to such small stages and your final mass on arrival includes the smallest efficient rocket motor / reactor you can build, zero fuel, and a velocity that is below escape velocity of your target solar system (once you are below escape velocity I’ll grant you maneuvers with zero mass cost, using solar sails; regardless, tiny solar-powered ion-drives appear reasonable, but generate not enough thrust to slow down from relativistic to below-escape in the time-frame before you have passed though your target system).
>But Eric Drexler is making some strong arguments that if you eject the payload and then decelerate the payload with a laser fired from the rest of the “ship”, then this doesn’t obey the rocket equation. The argument seems very plausible (the deceleration of the payload is *not* akin to ejecting a continuous stream of small particles—though the (tiny) acceleration of the laser/ship is). I’ll have to crunch the number on it.
That does solve the “cannot build a small motor” argument, potentially at the cost of some inefficiency.
It still obeys the rocket equation. The rocket equation is like the 2nd law of thermodynamics: It is not something you can trick by clever calculations. It applies for all propulsion systems that work in a vacuum.
You can only evade the rocket equation by finding (non-vacuum) stuff to push against; whether it be the air in the atmosphere (airplane or ramjet is more efficient than rocket!), the solar system (gigantic launch contraption), various planets (gravitational slingshot), the cosmic microwave background, the solar wind, or the interstellar medium. Once you have found something, you have three choices: Either you want to increase relative velocity and expend energy (airplane, ramjet), or you want to decrease relative velocities (air-braking, use of drag/friction, solar sails when trying to go with the solar wind, braking against the interstellar medium, etc), or you want an elastic collision, e.g. keep absolute relative velocity the same but reverse direction (gravitational slingshot).
Slingshots are cool because you extract energy from the fact that the planets have different velocities: Having multiple planets is not thermodynamic ground state, so you steal from the potential energy / negative entropy left over from the formation of the solar system. Alas, slingshots can’t bring you too much above escape velocity, nor slow you down to below escape if you are significantly faster.
Edit: probably stupid idea, wasn’t thinking straight <strike> Someone should tell me whether you can reach relativistic speeds by slingshotting in a binary or trinary of black holes. That would be quite elegant (unbounded escape velocity, yay! But you have time dilation when close to the horizon, so unclear whether this takes too long from the viewpoint of outside observers; also, too large shear will pull you apart).</strike>
edit2: You can afaik also push against a curved background space-time, if you have one. Gravity waves technically count as vacuum, but not for the purpose of the rocket equation. Doesn’t help, though, because space-time is pretty flat out there, not just Ricci-flat (=vacuum).

daozaich 24 Apr 2018 10:45 UTC
4 points
in reply to: paulfchristiano’s comment on: Weird question: could we see distant aliens?
If you have to use the rocket equation twice, then you effectively double delta-v requirements and square the launch-mass / payload-mass factor.
Using Stuart’s numbers, this makes colonization more expensive by the following factors:
0.5 c: Antimatter 2.6 / fusion 660 / fission 1e6
0.8 c: Antimatter 7 / fusion 4.5e5 / fission 1e12
0.99c Antimatter 100 / fusion 4.3e12 / fission 1e29
If you disbelieve in 30g fusion reactors and set a minimum viable weight of 500t for an efficient propulsion system (plus negligible weight for replicators) then you get an additional factor of 1e7.
Combining both for fusion at 0.8c would give you a factor of 5e12, which is significantly larger than the factor between “single solar system” and “entire galaxy”. These are totally pessimistic assumptions, though: Deceleration probably can be done cheaper, and with lower minimal mass for efficient propulsion systems. And you almost surely can cut off quite a bit of rocket-delta-v on acceleration (Stuart assumed you can cut 100% on acceleration and 0% on deceleration; the above numbers assumed you can cut 0% on acceleration and 0% on deceleration).
Also, as Stuart noted, you don’t need to aim at every reachable galaxy, you can aim at every cluster and spread from there.
So, I’m not arguing with Stuart’s greater claim (which is a really nice point!), I’m just arguing about local validity of his arguments and assumptions.

daozaich 23 Apr 2018 22:18 UTC
4 points
in reply to: gjm’s comment on: On exact mathematical formulae
You’re right, I should have made that clearer, thanks!

daozaich 23 Apr 2018 20:54 UTC
4 points
in reply to: Stuart_Armstrong’s comment on: Weird question: could we see distant aliens?
I would not fret too much about slight overheating of the payload; most of the launch mass is propulsion fuel anyway, and in worst-case the payload can rendezvous with the fuel in-flight, after the fuel has cooled down.
I would be very afraid of the launch mass, including solar sail / reflector loosing (1) reflectivity (you need a very good mirror that continues to be a good mirror when hot; imperfections will heat it) and (2) structural integrity.
I would guess that, even assuming technological maturity (can do anything that physics permits), you cannot keep structural integrity above, say, 3000K, for a launch mass that is mostly hydrogen. I think that this is still icy cold, compared to the power output you want.
So someone would need to come up with either
1. amazing schemes for radiative heat-dissipation and heat pumping (cannot use evaporative cooling, would cost mass),
2. something weird like a plasma mirror (very hot plasma contained by magnetic fields; this would be hit by the laser, which pushes it via radiation pressure and heats it; momentum is transferred from plasma to launch probe via magnetic field; must not loose too many particles, and might need to maintain a temperature gradient so that most radiation is emitted away from the probe; not sure whether you can use dynamo flow to extract energy from the plasma in order to run heat pumps, because the plasma will radiate a lot of energy in direction of the probe),
3. show that limiting the power so that the sail has relatively low equilibrium temperature allows for enough transmission of momentum.
No 3 would be the simplest and most convincing answer.
I am not sure whether a plasma mirror is even thermo-dynamically possible. I am not sure whether sufficient heat-pumps plus radiators are “speculative engineering”-possible, if you have a contraption where your laser pushes against a shiny surface (necessitating very good focus of the laser). If you have a large solar sail (high surface, low mass) connected by tethers, then you probably cannot use active cooling on the sail; therefore there is limited room for fancy future-tech engineering, and we should be able to compute some limits now.
---------------------
Since I already started raising objections to your paper, I’ll raise a second point: You compute the required launch mass from rocket-equation times final payload, with the final payload having very low weight. This assumes that you can actually build such a tiny rocket! While I am willing to suspend disbelieve and assume that a super efficient fusion-powered rocket of 500 tons might be built, I am more skeptical if your rocket, including fusion reactor but excluding fuel, is limited to 30 gram of weight.
Or did I miss something? While this would affect your argument, my heart is not really in it: Braking against the interstellar medium appears, to me, to circumvent a lot of problems.
---------------------
Because I forgot and you know your paper better than me: Do any implicit or explicit assumptions break if we lose access to most of the fuel mass for shielding during the long voyage?
If you could answer with a confident “no, our assumptions do not beak when cannot use the deceleration fuel as shielding”, then we can really trade-off acceleration delta-v against deceleration delta-v, and I stay much more convinced about your greater point about the Fermi paradox.

daozaich 23 Apr 2018 13:25 UTC
1 point
on: The many ways AIs behave badly
This was a very fun article. Notably absent from the list, even though I would absolutely have expected it (since the focus was on evolutionary algorithms, even though many observations also apply to gradient-descent):
Driving genes. Biologically, a “driving gene” is one that cheats in (sexual) evolution, by ensuring that it is present in >50% of offspring, usually by weirdly interacting with the machinery that does meiosis.
In artificial evolution that uses “combination”, “mutation” and “selection”, these would be regions of parameter-space that are attracting under “combination”-dynamics, and use that to beat selection pressure.

daozaich 23 Apr 2018 12:41 UTC
6 points
in reply to: Stuart_Armstrong’s comment on: Weird question: could we see distant aliens?
If you assume that Dysoning and re-launch take 500 years, this barely changes the speed either, so you are very robust.
I’d be interested in more exploration of deceleration strategies. It seems obvious that braking against the interstellar medium (either dust or magnetic field) is viable to some large degree; at the very least if you are willing to eat a 10k year deceleration phase. I have taken a look at the two papers you linked in your bibliography, but would prefer a more systematic study. Important is: Do we know ways that are definitely not harder than building a dyson swarm, and is one galaxy’s width (along smallest dimension) enough to decelerate? Or is the intergalactic medium dense enough for meaningful deceleration?
I would also be interested in a more systematic study of acceleration strategies. Your arguments absolutely rely on circumventing the rocket equation for acceleration; break this assumption, and your argument dies.
It does not appear obvious to me that this is possible: Say, coil guns would need a ridiculously long barrel and mass, or would be difficult to maneuver (you want to point the coil gun at all parts of the sky). Or, say, laser acceleration turns out to be very hard because of (1) lasers are fundamentally inefficient (high thermal losses), and cannot be made efficient if you want very tight beams and (2) cooling requirement for the probes during acceleration turn out to be unreasonable. [*]
I could imagine a world where you need to fall back to the rocket equation for a large part of the acceleration delta-v, even if you are a technologically mature superintelligence with dyson swarm. Your paper does not convince me that such a world is impossble (and it tries to convince me that hypothetical worlds are impossible, where it would be hard to rapidly colonize the entire universe if you have reasonably-general AI).
Obviously both points are running counter to each other: If braking against the interstellar medium allows you to get the delta-v for deceleration down to 0.05 c from, say 0.9 c, but acceleration turns out to be so hard that you need to get 0.8 c with rockets (you can only do 0.1c with coil guns / lasers, instead of 0.9 c), then we have not really changed the delta-v calculus; but we have significantly changed the amount of available matter for shielding during the voyage (we now need to burn most of the mass during acceleration instead of deceleration, which means that we are lighter during the long voyage).
[*] Superconductors can only support a limited amount of current / field-strength. This limits the acceleration. Hence, if you want larger delta-v, you need a longer barrel. How long, if you take the best known superconductors? At which fraction of your launch probe consisting of superconducting coils, instead of fusion fuel? Someone must do all these calculations, and then discuss how the resulting coil gun is still low-enough mass compared to the mass of a dyson swarm, and how to stabilize, power, cool and maneuver this gun. Otherwise, the argument is not convincing.
edit: If someone proposes a rigid barrel that is one light-hour long then I will call BS.