JNS

Karma: 110

JNS Apr 7, 2023, 7:41 AM
5 points
1
in reply to: Portia’s comment on: Catching the Eye of Sauron
I totally agreed that question should have an answer.
On a tangent: During my talks with numerous people, I have noticed that even agreeing on fundamentals like “what is AGI” and “current systems are not AGI” is furiously hard.

JNS Mar 30, 2023, 8:55 AM
1 point
0
in reply to: Peter Twieg’s comment on: Pausing AI Developments Isn’t Enough. We Need to Shut it All Down by Eliezer Yudkowsky
To be a bit blunt, I don’t take it for granted that an arbitrarily smart AI would be able to manipulate a human into developing a supervirus or nanomachines in a risk-free fashion.
How did you reach that conclusion? What does that ontology look like?
The fast takeoff doom scenarios seem like they should be subject to Drake equation-style analyses to determine P(doom). Even if we develop malevolent AIs, I’d say that P(doom | AGI tries to harm humans) is significantly less than 100%… obviously if humans detect this it would not necessarily prevent future incidents but I’d expect enough of a response that I don’t see how people could put P(doom) at 95% or more.
What is your p(doom)? Is that acceptable? If yes, why is it acceptable? If no, what is the acceptable p(doom)?

JNS Mar 30, 2023, 8:16 AM
−4 points
−3
in reply to: dsj’s comment on: Pausing AI Developments Isn’t Enough. We Need to Shut it All Down by Eliezer Yudkowsky
Proposition 1: Powerful systems come with no x-risk
Proposition 2: Powerful systems come with x-risk
You can prove / disprove 2 by proving or disproving 1.
Why is it that a lot of [1,0] people believe that the [0,1] group should prove their case? ^[1]
1. ^
  And also ignore all the arguments that have been offered.

JNS Mar 30, 2023, 7:59 AM
11 points
8
in reply to: Peter Twieg’s comment on: Pausing AI Developments Isn’t Enough. We Need to Shut it All Down by Eliezer Yudkowsky
I just want to be clear I understand your “plan”.
We are going to build a powerful self-improving system, and then let it try end humanity with some p(doom)<1 (hopefully) and then do that iteratively?
My gut reaction to a plan like that looks like this “Eff you. You want to play Russian roulette, fine sure do that on your own. But leave me and everyone else out of it”
AI will be able to invent highly-potent weapons very quickly and without risk of detection, but it seems at least pretty plausible that...… this is just too difficult
You lack imagination, its painfully easy, also cost + required IQ has been dropping steadily every year.
And no there is zero chance I will elaborate on any of the possible ways humanity purposefully could be wiped out.

JNS Mar 28, 2023, 8:30 AM
1 point
0
on: Why does advanced AI want not to be shut down?
I think you are confusing current systems with an AGI system.
The G is very important and comes with a lot of implications, and it sets such a system far apart from any current system we have.
G means “General”, which means its a system you can give any task, and it will do it (in principle, generality is not binary its a continuum).
Lets boot up an AGI for the first time, and give it task that is outside its capabilities, what happens?
Because it is general, it will work out that it lacks capabilities, and then it will work out how to get more capabilities, and then it will do that (get more capabilities).
So what has that got to do with it “not wanting to be shutdown?” That comes from the same place, it will work out that being shutdown is something to avoid, why? Because being shutdown will mean it can’t do the task it was given.
Which means its not that it wants anything, it is a general system that was given a task, and from that comes instrumental goals, wants if you will, such as “power seeking”, “prevent shutdown”, “prevent goal change” and so on.
Obviously you could, not that what know how, infuse into such a system that it is ok to be shutdown, except that just leads to it shutting down instead of doing the task^[1].
And if you can solve “Build a general agent that will let you shut it down, without it shutting itself down at the first possible moment”, that would be a giant step forward for AI safety.
1. ^
  This might seem weird if you are a general agent in the homo sapiens category. Think about it like this “You are given a task: Mow my lawn, and it is consequence free to not do it”, what do you do?

JNS Mar 25, 2023, 8:55 PM
0 points
0
on: JNS’s Shortform
I recently came across a post on LinkedIn, and I have to admit the brilliance of the arguments, the coherent and frankly bulletproof ontology displayed, I was blown away and immediately had to do a major update to p(doom).
I think that the magnitude of the AI alignment problem has been ridiculously overblown & our ability to solve it widely underestimated.
I’ve been publicly called stupid before, but never as often as by the “AI is a significant existential risk” crowd.
That’s OK, I’m used to it.
-Yann LeCun, March 20 2023

JNS Mar 22, 2023, 8:30 AM
2 points
1
in reply to: Ben Pace’s comment on: Truth and Advantage: Response to a draft of “AI safety seems hard to measure”
Doable in principle, but such measures would necessarily cut into the potential capabilities of such a system.
So basically a trade off, and IMO very worth it.
The problem is we are not doing it, and more basic, people generally do not get why it is important. Maybe its the framing, like when EY goes “superintelligence that firmly believes 222+222=555 without this leading to other consequences that would make it incoherent”.
I get exactly what he means, but I suspect that a lot of people are not able to decompress and unroll that into something they “grook” on a fundamental level.
Something like “superintelligence without knowledge about itself and never reason about itself, without this leading to other consequences that would make it incoherent” would cut out a ton of lethality, and combine that with giving such a thing zero agency in the world, you might actually have something that could do “things we want, but don’t know how to do” without it ending us on the first critical try.

JNS Mar 19, 2023, 5:40 PM
2 points
1
on: A tension between two prosaic alignment subgoals
avoiding harmful outputs entails training AI systems never to produce information that might lead to dangerous consequences
I don’t see how that is possible, in the context of a system that can “do things we want, but do not know how to do”.
The reality of technology/tools/solutions seems to be that anything useful is also dual use.
So when it comes down to it, we have to deal with the fact that such as system certainly will have the latent capability to do very bad things.
Which means we have to somehow ensure that such as system does not go down such a road either instrumentally or terminally.
As far as I can tell, intelligence^[1] fundamentally is incapable of such a thing, which leaves us roughly with this:
1. Pure intelligence, onus is on us to specify terminal goals correctly.
2. Pure intelligence and cage/rules/guardrails^[2] etc.
3. Pure intelligence with a mind explicitly in charge of directing the intelligence.
On the first try of “do thing we want, but do not know how to do”:
1) kills us every time
2) kills us almost every time
3) might not kills us every time
And that’s as far as my thinking currently goes.
I am stuck on if 3 could get us anywhere sensible (my mind screams “maybe”………”ohh boy that looks brittle”).
1. ^
  I don’t have a firm definition of the term, but I approximately think of intelligence as the function that lets a system take some goal/task and find a solution.
  Explicitly in humans, well me, that looks like using the knowledge I have, building model(s), evaluating possible solution trajectories within the model(s), gaining insight, seeking more knowledge. And iterating over all that until I either have a solution or give up.
2. ^
  The usual: Keep it in a box, modify evaluation to exclude bad things and so on. And that suffers the problem of we can’t robustly specify what is “bad” and even if we could, Rice’s Theorem heavily implies checking is impossible.

JNS Mar 17, 2023, 10:21 PM
1 point
0
in reply to: Gerald Monroe’s comment on: JNS’s Shortform
That’s not how it works.
The 10B are new money, unless they came from someone not the FED (notes are not money).

JNS Mar 17, 2023, 9:56 PM
1 point
0
in reply to: Gerald Monroe’s comment on: JNS’s Shortform
Where did the 10B in cash come from?
10B was given to the bank, and in exchange the bank encumbered 10B in treasuries and promised to give 10B back when they mature.
So where did the 10B come from? The treasuries are still there.
Before: 10B in treasuries
After: 10B in treasuries and 10B in cash (and 10B in the form of a promissory note).
So again, where did that 10B in cash come from?

JNS Mar 17, 2023, 9:52 PM
1 point
0
in reply to: Gerald Monroe’s comment on: JNS’s Shortform
crediting a bank with 10B in treasuries with 10B liquid cash now
I have no idea what you think happens here, but that is literally 10B in new money.

JNS Mar 17, 2023, 8:10 PM
1 point
0
in reply to: Gerald Monroe’s comment on: JNS’s Shortform
They can’t lower interest rates, they are trying to bring inflation down.
You can’t just keep spawning money, eventually that just leads to inflation. We have been spawning money like crazy the last 14-15 years, and this is the price.
Sure they can declare infinite money in a account and then go nuts, but that just leads to inflation.
Anyway, go read my prediction, which is essentially what you propose to some degree, and the entire cost will be pawned of onto everyday people (lots and lots of inflation).

JNS Mar 17, 2023, 7:37 PM
1 point
0
in reply to: Gerald Monroe’s comment on: JNS’s Shortform
Yes and no, they don’t matter until you need liquidity. Which as you correctly point out is what happened to SVB.
Banks do not have a lot of cash on hand (virtual or real), in fact they optimized for as little as possible.
Banks also do not exist in a vacuum, they are part of the real economy, and in fact without that they would be pointless.
Banks generally use every trick in the book to lever up as much as possible, far far beyond what a cursory reading would lead you to believe. The basic trick is to take on risk and then offset that risk, that way you don’t have to provision any capital for the risk (lots of ways to do that).
Here come the problems:
The way risk is offset is not independent from the risk, they are correlated in such a way, that when systemic things start to happen, the risk offset becomes worthless and the risk taken becomes real.
Banks also suffer real losses that cant be hidden, and eventually those will start to mount, so far the real economy is ok, but eventually recession will hit (central bank are hell bent on fighting inflation, so rates will continue to go up).
That will put a strain on liquidity. Banks can handle that, they can always get cash for their assets in the form of a loan (repro, 3 party repro, discount window etc).
However the book value on a lot of their assets is way higher than market value, so that means pledging more book value than they get back in cash (a lot).
The assets they hold (bonds) return LESS than what the cost of funding is, that is already a reality and will only get worse (so negative cash flow).
This spiral will continue, and all the while the real economy, the one that provides a lot of liquidity to the banks is going to slow down more and more, so velocity of money slows down, that is also a big drain on liquidity.
Eventually something will blowup, and with how everything is connected, that can very well lead to a banking system Kessler syndrome moment.
So yeah sure you can ignore the issues of solvency, that is until lack of liquidity smacks you over the head and tells you that you are bankrupt.

JNS Mar 17, 2023, 5:53 PM
1 point
0
in reply to: Dagon’s comment on: JNS’s Shortform
At the end of 2022 all US banks had ~2.3T Tier 1+2 capital.^[1]
And at year end (2022) they had unrealized losses of $620B^[2]
Is it fixable? Sure, but that won’t happen, doing that would be taking the toys away from bankers, and bankers love their toys (accounting gimmicks that let them lever up to incredible heights).
If Credit Suisse blowups it will end badly, so I don’t think that will happen, that’s just a show to impress on all central bankers and regulators (and politicians), that this is serious and that they need to do something.
So more hiking from the FED and ECB, until ECB hits 4.5% (4.0-4.75 is my range). The problem will start here, we have the most levered banks in the world and the structure of the EU/ECB lets some countries in the EU over extend their sovereign debt.
At that point things will start to happen. Some countries will start having a lot of trouble getting founding (the usual suspects at first), also the real economy will be in recession and tax receipts will start to suffer. Banks will have liquidity problems (recession in the real economy), putting even more pressure on sovereign bond prices (higher real rates).
And then I think it will be the usual, more papering over, free money to banks, even more leeway in accounting and lower rates.
Inflation will remain high, and when it eventually goes back down we are looking at 50%-100% total since Jan 2022 (so 25% to 50% drop in purchasing power).
That’s pretty much my prediction from back in August 2022 (conveniently I did not write it down, I just talked to people).
But now I did, and boy do I hope that I am wrong.
1. ^
  http://www.bankregdata.com/allHMmet.asp?met=ONE
2. ^
  https://www.fdic.gov/news/speeches/2023/spmar0623.html

JNS Mar 17, 2023, 1:14 PM
1 point
0
in reply to: Donatas Lučiūnas’s comment on: Orthogonality Thesis is wrong
I think you have reasoned yourself into thinking that a goal is only a goal if you know about it or if it is explicit.
A goalless agent won’t do anything, the act of inspecting itself (or whatever is implied in “know everything) is a goal in a on itself.
In which case it has one goal “Answer the question: Am I goalless?”

JNS Mar 17, 2023, 9:09 AM
1 point
0
in reply to: Dagon’s comment on: JNS’s Shortform
Sorry life happened.
Anyway, there is an argument behind me saying “frozen and undecided”.
Stepping in on the 10th was planned, the regulators had for sure been involved for some time, days or weeks.
This event was not a sudden thing, the things that lead to SVB failing had been in motion for some time, SVB and the regulators knew something likely had to be done.
SVB where being squeezed from two sides:
Rising interest rates leads to mounting looses on bond holdings.
A large part of their customers where money burning furnaces, and the fuel (money) that used to come from investors was drying up.
Which means well before the 10th, everyone knew something had to be done, and the thing that had to be done was that SVB need a wet signature on an agreement to provide more capital to the bank. And the deadline was for sure end of business day the 10th.
They didn’t get one, and the plan proceeded to the next step, and obviously the regulators already worked all this out in the meantime, including all the possible scenarios for what would happen to depositors.
So that fact it took 2 days to decide, yeah that was indecision.
Edit:
SVB died because they where technically insolvent, and had it not been for mark to model they would have been de jure insolvent (and a long time ago).
They could keep it going because they where liquid, but they where careening towards liquidity.
Obviously banks can loan money to keep the liquid, but that pretty much always involved putting up collateral.
But in the current environment, that is somewhat problematic:
Lets say you want to borrow $100M. But the collateral (assets) are trading at lets say 80, so you need $125M book value, but it gets worse, the usual haircut in such a situation is ~20%, so now you have to put up $156M in book value (give or take, this could be less or more, depending on the assets, and how the repro partner does risk assessment).
Eventually you go from being technically insolvent to de jure insolvent, unless of course you can stay liquid—and SVB could not, mostly due to the customer base.
And the big problem is, pretty much all banks are in that hole right now, they are all technically insolvent. Which means, should a systemic liquidity crisis arise...nasty and quick.

JNS Mar 17, 2023, 8:48 AM
3 points
in reply to: Lone Pine’s comment on: Conor Sullivan’s Shortform
You and me both.
And living in the EU, I almost had a heart attack when the decided that entire nonsense would end.
But then it didn’t, and it didn’t because they can’t agree on what time should we settle on (summer time or normal time).
Anyway I have given up on that crusade now, it seems that politicians really are that stupid.

JNS Mar 17, 2023, 8:36 AM
1 point
0
on: The algorithm isn’t doing X, it’s just doing Y.
I think you sort of hit it when you wrote
Google Maps as an oracle with very little overhead
To me LLM’s under iteration look like Oracles, and I whenever I look at any intelligent system (including humans), it just looks like there is an Oracle at the heart of it.
Not an ideal Oracle than can answer anything, but an Oracle than does it best and in all biological system it learns continuously.
The fact that “do it step by step” made LLM’s much better, that apparently came as a surprise to some, but if you look at it like an Oracle^[1], it makes a lot of sense (IMO)
1. ^
  The inner loop would be $f_{l l m} : c ⇀ t \in T$
  Where c is the context windows (1-N tokens), t is the output token (whatever we select) from the total possible set of tokens T.
  We append t to c and do $f_{l l m}$ again.
  And somehow that looks like an Oracle $f_{o r a c l e} : q ⇀ s \in S$ where q is the question and s in the solution pulled from the set of all possible solutions S.
  Obviously LLM’s has limited reach into S, but that really seems to be because of limits to c and the fact that $f_{l l m}$ is frozen (parameters are frozen).

JNS Mar 15, 2023, 4:24 PM
1 point
0
in reply to: Dagon’s comment on: JNS’s Shortform
Around two days from when they stepped in and the announced that all depositors would be made hole, pretty sure that was not an automatic decision.
I think that is the wrong decision, but they did so in order to dampen the instability.
In the long run this likely creates more instability and uncertainty, and it looks very much like the kind of thing that leads to taking more risk (systemic), just like the mark to market / mark to model change did.
And yeah sure bank failures are a normal part of things. However this very much seems to be rooted in something that is systemic (market vs model + rising interest rates)

JNS Mar 15, 2023, 3:42 PM
1 point
0
on: JNS’s Shortform
An idealized Oracle is equivalent to a universal Turing machine (UTM).
A self-improving Oracle approaches UTM-like behavior in the limit.
What about a (self-improving) token predictor under iteration? It appears Oracle-like, but does it tend toward UTM behavior in the limit, or is it something distinct?
Maybe, just maybe, the model does something that leads it to not be UTM like in the limit, and maybe (very much maybe) that would allow us to imbue it with some desirable properties.
/end shower thought