David Scott Krueger (formerly: capybaralet)(David Krueger)

Karma: 1,841

I’m more active on Twitter than LW/AF these days: https://twitter.com/DavidSKrueger

Bio from https://www.davidscottkrueger.com/:
I am an Assistant Professor at the University of Cambridge and a member of Cambridge’s Computational and Biological Learning lab (CBL). My research group focuses on Deep Learning, AI Alignment, and AI safety. I’m broadly interested in work (including in areas outside of Machine Learning, e.g. AI governance) that could reduce the risk of human extinction (“x-risk”) resulting from out-of-control AI systems. Particular interests include:

Reward modeling and reward gaming
Aligning foundation models
Understanding learning and generalization in deep learning and foundation models, especially via “empirical theory” approaches
Preventing the development and deployment of socially harmful AI systems
Elaborating and evaluating speculative concerns about more advanced future AI systems

David Scott Krueger (formerly: capybaralet) 26 Jan 2015 5:09 UTC
0 points
in reply to: loup-vaillant’s comment on: Q&A with Jürgen Schmidhuber on risks from AI
It’s at least partially a matter of how quick that actually is. Consider that the world is a big place, and there are currently significant power differentials.

There might be all sorts of practical issues that an AGI that lacks physical means could stumble on.

The whole scenario is highly dependent on what technologies of robotics exist, what sort of networks are in place, etc.

David Scott Krueger (formerly: capybaralet) 26 Jan 2015 5:17 UTC
−2 points
on: Q&A with Jürgen Schmidhuber on risks from AI
Does anyone know how big the constants involved in AIXI and implementations/modifications like Schmidhuber’s are?

I seriously question if the implications of “almost all imaginable problems are big enough to make the additive constant negligible” are in fact (paraphrasing): “Goedel machines will rule the universe”

David Scott Krueger (formerly: capybaralet) 26 Jan 2015 5:23 UTC
5 points
on: Bill Gates: problem of strong AI with conflicting goals “very worthy of study and time”
I found it interesting that he doesn’t think we should stop or slow down, but associates his position with Bill Joy, the author of “Why the Future Doesn’t Need Us” (2000), which argued for halting research in genetics, nanotech and robotics.

David Scott Krueger (formerly: capybaralet) 26 Jan 2015 5:32 UTC
6 points
on: Q&A with Shane Legg on risks from AI
I disagree with his “chicken and egg” response to Q4.

Working towards a “concrete and grounded understanding of AGI” is not the same as trying to build an AGI. We can, e.g., understand a design space, verify that it is safe, and then explore within that space.

David Scott Krueger (formerly: capybaralet) 26 Jan 2015 6:02 UTC
5 points
in reply to: poke’s comment on: Is That Your True Rejection?
Which transhumanist ideas are “not even wrong”?

And do you mean simply ‘not well specified enough’? Or more like ‘unfalsifiable’?

You also seem to be implying that scientists cannot discuss topics outside of their field, or even outside its current reach.

My philosophy on language is that people can generally discuss anything. For any words that we have heard (and indeed, many we haven’t), we have some clues as to their meaning, e.g. based on the context in which they’ve been used and similarity to other words.

Also, would you consider being cautious an inherently good thing?

Finally, from my experience as a Masters student in AI, many people are happy to give opinions on transhumanism, it’s just that many of those opinions are negative.

A Somewhat Vague Proposal for Grounding Ethics in Physics

David Scott Krueger (formerly: capybaralet)27 Jan 2015 5:45 UTC

−3 points

15 comments1 min readLW link

A Basic Problem of Ethics: Panpsychism?

David Scott Krueger (formerly: capybaralet)27 Jan 2015 6:27 UTC

−5 points

16 comments1 min readLW link

David Scott Krueger (formerly: capybaralet) 27 Jan 2015 6:32 UTC
0 points
on: Giulio Tononi’s “Integrated Information Theory” of Consciousness
One potential issue with the theory in practice is that it requires calculating the integration (or whatever it is called) for EVERY sub-system and super-system of a given system in order to determine if it is conscious.

David Scott Krueger (formerly: capybaralet) 27 Jan 2015 6:43 UTC
−1 points
on: The mystery of pain and pleasure
These are great questions. I’m not sure they have answers. But they seem extremely pertinent to making a good AGI.

Tegmark’s paper here: http://arxiv.org/pdf/1409.0813.pdf seems to be poking in the same direction.

Neglecting these questions is, IMO, tantamount to moral relativism or nihilism.

David Scott Krueger (formerly: capybaralet) 28 Jan 2015 2:59 UTC
0 points
in reply to: Viliam_Bur’s comment on: A Somewhat Vague Proposal for Grounding Ethics in Physics

This sounds wrong.

Of course. That’s why I proposed refining it.

But I am not even sure what exactly would we measure here

I thought it was obvious. It is the integral of total experience (suitably defined) through time that counts.

David Scott Krueger (formerly: capybaralet) 28 Jan 2015 3:01 UTC
0 points
in reply to: 9eB1’s comment on: A Somewhat Vague Proposal for Grounding Ethics in Physics

Any universe-wide configuration of particles contains the history of all of the events preceding it

So there is no way that we can arrive at the same state from different starting points? That seems ridiculous to me.

David Scott Krueger (formerly: capybaralet) 28 Jan 2015 3:08 UTC
2 points
in reply to: joaolkf’s comment on: A Somewhat Vague Proposal for Grounding Ethics in Physics
Yeah I am not happy about the way I’m being received. Any advice, other than avoiding interesting meta-ethics questions?

Wrt how new it is: how about if I put it this way:

Maybe experience is fundamentally not a function of brain state, but a function of brain state over time. Note that this is not strongly anti-physicalism. Especially if you believe in discrete time, in which case you can have experience be a function of the transitions that occur between states in successive time-steps:

Experience = f(s{t}, s{t-1}).

David Scott Krueger (formerly: capybaralet) 28 Jan 2015 3:10 UTC
−3 points
in reply to: jmmcd’s comment on: A Somewhat Vague Proposal for Grounding Ethics in Physics
Maybe tell me why I should? My time is valuable.

David Scott Krueger (formerly: capybaralet) 28 Jan 2015 3:18 UTC
0 points
in reply to: John_Maxwell’s comment on: A Basic Problem of Ethics: Panpsychism?
What’s the best argument you’ve seen for it?

see skeptical lurker’s comment, below.

What do you think of the reductionism sequence?

While I don’t have too much experience to back this up, I think it is probably a lot of things I’m familiar with, elaborated at length, with perhaps a few insights sprinkled in. Can you please give brief summaries of the things you link to, and how they are relevant? I skimmed that article, and it doesn’t seem relevant.

If something doesn’t have preferences, I’m not that concerned about it.

My point is: how do you evaluate if something has preferences? How do you disambiguate preferences from statements like “I prefer __”? Clearly we DO distinguish between these. If I write and run the following computer program, I don’t think you will be upset if I stop it :
```
while True:

print "I prefer not to be interrupted"
```

David Scott Krueger (formerly: capybaralet) 28 Jan 2015 3:28 UTC
−1 points
in reply to: Manfred’s comment on: A Basic Problem of Ethics: Panpsychism?
Unless I am mistaken, the best theory of how to make FAI ultimately points back to my (and all y’all’s) ideas and preferences.

So I guess we should taboo FAI.

I’d argue that you have no better idea of what physical properties correspond to consciousness than I do, you’ve simply chosen to ignore the question, because you believe you can rely on your own intuitive consciousness-detector.

I am worried about bias. Shouldn’t we all be?

David Scott Krueger (formerly: capybaralet) 28 Jan 2015 4:43 UTC
0 points
in reply to: skeptical_lurker’s comment on: A Basic Problem of Ethics: Panpsychism?
How do you know what a CEV AI should do?

How do you know that squids don’t have emotions?

Define agency.

You could have at least stepped up to the challenge you left to the reader.

David Scott Krueger (formerly: capybaralet) 28 Jan 2015 4:54 UTC
0 points
in reply to: John_Maxwell’s comment on: A Basic Problem of Ethics: Panpsychism?
I know all about philosophical zombies.

Agreed. I don’t have any easy answer to this question.

Do you have any answer at all? Or anything to say on the matter? Would you at least agree that it is of critical ethical importance, and hence worthy of discussion?

And it’s possible that you and I would disagree on how to carve reality in to that which has preferences we consider meaningful vs that which doesn’t.

Of course, but I assume you agree with me about the program I wrote?

In any case, I think it would be nice to try and forge some agreement and/or understanding on this matter (as opposed to ignoring it on the basis of our disagreement).

David Scott Krueger (formerly: capybaralet) 21 Aug 2015 17:35 UTC
0 points
in reply to: jmmcd’s comment on: A Somewhat Vague Proposal for Grounding Ethics in Physics
I put “enjoy itself” in quotes, because I don’t mean it literally. The questions that that sequence addresses according to the summary don’t seem relevant to what I am trying to get at.

I guess I need to be more precise. I just mean how can we maximize the integral of experience through time (whether we let experience take negative values is a detail). This was one of Tegmark’s proposals in that paper, already, except he is writing in terms of a final goal instead of a process, which was the point of my post...

“The amount of consciousness in our Universe, which Giulio Tononi has argued corresponds to integrated information”

David Scott Krueger (formerly: capybaralet) 7 Sep 2015 14:01 UTC
0 points
on: Completeness, incompleteness, and what it all means: first versus second order logic
“Every set of numbers has a least element” clearly does NOT define the natural numbers. Consider N U {-1}.

David Scott Krueger (formerly: capybaralet) 19 Sep 2015 19:24 UTC
0 points
in reply to: jacob_cannell’s comment on: MIRI’s Approach
While the methods for finding a solution to a well-formed problem currently used in Machine Learning are relatively well understood, the solutions found are not.

And that is what really matters from a safety perspective. We can and do make some headway in understanding the solutions, as well, but the trend is towards more autonomy for the learning algorithm, and correspondingly more opaqueness.

As you mentioned, the solutions found are extremely complex. So I don’t think it makes sense to view them only in terms of approximations to some conceptually simple (but expensive) ideal solution.

If we want to understand their behaviour, which is what actually matters for safety, we will have to grapple with this complexity somehow.

Personally, I’m not optimistic about experimentation (as it is currently practiced in the ML community) being a good enough solution. There is, at least, the problem of the treacherous turn. If we’re lucky, the AI jumps the gun, and society wakes up to the possibility of an AI trying to take over. If we’re unlucky, we don’t get any warning, and the AI only behaves for long enough to gain our trust and discover a nearly fail-proof strategy. VR could help here, but I think it’s rather far from a complete solution.

BTW, SOTA for Computer Go uses ConvNets (before that, it was Monte-Carlo Tree Search, IIRC): http://machinelearning.wustl.edu/mlpapers/paper_files/icml2015_clark15.pdf ;)

David Scott Krueger (formerly: capybaralet)(David Krueger)

A Some­what Vague Pro­posal for Ground­ing Ethics in Physics

A Ba­sic Prob­lem of Ethics: Panpsy­chism?

A Somewhat Vague Proposal for Grounding Ethics in Physics

A Basic Problem of Ethics: Panpsychism?