quanticle

Karma: 2,747

quanticle Apr 20, 2023, 3:33 AM
2 points
0
in reply to: Oliver Siegel’s comment on: Consciousness is irrelevant—instead solve alignment by asking this question

We’ve taught AI how to speak, and it appears that openAI has taught their AI how to produce as little offensive content as possible.

The problem is that the AI can (and does) lie. Right now, ChatGPT and its ilk are a less than superhuman levels of intelligence, so we can catch their lies. But when a superhuman AI starts lying to you, how does one correct for that? If a superhuman AI starts veering off in a direction that is unexpected, how does one bring it back on track?

@gwern short story, Clippy highlights many of the issues with naively training a superintelligent algorithm on human-generated data and expecting that algorithm to pick up human values as a result. Another post to consider is The Waluigi Effect, which raises the possibility that the more you train an agent to say correct, inoffensive things, the more you’ve also trained a shadow-agent to say incorrect, offensive things.

quanticle Apr 19, 2023, 4:28 AM
2 points
0
in reply to: YafahEdelman’s comment on: Excessive AI growth-rate yields little socio-economic benefit.
How would you measure the usage? If, for example, Google integrates Bard into its main search engine, as they are rumored to be doing, would that count as usage? If so, I would agree with your assessment.

However, I disagree that this would be a “drastic” impact. A better Google search is nice, but it’s not life-changing in a way that would be noticed by someone who isn’t deeply aware of and interested in technology. It’s not like, e.g. Google Maps navigation suddenly allowing you to find your way around a strange city without having to buy any maps or decipher local road signs.

quanticle Apr 18, 2023, 7:34 PM
11 points
2
in reply to: electroswing’s comment on: Attributes of successful professors
What I’m questioning is the implicit assumption in your post that AI safety research will inevitably take place in an academic environment, and therefore productivity practices derived from other academic settings will be helpful. Why should this be the case when, over the past few years, most of the AI capabilities research has occurred in corporate research labs?

Some of your suggestions, of course, work equally well in either environment. But not all, and even the ones which do work would require a shift in emphasis. For example, when you say professors should be acquainted with other professors, that’s valid in academia, where roughly everyone who matters either has tenure or is on a tenure track. However, that is not true in a corporate environment, where many people may not even have PhDs. Furthermore, in a corporate environment, limiting one’s networking to just researchers is probably ill advised, given that there are many other people who would have influence upon the research. Knowing a senior executive with influence over product roadmaps could be just as valuable, even if that executive has no academic pedigree at all.

Prioritizing high value research and ignoring everything else is a skill that works in both corporate and academic environments. But 80/20-ing teaching? In a corporate research lab, one has no teaching responsibilities. One would be far better served learning some basic software engineering practices, in order to better interface with product engineers. Similarly, with regards to publishing, for a corporate research lab, having a working product is worth dozens of research papers. Research papers bring prestige, but they don’t pay the bills. Therefore, I would argue that AI safety researchers should be keeping an eye on how their findings can be applied to existing AI systems. This kind of product-focused development is something that academia is notoriously bad at.

I also question your claim that academic bureaucracy doesn’t slow good researchers down very much. That’s very much not in line with what anecdotes I’ve heard. From what I’ve seen, writing grant proposals, dealing with university bureaucracy, and teaching responsibilities are a significant time suck. Maybe with practice and experience, it’s possible for a good researcher to complete these tasks on “autopilot”, and therefore not notice the time that’s being spent. But the tasks are still costing time and mental energy that, ideally, would be devoted to research or writing.

I don’t think it’s inevitable that academia will take over AI safety research, given the trend in AI capabilities research, and I certainly don’t think that academia taking over AI safety research would be a good thing. For this reason I question whether it’s valuable for AI safety researchers to develop skills valuable for academic research, specifically, as opposed to general time management, software engineering and product development skills.

quanticle Apr 18, 2023, 5:23 AM
2 points
0
in reply to: Raemon’s comment on: AutoBound on neural network can achieve OOMs lower training loss
What is the purpose, beyond mere symbolism, of hiding this post to logged out users when the relevant data is available, in far more detail, on Google’s official AI blog?

quanticle Apr 17, 2023, 6:03 AM
11 points
8
in reply to: electroswing’s comment on: Attributes of successful professors
I am saying that successful professors are highly successful researchers

Are they? That’s why I’m focusing on empirics. How do you know that these people are highly successful researchers? What impressive research findings have they developed, and how did e.g. networking and selling their work enable them to get to these findings? Similarly, with regards to bureaucracy, how did successfully navigating the bureaucracy of academia enable these researchers to improve their work?

The way it stands right now, what you’re doing is pointing at some traits that correlate with academic success, and are claiming that
1. Aspiring to the standards of prestigious academic institutions will speed up AI safety research
2. Researchers at prestigious academic institutions share certain traits
3. Therefore adopting these traits will lead to better AI safety research
This reasoning is flawed. First, why should AI safety research aspire to the same standards of “publish or perish” and the emphasis on finding positive results that gave us the replication crisis? It seems to me that, to the greatest extent possible, AI safety research should reject these standards, and focus on finding results that are true, rather than results that are publishable.

Secondly, correlation is not causation. The fact that many researchers from an anecdotal sample share certain attributes doesn’t mean that those attributes are causative of those researchers’ success. There are lots of researchers who do all of the things that you describe, managing their time, networking aggressively, and focusing on understanding grantmaking, who do not end up at prestigious institutions. There are lots of researchers who do all of those things who don’t end up with tenure at all.

This is why I’m so skeptical of your post. I’m not sure that the steps your take are actually causative of academic success, rather than merely correlating with academic success, and furthermore, I’m not even sure that the standards of academic success are even something that AI safety research should aspire to.

quanticle Apr 16, 2023, 12:31 PM
2 points
0
on: The Truth About False

Well, augmenting reality with an extra dimension containing the thing that previously didn’t exist is the same as “trying and seeing what would happen.” It worked swimmingly for the complex numbers.

No it isn’t. The difference between $i$ and the values returned by $M a v e r i c k ()$ is that $i$ can be used to prove further theorems and model phenomena, such as alternating current, that would be difficult, if not impossible to model with just the real numbers. Whereas positing the existence of $M a v e r i c k ()$ is just like positing the existence of a finite value $h$ that satisfies $h = \frac{1}{0}$ . We can posit values that satisfy all kinds of impossibilities, but if we cannot prove additional facts about the world with those values, they’re useless.

quanticle Apr 16, 2023, 5:47 AM
28 points
30
in reply to: RobertM’s comment on: Moderation notes re: recent Said/Duncan threads
For what it’s worth, I had a very similar reaction to yours. Insects and arthropods are a common source of disgust and revulsion, and so comparing anyone to an insect or an arthropod, to me, shows that you’re trying to indicate that this person is either disgusting or repulsive.

quanticle Apr 16, 2023, 12:43 AM
2 points
−1
in reply to: SCP’s comment on: What’s this probability you’re reporting?

Probabilities as credences can correspond to confidence in propositions unrelated to future observations, e.g., philosophical beliefs or practically-unobservable facts. You can unambiguously assign probabilities to ‘cosmopsychism’ and ‘Everett’s many-worlds interpretation’ without expecting to ever observe their truth or falsity.

You can, but why would you? Beliefs should pay rent in anticipated experiences. If two beliefs lead to the same anticipated experiences, then there’s no particular reason to choose one belief over the other. Assigning probability to cosmopsychism or Everett’s many-worlds interpretation only makes sense insofar as you think there will be some observations, at some point in the future, which will be different if one set of beliefs is true versus if the other set of beliefs is true.

quanticle Apr 16, 2023, 12:22 AM
13 points
10
in reply to: electroswing’s comment on: Attributes of successful professors

One crude way of doing it is saying that a professor is successful if they are a professor at a top 10-ish university.

But why should that be the case? Academia is hypercompetitive, but the way it selects is not solely on the quality of one’s research. Choosing the trendiest fields has a huge impact. Perhaps the professors that are chosen by prestigious universities are the ones that the prestigious universities think are the best at drawing in grant money and getting publications into high-impact journals, such as Nature, or Science.

Specifically I think professors are at least +2σ at “hedgehog-y” and “selling work” compared to similarly intelligent people who are not successful professors, and more like +σ at the other skills.

How does one determine this?

Overall, it seems like your argument is that AI safety researchers should behave more like traditional academia for a bunch reasons that have mostly to do with social prestige. While I don’t discount the role that social prestige has to play in drawing people into a field and legitimizing it, it seems like overall, the pursuit of prestige has been a net negative for science as a whole, leading to, for example, the replication crisis in medicine and biology, or the nonstop pursuit of string theory over alternate hypotheses in physics. Therefore, I’m not convinced that importing these prestige-oriented traits from traditional science would be a net positive for AI safety research.

Furthermore, I would note that traditional academia has been moving away from these practices, to a certain extent. During the early days of the COVID pandemic, quite a lot of information was exchanged not as formal peer-reviewed research papers, but as blog posts, Twitter threads, and preprints. In AI capabilities research, many new advances are announced as blog posts first, even if they might be formalized in a reseach paper later. Looking further back in the history of science, James Gleick, in Chaos relates how the early researchers into chaos and complexity theories did their research by informally exchanging letters and draft papers. They were outside the normal categories that the bureaucracy of academia had established, so no journal would publish them.

It seems to me that the foundational, paradigm-shifting research always takes place this way. It takes place away from the formal rigors of academia, in informal exchanges between self-selected individuals. Only later, once the core paradigms of the new field have been laid down, does the field become incorporated into the bureaucracy of science, becoming legible enough for journals to routinely publish findings from the new field. I think AI safety research is at this early stage of maturity, and therefore it doesn’t make sense for it to import the practices that would help practitioners survive and thrive in the bureaucracy of “Big Science”.

quanticle Apr 14, 2023, 9:11 PM
2 points
0
on: What’s this probability you’re reporting?
I was thinking more about the inside view/outside view distinction, and while I agree with Dagon’s conclusion that probabilities should correspond to expected observations and expected observations only, I do think there is a way to salvage the inside view/outside view distinction. That is to treat someone saying, “My ‘inside view’ estimate of event $E$ is $X$ ,” as being equivalent to someone saying that $P (E | model is correct) = X$ . It’s a conditional probability, where they’re telling you what their probability of a given outcome is, assuming that their understanding of the situation is correct.

In the case of deterministic models, this might seem like a tautology — they’re telling you what the outcome is, assuming the validity of a process that deterministically generates that outcome. However, there is another source of uncertainty: observational uncertainty. The other person might be uncertain whether they have all the facts that feed into their model, or whether their observations are correct. So, in other words, when someone says, “My inside view probability of $E$ is $X$ ,” that’s a statement about the confidence level they have in their observations.

quanticle Apr 14, 2023, 7:37 PM
14 points
12
on: Attributes of successful professors
I agree that all of these attributes are plausible attributes of successful professors. However, I’d still like to know where you’re drawing these observations from? Is it personal observation? And if so, how have you determined whether a professor is successful or not? Is there a study that correlates academic impact across these traits?

quanticle Apr 14, 2023, 7:25 PM
2 points
1
in reply to: Dagon’s comment on: What’s this probability you’re reporting?
I agree that the betting approach is better at clarification, but the problem is that it’s often too much better. For example, if I say, I’ll bet $10 at 80% odds that the weather tomorrow will be sunny, the discussion rapidly devolves into the definitional question of what is a sunny day, exactly? Do I win if I see the sun at any point in the day? Is there a certain amount of cloud cover at which point the day no longer counts as sunny? Where is the cloud cover measured from? If the sky starts out with < 5% clouds, clouds over to > 50%, but then the clouds clear later in the day, does the day still count as “sunny”? Etc.

Sometimes I want to make a certain probability judgement about an outcome defined by a colloquially understood category (such as “sunny day”) without having to precisely specify all of my definitions exactly.

quanticle Apr 14, 2023, 7:14 AM
4 points
2
on: AI shouldn’t be aligned with human morality

We value morality because of evolution. Not because its rational.

Why are those two things mutually exclusive? We understand that $a^{2} + b^{2} = c^{2}$ is true for the legs of a right triangle, because we have brains that are the result of evolution. Does that make the Pythagorean Theorem “irrational” or untrue, somehow?

quanticle Apr 13, 2023, 11:04 PM
2 points
0
in reply to: Kaj_Sotala’s comment on: [Link] Sarah Constantin: “Why I am Not An AI Doomer”

This seems to make a jump from “the prompt requires agency to execute well” to “the AI develops the cognitive capability for agency”?

In my scenario the AI already has the cognitive capability for agency. It’s just that the capability is latent until the right prompt causes it to be expressed. We’ve seen early examples of this with ChatGPT, where, if you ask it to plan something or think about adversarial scenarios, it will demonstrate agent-ish behavior.

My point is that while current AIs are probably incapable of having agency, future AIs probably will have that capability. Furthermore, we may not be able to tell the difference between an AI that is capable of building a world-model and engaging in long-term goal directed behavior and the current AI systems that mostly aren’t.

quanticle Apr 12, 2023, 11:20 AM
18 points
11
on: [Link] Sarah Constantin: “Why I am Not An AI Doomer”
I feel like a lot of the objections around agency are answered by the Clippy scenario, and gwern’s other essay on the topic, Tool AIs want to be Agent AIs. The AGI need not start with any specific goal or agency. However, the moment it starts executing a prompt that requires it to exhibit agency or goal directed behavior, it will. And at that point, unless the goal is set up such that the agent pursues its goal in a manner that is compatible with the continued existence of humanity over the long term, humanity is doomed. Crafting a goal in this manner is very difficult, and making sure that the AGI pursues this goal and no others are both very difficult tasks individually. Together, they are nigh impossible. Thus, with a very strong likelihood, the moment the AGI either receives a prompt or discovers a prompt that requires it to behave like an agent, humanity is doomed.

I agree that AGIs need to possess a world model, but I disagree that we will be able to distinguish an AI that possesses a world model from an AI that “merely” knows word associations. The internals of an AI are opaque, despite the best efforts of interpretability research to shine light on the giant inscrutable matrices. An AI with a world model, I predict, won’t look much different from an AI without a world model. Maybe some weights will be different, and some update functions will have changed. Will we be able to point to any specific weight or combination of weights and say, “Aha, the AI has developed a world model!” Probably not, no more than we can look at any specific set of neurons in the human brain and say, “Aha, there lies the seat of consciousness!”

Given the two points above, we may not be able to tell when any given AI passes the threshold to becoming an AGI. And once an AI has passed the threshold, we won’t necessarily be able to control which prompt causes the AI to begin simulating an agent. Given those two, I fail to see why we shouldn’t behave as if AGI is on a short timeline. After all, if one is approaching a cliff from an unknown distance in the darkness, the wise thing to do is not to assume that the cliff is still miles away and stride boldly into the unknown. Instead it behooves us to probe carefully, trying to determine whether there’s solid ground or empty space ahead.

quanticle Apr 9, 2023, 10:22 AM
2 points
−5
in reply to: tailcalled’s comment on: The surprising parameter efficiency of vision models

I think LLMs are great and plausibly superhuman at language

I think the problem might be that “language” encompasses a much broader variety of tasks than image generation. For example, generating poetry with a particular rhyming structure or meter seems to be a pretty “pure” language task, yet even GPT-4 struggles with it. Meanwhile, diffusion models with a quarter of the parameter count of GPT-4 can output art in a dizzying variety of styles, from Raphael-like neoclassical realism to Picasso-like cubism.

quanticle Apr 9, 2023, 10:15 AM
3 points
1
in reply to: anonymousaisafety’s comment on: The surprising parameter efficiency of vision models
Okay, that’s all fair, but it still doesn’t answer my question. We don’t do any of these things for diffusion models that output images, and yet these diffusion models manage to be much smaller than models that output words, while maintaining an even higher level of output quality. What is it about words that makes the task different?

Or are you suggesting that image generators could also be greatly improved by training minimal models, and then embedding those models within larger networks?

quanticle Apr 9, 2023, 6:13 AM
3 points
−1
in reply to: anonymousaisafety’s comment on: The surprising parameter efficiency of vision models
That’s a fair criticism, but why would it apply to only language models? We also train visual models with a randomized curriculum, and we seem to get much better results. Why would randomization hurt training efficiency for language generation but not image generation?

quanticle Apr 9, 2023, 5:24 AM
4 points
2
in reply to: dsj’s comment on: Can we evaluate the “tool versus agent” AGI prediction?
On the flip side, as gwern pointed out in his Clippy short story, it’s possible for a “neutral” GPT-like system to discover agency and deception in its training data and execute upon those prompts without any explicit instruction to do so from its human supervisor. The actions of a tool-AI programmed with a more “obvious” explicit utility function is easier to predict, in some ways, than the actions of something like ChatGPT, where the actions that it’s making visible to you may be a subset (and a deliberately deceptively chosen subset) of all the actions that it is actually taking.

quanticle Apr 8, 2023, 3:40 AM
3 points
1
on: ChatGTP “Writing ” News Stories for The Guardian?
It’s not that odd. Ars Technica has a good article on why generative AIs have such a strong tendency to confabulate. The short answer is that, given a prompt (consisting of tokens, which are similar to, but not quite the same as words), GPT will come up with new tokens that are more or less likely to come after the given tokens in the prompt. This is subject to a temperature parameter, which dictates how “creative” GPT is allowed to be (i.e. allowing GPT to pick less probable next-tokens with some probability). The output token is added to the prompt, and the whole thing is then fed back into GPT in order to generate another new token.

In other words, GPT is incapable of “going backwards”, as a human might and editing its previous output in to correct inaccuracies or inconsistencies. Instead, what it has to do is take the previous output as a given, and try to come up with new tokens that are likely to be generated given the already generated incorrect tokens. This is how GPT ends up with confabulated citations. Given the prompt, GPT generates some tokens, representing an author, for example. It then tries to generate the most likely words associated with that author, and the rest of the prompt, which is presumably asking for citations. As it generates a title, it chooses a word that doesn’t exist in any existing article titles written by that author. But it doesn’t “know” that, and it has no way of going back and editing prior output in order to correct itself. Instead GPT presses on, generating more tokens that are deemed to be likely given the mixture of correct and incorrect tokens that it has generated.

Scott Alexander has a great post, about human psychology, which touches on a similar theme, called The Apologist and the Revolutionary. Using the terms of that post, a GPT is 100% apologist, 0% revolutionary. No matter how nonsensical its previous output, GPT, by its very design, must take that previous output as axiomatic, and generate new output based upon that. That is what leads to uncanny results when GPT is asked for specific facts.