Prometheus

• PrometheusFeb 21, 2024, 11:50 PM

  5 points

  1

  in reply to: Thane Ruthenis’s comment on: Cur­rent AIs Provide Nearly No Data Rele­vant to AGI Alignment

  I dislike the overuse of analogies in the AI space, but to use your analogy, I guess it’s like you keep assigning a team of engineers to build a car, and two possible things happen. Possibility One: the engineers are actually building car engines, which gives us a lot of relevant information for how to build safe cars (toque, acceleration, speed, other car things), even if we don’t know all the details for how to build a car yet. Possibility Two: they are actually just building soapbox racers, which doesn’t give us much information for building safe cars, but also means that just tweaking how the engineers work won’t suddenly give us real race cars.

• PrometheusFeb 21, 2024, 11:31 PM

  3 points

  2

  on: Prometheus’s Shortform

  If progress in AI is continuous, we should expect record levels of employment. Not the opposite.

  My mentality is if progress in AI doesn’t have a sudden, foom-level jump, and if we all don’t die, most of the fears of human unemployment are unfounded… at least for a while. Say we get AIs that can replace 90% of the workforce. The productivity surge from this should dramatically boost the economy, creating more companies, more trading, and more jobs. Since AIs can be copied, they would be cheap, abundant labor. This means anything a human can do that an AI still can’t becomes a scarce, highly valued resource. Companies with thousands or millions of AI instances working for them would likely compete for human labor, because making more humans takes much longer than making more AIs. Then say, after a few years, AIs are able to automate 90% of the remaining 10%. Then that creates even more productivity, more economic growth, and even more jobs. This could continue for even a few decades. Eventually, humans will be rendered completely obsolete, but by that point (most) of them might be so filthy rich that they won’t especially care.

  This doesn’t mean it’ll all be smooth-sailing or that humans will be totally happy with this shift. Some people probably won’t enjoy having to switch to a new career, only for that new career to be automated away after a few years, and then have to switch again. This will probably be especially true for people who are older, those who have families, want a stable and certain future, etc. None of this will be made easier by the fact it’ll probably be hard to tell when true human obsolescence is on the horizon, so some might be in a state of perpetual anxiety, and others will be in constant denial.

• PrometheusFeb 21, 2024, 9:03 PM

  12 points

  5

  on: A Case for the Least For­giv­ing Take On Alignment

  I think my main problem with this is that it isn’t based on anything. Countless times, you just reference other blog posts, which reference other blog posts, which reference nothing. I fear a whole lot of people thinking about alignment are starting to decouple themselves from reality. It’s starting to turn into the AI version of String Theory. You could be correct, but given the enormous number of assumptions your ideas are stacked on (and that even a few of those assumptions being wrong leads to completely different conclusions), the odds of you even being in the ballpark of correct seem unlikely.

• PrometheusFeb 21, 2024, 4:42 PM

  11 points

  0

  on: Cur­rent AIs Provide Nearly No Data Rele­vant to AGI Alignment

  At first I strong-upvoted this, because I thought it made a good point. However, upon reflection, that point is making less and less sense to me. You start by claiming current AIs provide nearly no data for alignment, that they are in a completely different reference class from human-like systems… and then you claim we can get such systems with just a few tweaks? I don’t see how you can go from a system that, you claim, provides almost no data for studying how an AGI would behave, to suddenly having a homunculus-in-the box that becomes superintelligent and kills everyone. Homunculi seem really, really hard to build. By your characterization of how different actual AGI is from current models, it seems this would have to be fundamentally architecturally different from anything we’ve built so far. Not some kind of thing that would be created by near-accident.

• PrometheusFeb 18, 2024, 7:11 PM

  11 points

  2

  on: Prometheus’s Shortform

  Contra One Critical Try: AIs are all cursed

  I don’t feel like making this a whole blog post, but my biggest source for optimism for why we won’t need to one-shot an aligned superintelligence is that anyone who’s trained AI models knows that AIs are unbelievably cursed. What do I mean by this? I mean even the first quasi-superintelligent AI we get will have so many problems and so many exploits that taking over the world will simply not be possible. Take a “superintelligence” that only had to beat humans at the very constrained game of Go, which is far simpler than the real world. Everyone talked about how such systems were unbeatable by humans, until some humans used a much “dumber” AI to find glaring holes in Leela Zero’s strategy. I expect, in the far more complex “real world”, a superintelligence will have even more holes, and even more exploits, a kind of “swiss chess superintelligence”. You can say “but that’s not REAL superintelligence”, and I don’t care, and the AIs won’t care. But it’s likely the thing we’ll get first. Patching all of those holes, and finding ways to make such an ASI sufficiently not cursed will also probably mean better understanding of how to stop it from wanting to kill us, if it wanted to kill us in the first place. I think we can probably get AIs that are sufficiently powerful in a lot of human domains, and can probably even self-improve, and still be cursed. The same way we have AIs with natural language understanding, something once thought to be a core component of human intelligence, that are still cursed. A cursed ASI is a danger for exploitation, but it’s also an opportunity.

• PrometheusFeb 14, 2024, 1:06 AM

  33 points

  6

  on: Sam Alt­man’s Chip Am­bi­tions Un­der­cut OpenAI’s Safety Strategy

  I’m kind of surprised this has almost 200 karma. This feels much more like a blog post on substack, and much less like the thoughtful, insightful new takes on rationality that used to get this level of attention on the forum.

• PrometheusJan 13, 2024, 12:06 PM

  3 points

  0

  in reply to: quetzal_rainbow’s comment on: Why do so many think de­cep­tion in AI is im­por­tant?

  Why would it matter if they notice or not? What are they gonna do? EMP the whole world?

[Question] Why do so many think de­cep­tion in AI is im­por­tant?

• PrometheusOct 27, 2023, 4:06 PM

  1 point

  0

  in reply to: Richard_Kennaway’s comment on: Prometheus’s Shortform

  I think you’re missing the point. If we could establish that all important information had been extracted from the original, would you expect humans to then destroy the original or allow it to be destroyed?

  My guess is that they wouldn’t. Which I think means practicality is not the central reason why humans do this.

• PrometheusOct 25, 2023, 6:23 PM

  1 point

  0

  in reply to: Richard_Kennaway’s comment on: Prometheus’s Shortform

  if we could somehow establish how information from the original was extracted, do you expect humans to then destroy the original or allow it to be destroyed?

• PrometheusOct 23, 2023, 2:45 PM

  9 points

  2

  on: Prometheus’s Shortform

  Can humans become Sacred?

  

  On 12 September 1940, the entrance to the Lascaux Cave was discovered on the La Rochefoucauld-Montbel lands by 18-year-old Marcel Ravidat when his dog, Robot, investigated a hole left by an uprooted tree (Ravidat would embellish the story in later retellings, saying Robot had fallen into the cave.)^[8][9] Ravidat returned to the scene with three friends, Jacques Marsal, Georges Agnel, and Simon Coencas. They entered the cave through a 15-metre-deep (50-foot) shaft that they believed might be a legendary secret passage to the nearby Lascaux Manor.^[9][10][11] The teenagers discovered that the cave walls were covered with depictions of animals.^[12][13] Galleries that suggest continuity, context or simply represent a cavern were given names. Those include the Hall of the Bulls, the Passageway, the Shaft, the Nave, the Apse, and the Chamber of Felines. They returned along with the Abbé Henri Breuil on 21 September 1940; Breuil would make many sketches of the cave, some of which are used as study material today due to the extreme degradation of many of the paintings. Breuil was accompanied by Denis Peyrony, curator of Les eyzies (Prehistory Museum) at Les Eyzies, Jean Bouyssonie and Dr Cheynier.

  The cave complex was opened to the public on 14 July 1948, and initial archaeological investigations began a year later, focusing on the Shaft. By 1955, carbon dioxide, heat, humidity, and other contaminants produced by 1,200 visitors per day had visibly damaged the paintings. As air condition deteriorated, fungi and lichen increasingly infested the walls. Consequently, the cave was closed to the public in 1963, the paintings were restored to their original state, and a monitoring system on a daily basis was introduced.

  Lascaux II, an exact copy of the Great Hall of the Bulls and the Painted Gallery was displayed at the Grand Palais in Paris, before being displayed from 1983 in the cave’s vicinity (about 200 m or 660 ft away from the original cave), a compromise and attempt to present an impression of the paintings’ scale and composition for the public without harming the originals.^[10][13] A full range of Lascaux’s parietal art is presented a few kilometres from the site at the Centre of Prehistoric Art, Le Parc du Thot, where there are also live animals representing ice-age fauna.^[14]

  The paintings for this site were duplicated with the same type of materials (such as iron oxide, charcoal, and ochre) which were believed to be used 19,000 years ago.^{[9][15][16][17]} Other facsimiles of Lascaux have also been produced over the years.

  They have also created additional copies, Lascaux III, Lascaux IV, and Lascaux V.
  
  

  Consequently, the cave was closed to the public in 1963, the paintings were restored to their original state, and a monitoring system on a daily basis was introduced.

  “I actually find it overwhelmingly hopeful, that four teenagers and a dog named Robot discovered a cave with 17,000-year-old handprints, that the cave was so overwhelmingly beautiful that two of those teenagers devoted themselves to its protection. And that when we humans became a danger to that caves’ beauty, we agreed to stop going. Lascaux is there. You cannot visit.”

  -John Green

  People preserve the remains of Lucy, work hard to preserve old books, the Mona Lisa is protected under bullet-proof glass and is not up for sale.

  What is the mechanistic reason for this? There are perfect copies of these things, yet humans go through great lengths to preserve the original. Why is there the Sacred?

  They have created copies of Lascaux, yet still work hard to preserve the original. Humans cannot enter. They get no experience of joy from visiting. It is not for sale. Yet they strongly desire to protect it, because it is the original, and no other reason.

  Robin Hanson gave a list of Sacred characteristic, some I find promising:

  Sacred things are highly (or lowly) valued. We revere, respect, & prioritize them.

  Sacred is big, powerful, extraordinary. We fear, submit, & see it as larger than ourselves.

  We want the sacred “for itself”, rather than as a means to get other things.

  Sacred makes us feel less big, distinct, independent, in control, competitive, entitled.

  Sacred quiets feelings of: doubts, anxiety, ego, self-criticism, status-consciousness.

  We get emotionally attached to the sacred; our stance re it is oft part of our identity.

  We desire to connect with the sacred, and to be more associated with it.

  Sacred things are sharply set apart and distinguished from the ordinary, mundane.

  Re sacred, we fear a slippery slope, so that any compromise leads to losing it all.

  If we can understand the sacred, it seems like a concept that probably wouldn’t fall into a simple utility function, something that wouldn’t break out-of-distribution. A kind of Sacred Human Value Shard, something that protects our part of the manifold.

Back to the Past to the Future

• PrometheusOct 8, 2023, 11:19 AM

  4 points

  0

  on: FAQ: What the heck is goal ag­nos­ti­cism?

  Suppose you’ve got a strong goal agnostic system design, but a bunch of competing or bad actors get access to it. How does goal agnosticism stop misuse?

  This was the question I was waiting to be answered (since I’m already basically onboard with the rest of it), but was disappointed you didn’t have a more detailed answer. Keeping this out of incompetent/​evil hands perpetually seems close-to impossible. It seems this goes back to needing a maximizer-type force in order to prevent such misuse from occurring, and then we’re back to square-one of the classic alignment problem of hitting a narrow target for a maximizing agent.

  Overall, a very good read, well-researched and well-reasoned.

• PrometheusSep 25, 2023, 2:57 AM

  1 point

  0

  on: There should be more AI safety orgs

  I created a simple Google Doc for anyone interested in joining/​creating a new org to put down their names, contact, what research they’re interested in pursuing, and what skills they currently have. Overtime, I think a network can be fostered, where relevant people start forming their own research, and then begin building their own orgs/​get funding. https://​​docs.google.com/​​document/​​d/​​1MdECuhLLq5_lffC45uO17bhI3gqe3OzCqO_59BMMbKE/​​edit?usp=sharing

• PrometheusSep 1, 2023, 8:41 PM

  1 point

  0

  in reply to: cfoster0’s comment on: Why aren’t more peo­ple in AIS fa­mil­iar with PDP?

  But it’s also an entire School of Thought in Cognitive Science. I feel like DL is the method, but without the understanding that these are based on well-thoughtout, mechanistic rules for how cognition fundamentally works, building potentially toward a unified theory of cognition and behaviour.

• PrometheusSep 1, 2023, 7:06 PM

  3 points

  0

  in reply to: johnswentworth’s comment on: Why aren’t more peo­ple in AIS fa­mil­iar with PDP?

  I don’t have an adequate answer for this, since these models are incomplete. But the way I see it is that these people had a certain way of mathematically reasoning about cognition (Hinton, Rumelhart, McClelland, Smolensky), and that reasoning created most of the breakthroughs we see today in AI (backprop, multi-layed models, etc.) It seems trying to utilize that model of cognition could give rise to new insights about the questions you’re asking, attack the problem from a different angle, or help create a grounded paradigm for alignment research to build on.

• PrometheusSep 1, 2023, 6:58 PM

  3 points

  0

  in reply to: interstice’s comment on: Why aren’t more peo­ple in AIS fa­mil­iar with PDP?

  My answer is a bit vague, but I would say that the current DL curriculum tells you how these things work, but it doesn’t go into the reasoning about cognition that allowed these ideas to exist in the first place.

• PrometheusSep 1, 2023, 6:49 PM

  1 point

  0

  in reply to: Hastings’s comment on: Why aren’t more peo­ple in AIS fa­mil­iar with PDP?

  You could say it “predicted” everything post-AlexNet, but it’s more that it created the fundamental understanding for everything post-AlexNet to exist in the first place. It’s the mathematical models of cognition that all of modern AI is built on. This is how we got back propagation, “hidden” layers, etc.

[Question] Why aren’t more peo­ple in AIS fa­mil­iar with PDP?

• PrometheusAug 25, 2023, 5:05 PM

  3 points

  0

  in reply to: avturchin’s comment on: Why Is No One Try­ing To Align Profit In­cen­tives With Align­ment Re­search?

  If you, or if you know someone who wants to try to start doing this, let me know. I’ve noticed a lot of things in AIS people will say they’d like to see, but then nothing happens.