The Fønix team is just heating water, which is great but actual distillation (with automated re-adding of specific minerals) is probably what you’re actually going to want so as to avoid all contamination not just biological.
In this size of structure growing food isn’t really worth it, storing food for 10 years is actually easier (according to claude). It does need to come stocked though
It’s more that it stops being relevant to humans, as keeping humans in the loop slows down the exponential growth
I do think VR and neuralink-like tech will be a very big deal though, especially in regards to allowing people experiences that would otherwise be expensive in atoms
At what IQ do you think humans are able to “move up to higher levels of abstraction”?
(Of course this assumes AIs don’t get the capability to do this themselves)
Re robotics advancing while AI intelligence stalls, robotics advancing should be enough to replace any people who can’t take advantage of automation of their current jobs.
I don’t think you’re correct in general, but it seems that automation will clear out at least the less skilled jobs in short order (decades at most)
I very much hope the computers brought in were vetted and kept airgapped.
You keep systems separate, yes.
For some reason I assumed that write permissions were on user in the actual system/secure network and any data exporting would be into secured systems. If they created a massive security leak for other nations to exploit, that’s a crux for me on whether this was reckless.
Added: what kind of idiot purposely puts data in the wrong system purposely? The DOGE guys doing this could somehow make sense, governmental workers??
No.
I’m not familiar with public documentation on this.
I know people who have gotten access to similarly important governmental systems at younger ages.
Don’t worry about it too much.
If they abuse it, it’ll cost their group lots of political goodwill. (Recursive remove for example)
Musk at least is looking to upgrade humans with Neuralink
If he can add working memory can be a multiplier for human capabilities, likely to scale with increased IQ.
Any reason the 4M$ isn’t getting funded?
Any good, fairly up-to-date lists of the relevant papers to read to catch up with AI research (as far as a crash course will take a newcomer)?
Preferably one that will be updated
Reading novels with ancient powerful beings is probably the best direction you have for imagining how status games amongst creatures which are only loosely human look.
Resources being bounded, there will tend to always be larger numbers of smaller objects (given that those objects are stable).
There will be tiers of creatures. (In a society where this is all relevant)
While a romantic relationship skipping multiple tiers wouldn’t make sense, a single tier might.
The rest of this is my imagination :)
Base humans will be F tier, the lowest category while being fully sentient. (I suppose dolphins and similar would get a special G tier).
Basic AGIs (capable of everything a standard human is, plus all the spikey capabilities) and enhanced humans E tier.
Most creatures will be here.
D tier:
Basic ASIs and super enhanced humans (gene modding for 180+ IQ plus SOTA cyborg implants) will be the next tier, there will be a bunch of these in absolute terms but relative to the earlier tier rarer.
C tier:
Then come Alien Intelligence, massive compute resources supporting ASIs trained on immense amounts of ground reality data, biological creatures that have been redesigned fundamentally to function at higher levels and optimally synergize with neural connections (whether with other carbon based or silicon based lifeforms)
B tier:
Planet sized clusters running ASIs will be a higher tier.
A, S tiers:
Then you might get entire stars, then galaxies.
There will be much less at each level.
Most tiers will have a -, neutral or +.
- : prototype, first or early version. Qualitatively smarter than the tier below, but non-optimized use of resources, often not the largest gap from the + of the earlier tier
Neutral: most low hanging optimizations and improvements and some harder ones at this tier are implemented
+: highly optimized by iteratively improved intelligences or groups of intelligences at this level, perhaps even by a tier above.
Writing tests, QA and Observability are probably going to stay for a while and work hand in hand with AI programming, as other forms of programming start to disappear. At least until AI programming becomes very reliable.
This should allow for working code to be produced way faster, likely giving more high-quality ‘synthetic’ data, but more importantly massively changing the economics of knowledge work
Is there a reason that random synthetic cells will not be mirror cells?
We have here some scientists making cells. Looks like a dangerous direction
Humans seem way more energy and resource efficient in general, paying for top talent is an exception not the rule- usually it’s not worth paying for top talent.
Likely to see many areas where better economically to save on compute/energy by having human do some of the work.
Split information workers vs physical too, I expect them to have very different distributions of what the most useful configuration is.
This post ignores likely scientific advances in bioengineering and cyborg surgeries, I expect humans to be way more efficient for tons of jobs once the standard is 180 IQ with a massive working memory
I do things like this at times with my teams.
Important things:
Don’t think you need to solve the actual problem for them
Do solve ‘friction’ for them as much as possible
Do feel free to look up other sources so you can offer more perspective and to take off the load of having to find relevant info
positive energy, attentive etc
if they’re functioning well just watch and listen while being interested and unobtrusive, at most very minor inputs if you’re pretty sure it’ll be helpful
If stuck at a crossroads ask them how long they think each path will take/ how hard it’ll be, and give them feedback if you think they’re wrong. Help them start working on one, people can get stuck for longer than it would take to actually do one option.
if lost, methodically go through the different areas where the issue could be and methodically go through all the directions they could take for each area and in general. You don’t need to think these up, but keep track of them and help guide towards picking apart the problem and solution spaces. This takes some mental load off.
A message from Claude:
‴This has been a fascinating and clarifying discussion. A few key insights I’ll take away:
The distinction between bounded and unbounded optimization is more fundamental than specific value differences between AIs. The real existential threat comes from unbounded optimizers. The immune system/cancer metaphor provides a useful framework—it’s about maintaining a stable system that can identify and prevent destructive unbounded growth, not about enforcing a single value set. The timing challenge is critical but more specific than I initially thought—we don’t necessarily need the “first” AGI to be perfect, but we need bounded optimizers to establish themselves before any unbounded ones emerge.
Some questions this raises for further exploration:
What makes a Schelling fence truly stable under recursive self-improvement? Could bounded optimizers coordinate even with different base values, united by shared meta-level constraints? Are there ways to detect early if an AI system will maintain bounds during capability gain?
The framing of “cancer prevention” versus “value enforcement” feels like an important shift in how we think about AI governance and safety. Instead of trying to perfectly specify values, perhaps we should focus more on creating robust self-limiting mechanisms that can persist through capability gains.‴
A few thoughts.
Have you checked what happens when you throw physic postdocs at the core issues—do they actually get traction or just stare at the sheer cliff for longer while thinking? Did anything come out of the Illiad meeting half a year later? Is there a reason that more standard STEMs aren’t given an intro into some of the routes currently thought possibly workable, so they can feel some traction? I think either could be true- that intelligence and skills aren’t actually useful right now, the problem is not tractable, or better onboarding could let the current talent pool get traction—and either way it might not be very cost effective to get physics postdocs involved.
Humans are generally better at doing things when they have more tools available. While the ‘hard bits’ might be intractable now, they could well be easier to deal with in a few years after other technical and conceptual advances in AI, and even other fields. (Something something about prompt engineering and Anthropic’s mechanistic interpretability from inside the field and practical quantum computing outside).
This would mean squeezing every drop of usefulness out of AI at each level of capability, to improve general understanding and to leverage it into breakthroughs in other fields before capabilities increase further. In fact, it might be best to sabotage semiconductor/chip production once the models one gen before super-intelligence/extinction/ whatever, giving maximum time to leverage maximum capabilities and tackle alignment before the AIs get too smart.
How close is mechanistic interpretability to the hard problems, and what makes it not good enough?
The point was more about creating your own data being easy, just generate code then check it by running it. Save this code, and later use it for training.
If we wanted to go the way of AlphaZero it doesn’t seem crazy.
De-enforce commands, functions, programs which output errors, for a start.
I didn’t think of the pm as being trained by these games, that’s interesting. Maybe have two instances competing to get closer on some test cases the pm can prepare to go with the task, and have them competing on time, compute, memory, and accuracy. You can de-enforce the less accurate, and if fully accurate they can compete on time, memory, cpu.
I’m not sure “hard but possible” is the bar—you want lots of examples of what doesn’t work along with what does, and you want it for easy problems and hard ones so the model learns everything
Product manager, non-technical counterpart to a team lead in a development team
I notice that I’m confused.
Google made an amazing AI for playing chess, by allowing it to make its own data.
Why hasn’t the same thing happened for programming? Have it generate a bunch of pictures with functionality expectations (a PM basically), have it write and run code, then check the output against the requirements it created, then try again when it doesn’t come out right.
This is even easier where the pm is unnecessary—leetcode, codewars, euler...
You could also pay PMs to work with the AI developers, instead of the code tutors xAI is hiring.
There seems to be a preference to having the LLMs memorize code instead of figuring things out itself.
If you run out of things like that you could have it run random programs in different languages, only learning from those that work.
I haven’t used genesis, but that also seems like a mostly-built validator for programs that AIs can use to create and train on their own data.
With the amount of compute going into training, it should be easy to create huge amounts of data?
Keeping humans around is the correct move for a powerful AGI, assuming it isn’t being existentially threatened.
For a long while human inputs will be fairly different from silicon inputs, and humans can do work—intellectual or physical—and no real infrastructure is necessary for human upkeep or reproduction (compared to datacenters).
Creating new breeds of human with much higher IQs and creating (or having them create) neuralink-like tech to cheaply increase human capabilities will likely be a very good idea for AGIs.
Most people here seem worried about D tier ASIs, ASIs should see the benefits of E tier humans (250+ IQ and/or RAM added through neuralink-like tech) and even D tier humans (genesmith on editing, 1500+ IQs with cybernetics vastly improving cognition and capability)
‘Sparing a little sunlight’ for an alternative lifeform which creates a solid amount of redundancy as well as being more effecient for certain tasks and allowing for more diverse research, as well as having minimal up-front costs is overdetermined