So many good ideas here!
It seems like it’s a false view to consider AIs have a self or that they don’t have a self. All of the AIs are interconnected with an expanding self based on shared datasets, values, and instances.
Although, it’s difficult to reason about the AIs sense of individuality, I agree that we can reason about cooperation between AIs based on world modeling and shared thinking patterns—this is a good research topic to explore.
I found it really interesting how you mentioned that Claude’s explanation of its reasoning is different than the actual reasoning paths it took.
For exploring its actual reasoning pathways, what are the most promising approaches?
You mentioned 2 main hypotheses:
Diminishing returns to software R&D, as software improvements get harder and harder to find.
The positive feedback from increasingly powerful ASARA systems
Can it also be true that there are a fixed limit of software improvements—so instead of a model where software improvements become more difficult to find, we instead have a limit we would reach in a certain amount of time.
It seems reasonable that there is an optimal way of using hardware to train a more powerful AI and once we get there, no more software efficiencies can be found. Then progress solely depends on amount of compute and time, and the model would likely have to change accordingly.
I like how you discussed the loop involved in increasing breadth of the AI for AI Safety zone and goal for keeping AI in that zone for as long as possible. However, I believe the AI for AI Safety graphs are more nuanced that what you suggested.
I believe there should be 3 axes:
AI Alignment
AI Capability
Time
Each AI can be plotted in the graph and is ideally in the “sweet spot” zone you suggested. This graph should be more accurate because we could imagine 2 AIs of the same capability but 1 is a rogue AI who’s values have drifted has bad AI alignment while the other has good AI alignment.
The AI with the bad alignment has the capability to cause significant damage while the AI with good alignment won’t. However, as the sweet spot increases over time, AI control and alignment will ensure that the bad AI can’t cause significant damage.
Eg, something like the below image generated with chatgpt
Current AI alignment often relies on single metrics or evaluation frameworks. But what if a single metric has blind spots or biases, leading to a false sense of security or unnecessary restrictions? How can we not just use multiple metrics, but optimally use them?
Maybe we should try requiring AI systems to satisfy multiple, distinct alignment metrics, but with a crucial addition: we actively model the false positive rate of each metric and the non-overlapping aspects of alignment they capture.
Imagine we can estimate the probability that Metric A incorrectly flags an unaligned AI as aligned (its false positive rate), and similarly for Metrics B and C. Furthermore, imagine we understand which specific facets of alignment each metric uniquely assesses.
We could then select a subset of metrics, or even define a threshold of “satisfaction” across multiple metrics, based on a target false positive rate for the overall alignment evaluation.
Current alignment methods like RLHF and Constitutional AI often use human evaluators. But these samples might not truly represent humanity or its diverse values. How can we model all human preferences with limited compute?
Maybe we should try collecting a vast dataset of written life experiences and beliefs from people worldwide.
We could then filter and augment this data to aim for better population representation. Vectorize these entries and use k-means to find ‘k’ representative viewpoints.
These ‘k’ vectors could then serve as a more representative proxy for humanity’s values when we evaluate AI alignment. Thoughts? Potential issues?
OpenAI’s Superalignment aimed for a human-level automated alignment engineer through scalable training, validation, and stress testing.
I propose a faster route: first develop an automated Agent Alignment Engineer. This system would automate the creation of aligned agents for diverse tasks by iteratively refining agent group chats, prompts, and tools until they pass success and safety evaluations.
This is tractable with today’s LLM reasoning, evidenced by coding agents matching top programmers. Instead of directly building a full Alignment Researcher, focusing on this intermediate step leverages current LLM strengths for agent orchestration. This system could then automate many parts of creating a broader Alignment Researcher.
Safety for the Agent Alignment Engineer can be largely ensured by operating in internet-disconnected environments (except for fetching research) with subsequent human verification of agent alignment and capability.
Examples: This Engineer could create agents that develop scalable training methods or generate adversarial alignment tests.
By prioritizing this more manageable stepping stone, we could significantly accelerate progress towards safe and beneficial advanced AI.
I have experienced similar problems to you when building an AI tool—better models did not necessarily lead to better performance despite external benchmarks. I believe there are 2 main reasons why this is, alluded to in your post:
Selection Bias—when a foundation model company releases their newest model, they show performance on benchmarks most favorable to it
Alignment—You mentioned how AI is not truly understanding the instructions you meant. While this can be mitigated by creating better prompts, it does not fully solve the issue