it would make the problem more tractable
The problem of creating a strong AI and surviving, that is. We’d still get Hanson’s billions of self-directed EMs.
Kinrany
Karma: 67
Thanks!
It has been explored (multiple times even on this site), and doesn’t avoid doom. It does close off some specific paths that might otherwise lead to doom, but not all or even most of them.
Do you have any specific posts in mind?
To be clear, I’m not suggesting that because of this possibility we can just hope that this is how it plays out and we will get lucky.
If we could find a hard limit like this, it seems like it would make the problem more tractable, however. It doesn’t have to exist simply because we want it to exist. Searching for it still seems like a good idea.
There’s a hundred problems to solve, but it seems like it could avoid the main bad scenario at least: that of AI rapidly self-improving. Improving its hardware wouldn’t be trivial for a human-level AI, and it wouldn’t have options present in other scenarios. And scaling beyond a single machine seems likely to be a significant barrier at least.
It could still create millions of copies of itself. That’s still a problem, but also still a better problem to have than a single AI with no coordination overhead.
[Question] If alignment problem was unsolvable, would that avoid doom?
This should be mitigated by pools of mutual trust that naturally form whenever there’s a loop in the trust graph.
When the number of layers grows, the only thing that really works is metrics that cannot be goodhearted. Whenever those metrics exist, money becomes a perfectly good expression of success.
It might work to completely prohibit more than one layer of middle management. Instead, when middle manager Bob wants more people, he and his boss Alice come up with a contract that can’t be gamed too much. Alice spins out Bob’s org subtree into a new organization, and then it becomes Bob’s job to buy the service from the new org as necessary. Alice also publishes the contract, so that entrepreneurs can swoop in and offer a better/cheaper service.
The ability to put up with bullshit is valuable: bullshit cannot be ignored once it is reified into real world objects, documents, habits.
Markdown has syntax for quotes: a line with
> thison it will look likethis
Honestly “fiction” was enough of a spoiler. “As a child, we were always told that every sapient life is precious.” made it a certainty.
Suggestion: “sangaku proving the Pythagoras’ theorem”. I wonder if it can do visual explanations.
Since Semyonova did not care to look at things from the peasants’ point of view and mixed her research with attempts to convert, I wonder how many of the things she recorded were directly intended to shock her.
Hmm, I guess conflict resolution would be garbage, but simultaneous editing is rarely a good experience anyway. Otherwise storing and sharing text files using a file sync service is fairly good compared to other options. Thanks!
Thanks!
I wasn’t aware of Etherpad. Other Google Docs equivalents seemed impossible to self-host and extend, so a non-starter.
I agree with your overview:
Etherpad provides collaborative editing, but integrating it with other services will probably take extra work
Logseq has better structure, but worse automation
Emacs can do most things on one computer, but rapid sharing is even harder
Pieces for a general purpose personal computing system. Ideally:
Edit data by hand
Store as plain text
Self-host, access from any device
Write formulas to derive data automatically
Mix and match structured data (markdown, tables, nested lists, whiteboard)
Search and navigate, like any wiki
Automate through a web API and webhooks
Collaborate in real time
The strict divide between high slack and low slack reminds me of synchronous and asynchronous companies: hybrids seem to work poorly.
Seven sketches in compositionality explores compositionality (category theory, really) with examples:
Dish recipes
Chemistry, resource markets and manufacturing
Relational database schemas and data migrations
Projects and teams with conflicting design trade-offs
Cyber-physical systems, signal flow graphs, circuits
I mean, it’s easier to find two people willing to play than ten. So you’ll get more data. With one or two teams it will be hard to draw any conclusions at all.
It seems picking a 1v1 game would work better as an experiment.
Caveat: ask each person to name someone they personally worked with.
Hard to get right, but not sure whether it’s harder than knowledge investment.
Wouldn’t have helped Louis XV. We might need infrastructure in place that would incentivize people to make themselves easy to find.
Is it really true that money can’t buy knowledge?
We can ask the most knowledgeable person we know to name the most knowledgeable person they know, and do that until we find the best expert. Or alternatively, ask a bunch of people to name a few, and keep walking this graph for a while.
This won’t let us buy knowledge that doesn’t exist, but seems good enough for learning from experts, given enough money and modern communication technology that Louis XV didn’t have.
An agent created in a computer would be an exception to that?