You might also want a mechanism to handle “staples” that individuals want. I have a few foods / ingredients I like to keep on hand at all times, and be able to rely on having. I’d have no objections to other people eating them, but if they did I’d want them to take responsibility for never leaving the house in a state of “no X on hand”.
evand
Karma: 1,370
Those numbers sound like reasonable estimates and goals. Having taught classes at TechShop, that first handful of hours is important. 20 hours of welding instruction ought to be enough that you know whether you like it and can build some useful things, but probably not enough to get even an intro-level job. It should give you a clue as to whether signing up for a community college class is a good idea or not.
Also I’m really confused by your inclusion of EE in that list; I’d have put it on the other one.
However, I’m skeptical of systems that require 99.99% reliability to work. Heuristically, I expect complex systems to be stable only if they are highly fault-tolerant and degrade gracefully.
On the other hand… look at what happens when you simply demand that level of reliability, put in the effort, and get it. From my engineering perspective, that difference looks huge. And it doesn’t stop at 99.99%; the next couple nines are useful too! The level of complexity and usefulness you can build from those components is breathtaking. It’s what makes the 21st century work.
I’d be really curious to see what happens when that same level of uncompromising reliability is demanded of social systems. Maybe it doesn’t work, maybe the analogy fails. But I want to see the answer!
What happens when the committed scorched-earth-defender meets the committed extortionist? Surely a strong precommitment to extortion by a powerful attacker can defeat a weak commitment to scorched earth by a defender?
It seems to me this bears a resemblence to Chicken or something, and that on a large scale we might reasonably expect to see both sets of outcomes.
What’s that? If I don’t give into your threat, you’ll shoot me in the foot? Well, two can play at that game. If you shoot me in the foot, just watch, I’ll shoot my other foot in revenge.
On the other hand… what level do you want to examine this at?
We actually have pretty good control of our web browsers. We load random untrusted programs, and they mostly behave ok.
It’s far from perfect, but it’s a lot better than the desktop OS case. Asking why one case seems to be so much farther along than the other might be instructive.
Again, I’m going to import the “normal computer control” problem assumptions by analogy:
The normal control problem allows minor misbehaviour, but that it should not persist over time
Take a modern milling machine. Modern CNC mills can include a lot of QC. They can probe part locations, so that the setup can be imperfect. They can measure part features, in case a raw casting isn’t perfectly consistent. They can measure the part after rough machining, so that the finish pass can account for imperfections from things like temperature variation. They can measure the finished part, and reject or warn if there are errors. They can measure their cutting tools, and respond correctly to variation in tool installation. They can measure their cutting tools to compensate for wear, detect broken tools, switch to the spare cutting bit, and stop work and wait for new tools when needed.
Again, I say: we’ve solved the problem, for things literally as simple as pounding a nail, and a good deal more complicated. Including variation in the nails, the wood, and the hammer. Obviously the solution doesn’t look like a fixed set of voltages sent to servo motors. It does look like a fixed set of parts that get made.
How involved in the field of factory automation are you? I suspect the problem here may simply be that the field is more advanced than you give it credit for.
Yes, the solutions are expensive. We don’t always use these solutions, and often it’s because using the solution would cost more and take more time than not using it, especially for small quantity production. But the trend is toward more of this sort of stuff being implemented in more areas.
The “normal computer control problem” permits some defects, and a greater than 0% error rate, provided things don’t completely fall apart. I think a good definition of the “hammer control problem” is similar.
It bends the nails, leaves dents in the surface and given the slightest chance will even attack your fingers!
We’ve mostly solved that problem.
I’m not sure that being able to nearly perfectly replicate a fixed set of physical actions is the same thing as solving a control problem.
It’s precisely what’s required to solve the problem of a hammer that bends nails and leaves dents, isn’t it?
Stuxnet-type attacks
I think that’s outside the scope of the “hammer control problem” for the same reasons that “an unfriendly AI convinced my co-worker to sabotage my computer” is outside the scope of the “normal computer control problem” or “powerful space aliens messed with my FAI safety code” is outside the scope of the “AI control problem”.
It is worth noting that the type of control that you mention (e.g. “computer-controlled robots”) is all about getting as far from “agenty” as possible.
I don’t think it is, or at least not exactly. Many of the hammer failures you mentioned aren’t “agenty” problems, they’re control problems in the most classical engineering sense: the feedback loop my brain implements between hammer state and muscle output is incorrect. The problem exists with humans, but also with shoddily-built nail guns. Solving it isn’t about removing “agency” from the bad nail gun.
Sure, if agency gets involved in your hammer control problem you might have other problems too. But if the “hammer control problem” is to be a useful problem, you need to define it as not including all of the “normal computer control problem” or “AI control problem”! It’s exactly the same situation as the original post:
The normal control problem assumes that no specific agency in the programs (especially not super-intelligent agency)
They usually don’t have any way to leverage their models to increase the cost of not buying their product or service though; so such a situation is still missing at least one criterion.
Modern social networks and messaging networks would seem to be a strong counterexample. Any software with both network effects and intentional lock-in mechanisms, really.
And honestly, calling such products a blend of extortion and trade seems intuitively about right.
To try to get at the extortion / trade distinction a bit better:
Schelling gives us definitions of promises and threats, and also observes there are things that are a blend of the two. The blend is actually fairly common! I expect there’s something analogous with extortion and trade: you can probably come up with pure examples of both, but in practice a lot of examples will be a blend. And a lot of the ‘things we want to allow’ will look like ‘mostly trade with a dash of extortion’ or ‘mostly trade but both sides also seem to be doing some extortion’.
We’ve (mostly) solved the hammer control problem in a restricted domain. It looks like computer-controlled robots. With effort, we can produce an entire car or similar machine without mistakes.
Obviously we haven’t solved the control problem for those computers: we don’t know how to produce that car without mistakes on the first try, or with major changes. We have to be exceedingly detailed in expressing our desires. Etc.
This may seem like we’ve just transformed it into the normal computer control problem, but I’m not entirely sure. Air-gapped CNC machinery running embedded OSes (or none at all) is pretty well behaved. It seems to me more like “we don’t know how to write programs without testing them” than the “normal computer control problem”.
You May Not Believe In Guess[Infer] Culture But It Believes In You
I think this comment is the citation you’re looking for.
On the legality of selecting your buyers: What if you simply had a HOA (or equivelent) with high dues, that did rationalist-y things with the dues? Is that legal, and do you think it would provide a relevant selection effect?
We might also want to compute the sum of the GDP of A and B: does that person moving cause more net productivity growth in B than loss in A?
Possibly a third adversarial AI? Have an AI that generates the questions based on P, is rewarded if the second AI evaluates their probability as close to 50%, is rewarded for the first AI being able to get them right based on P’, and for the human getting them wrong.
That’s probably not quite right; we want the AI to generate hard but not impossible questions. Possibly some sort of term about the AIs predicting whether the human will get a question right?
That seems amazingly far from a worst case scenario.
Have you read Politics is the Mind-Killer? I get the vague sense you haven’t, and I see lots of references here to it but no direct link. If you haven’t, you should go read it and every adjacent article.
Edit: actually there is a link below already. Oops.
I strongly favor this project and would love to read more on the subject. I’m hopeful that its online presence happens here where I’m likely to read it, and that it doesn’t vanish onto Tumblr or Facebook or something similarly inaccessible.
I notice that I (personally) feel an ugh response to link posts and don’t like being taken away from LW when I’m browsing LW. I’m not sure why.
I do too. I don’t know all the reasons, but one is simply web page design. The external page is often slow to load and unpleasant to read in comparison. This often comes with no benefit relative to just having the text in the post on LW.
Additionally, I assume that authors on other sites are a lot less likely to engage in discussion on LW, whether in comments or further posts. That seems like a big minus to me.
Seeing as this is an entire article about nitpicking and mathematical constructs...
perfect rationality means to me more rational than any other agent. I think that is a reasonable definition.
Surely that should be “at least as rational as any other agent”?
Well, in general, I’d say achieving that reliability through redundant means is totally reasonable, whether in engineering or people-based systems.
At a component level? Lots of structural components, for example. Airplane wings stay attached at fairly high reliability, and my impression is that while there is plenty of margin in the strength of the attachment, it’s not like the underlying bolts are being replaced because they failed with any regularity.
I remember an aerospace discussion about a component (a pressure switch, I think?). NASA wanted documentation for 6 9s of reliability, and expected some sort of very careful fault tree analysis and testing plan. The contractor instead used an automotive component (brake system, I think?), and produced documentation of field reliability at a level high enough to meet the requirements. Definitely an example where working to get the underlying component that reliable was probably better than building complex redundancy on top of an unreliable component.