Lessons On How To Get Things Right On The First Try
This post is based on several true stories, from a workshop which John has run a few times over the past year.
John: Welcome to the Ball → Cup workshop! Your task for today is simple: I’m going to roll this metal ball:
… down this hotwheels ramp:
… and off the edge. Your job is to tell me how far from the bottom of the ramp to place a cup on the floor, such that the ball lands in the cup.
Oh, and you only get one try.
General notes:
I won’t try to be tricky with this exercise.
You are welcome to make whatever measurements you want of the ball, ramp, etc.
You can even do partial runs, e.g. roll the ball down the ramp and stop it at the bottom, or throw the ball through the air.
But you only get one full end-to-end run (from top of the ramp to the cup/floor), and anything too close to an end-to-end run (let’s say more than ~half the run) is discouraged. After all, in the AI situation for which the exercise is a metaphor, we don’t know exactly when something might foom; we want elbow room.
That’s it! Good luck, and let me know when you’re ready to give it a shot.
[At this point readers may wish to stop and consider the problem themselves.]
Alison: Let’s get that ball in that cup. It looks like this is probably supposed to be a basic physics kind of problem…but there’s got to be some kind of twist or else why would he be having us do it? Maybe the ball is surprisingly light….or maybe the camera angle is misleading and we are supposed to think of something wacky like that??
The Unnoticed Observer: Muahahaha.
Alison: That seems…hard. I’ll just start with the basic physics thing and if I run out of time before I can consider the wacky stuff, so be it.
So I should probably split this problem into two parts. The part where the ball arcs through the air once off the table is pretty easy…
The Unnoticed: True in this case, but how would you notice if it were false? What evidence have you seen?
Alison: …but the trouble is getting the exact velocity. What information do I have? Well, I can ask whatever I want, so I should be able to get all the parameters I need for the standard equations. Let’s make a shopping list: I want the starting height of the ball on the ramp (from the table), the mass of the ball, the height of the ramp off the table from multiple points along it (to estimate the curvature,) uhhh… oh shit maybe the bendiness matters! That seems really tricky. I’ll look at that first. Hey, John, can you poke the ramp a bit to demonstrate how much it flexes?
*John pokes at the ramp and the ramp bends.*
Well it did flex, but… it can’t have that much of an effect.
The Unnoticed: False in this case. Such is the danger of guessing without checking.
Alison: Calculating the effect of the ramp’s bendiness seems unreasonably difficult and this workshop is only meant to take an hour or so, so let’s forget that.
The Unnoticed: I am reminded of a parable about a quarter and a streetlight.
Alison: On to curve estimation!
The Unnoticed: Why on earth is she estimating the ramp’s curve anyway?
Alison: …Well I don’t actually know how to do much better than the linear approximation I got from the direct measurements. I guess I can treat part of the ramp as linear and then the end part as part of a circle. That will probably be good enough. Ooh if I take a frame from the video, I can just directly measure what the radius circle with arc of best fit is! Okay now that I’ve got that… Well I guess it’s time to look up how to do these physics problems, guess I’m rustier than I thought. I’ll go do that now.
Arrrgh okay I didn’t need to do any of that curve stuff after all, I just needed to do some potential/kinetic energy calculations (ignoring friction and air resistance etc) and that’s it! I should have figured it wouldn’t be that hard, this is just a workshop after all.
*Blackboard Montage*
implies = 3.4 m/s
implies
Result: = 0.8 m
Aaaaand that should do it. Given the height of the ball from the table, the mass, and the gravitational constant, I declare that the velocity of the ball at the end of the ramp is 3.4 m/s directly horizontal to the ground. Alright now the easy part. Just have to look up some Newtonian mechanics real quick…aaand yeah with that velocity, mass, height, and gravity blah blah blah, the ball should hit the ground exactly… 0.8 m from the end of the ramp. Nice job, self!
Okay, before running it I should probably check my work. The ballistics are definitely right, don’t need the curve stuff, potential energy gets converted to kinetic… oh shit. I vaguely remember something about rotational energy. Better look that up….
The Unnoticed: Could it be… the heavily foreshadowed “some kind of twist”?
Alison: Ahhhh shit okay so I need to factor that in too. Back to the blackboard.
*More Blackboard Montage*
Yup alright great so with that adjustment in place let’s look over it one more time. Ballistics, check. Energy calculations, balanced. Final velocity based on that… yup no arithmetic errors. Alright, if this problem is at all reasonable then this is the answer! Phew, that took some careful thought! Let’s see how it goes…
*Alison declares her official prediction. Trumpets blare; a drum roll sounds. The ball is dropped from the top of the ramp before the mayor, two judges, a jury of thirteen of Alison ’s peers, two armed guards, three visiting foreign dignitaries, a public notary, a priest, a rabbi, an imam, and one very confused bartender. The ball rolls down the ramp…
… and entirely fails to land in the cup. Or even hit it.*
Alison: …Well. I guess the lesson is to not trust guys in black fedoras.
The Suddenly Noticed Observer: Surely there is some takeaway other than just that?
Alison: Well, I did everything I could think of with what I’d been given and it still didn’t work. I even accounted for rotational energy! So look, maybe I made an error somewhere but at this point I’m pretty suspicious that there is any solution at all given what I have to work with.
The Noticed Observer: Sounds like we should place a bet, I might stand to make some money. Let’s see another’s effort… *snaps fingers with entirely unnecessary, but very satisfying, drama*
Robert (M.D.): Boy do I love my job. Surgery is just so rewarding! Cutting people up, putting them back together, and getting paid to do it! And also the lives saved, of course.
The Observer Who Is Back To Being Unnoticed: Now that’s just lazy writing, folks.
Robert (M.D.): Alright so what’s this about a ball in a cup? Doesn’t seem too bad. I mean, I don’t really remember physics that well, but it can’t be that hard. It’s just a workshop after all. Well, I never did like physics so let’s just look at the problem as a whole before we dissect it into tiny little—erm, do the physics.
Alison, Who Is Also Observing And Unnoticed: Wait, is this, like, a metaphorical surgeon?
Robert: So as far as I can tell, we’d like to slice the problem into at least two pieces, the part where the ball goes down the ramp and the part where the ball flies off the table and into the cup. I think that the second one is pretty standard. It’s just like tossing a ball through the air—that’s like example no.1 in freshman physics. Let’s not be too hasty, though… is there anywhere we can cut to make the problem even simpler? Hmm, maybe there is something more to see about the part just after the ball leaves the table, when it’s traveling through the air, and just before it hits the ground? Maybe the angle of travel through the air affects things? Well it’s certainly speeding up as it falls…Ooh maybe the air resistance changes as a result? Huh. Well I can test that one pretty easily by having John drop the ball on camera. I could also have him toss the ball alongside some other object to see if the air resistance or whatever else causes any significant differences.
*Robert has John drop the ball on camera, records it, then marches through frame-by-frame comparing the ball’s position to the predictions of a ballistic calculation*
The Unnoticed: Yessssss… check your model against reality… so many bits of information… might even notice if Something Weird were going on…
*Alison gives The Unnoticed a weird look, and scootches away a little in the Unseen Space.*
Robert: Well. I got the test results, I’ve googled physics 101, and, unfortunately, no further slicing to be done here as it seems like the part where the ball flies through the air might be as simple as it gets. Just basic Newtonian mechanics.
The Unnoticed: Yeah, turns out there’s nothing weird going on in this part. And now he knows that!
Robert: On to the ramp part! Ooh, this one seems complicated. Lots of places to slice ’n dice here. John did say we could run partial tests…but…what to test? I could test how long it takes the ball to get to various places on the ramp. Probably the end of the ramp is of particular interest. Maybe. I should probably get the height of the ramp…maybe its length?
….Argh I don’t really have a plan here. I need to cut the problem down more.
Where to slice…where to slice… Okay let’s try that thing I tried earlier: There’s a beginning part, a middle part, and an end part. The ball travels down a pretty straight path to begin with…. Then it passes through the curved part, and then flies off the end. The internet said that ramp problems really just boil down to potential/kinetic/rotational energy calculations so I guess that’s all I need to do?
Something…feels off about this. I feel like that’s a little too… epistemically-deferential? Or something? Why does this feel wrong… I guess I just don’t believe that the physics problems online are definitely talking about the thing I’m looking at. Maybe I should look at it some more. I’ll take a video of one full ramp-run, stopped at the bottom. Maybe that will help.
*Robert takes a video of the ball rolling down the ramp.*
*Unbeknownst to Robert, in the Unseen Space where Alison and The Unnoticed watch, a choir of angels suddenly appears and begins singing. The Unnoticed shoos them away.*
The Unnoticed: Guess the prediction markets on Robert have shot up. Damn, I should have placed my bet after he tested the air resistance thing, he clearly had the right habits in place to test the ramp too.
Robert: … Okay I’ve watched this thing about 10 times now and first thing’s first: no matter how you slice it, the goddamn ramp is goddamn BENDY. That is definitely not a “standard physics” problem as far as I’ve found online. All the potential/kinetic/rotational energy approaches assume a rigid ramp! That might not be a problem…but I honestly just don’t know either way.
*In the Unseen Space, the choir of angels try to sneak back in. The Unnoticed lands a solid kick on one of them, and they scurry off.*
Robert: The second thing I’ve realized: I am an idiot. I’ve been trying to do all this physics nonsense when I can just directly measure the speed of the ball. I started trying to calculate the speed of the ball going down the ramp at various points to test whether the bending made it deviate from the rigid-ramp model, and I did it just by taking the pixel-length difference in location and combining that with the camera’s frame rate. But why bother with any of those calculations when I can just get the speed of the ball at the end of the ramp just like that? Pains me as it does to stay my delicate scalpel of inquisition on the deep nature of the ramp, the final speed is all I need.
Is anything else missing, then? I get the speed from pixel calculations, then do the standard ballistics thing and that’s it?
…. Ah! Okay well just to be sure, I should probably get John to take a bunch of videos so I can measure how consistent the pixel calculation thing is.
The Unnoticed: Smart move. Unnecessary this time, but smart in general.
Robert: …And it’s fairly consistent! I guess it’s time?
*Robert declares his official prediction. Guitars ring out - *
Robert: WAIT! I don’t really know why but after looking at the video a bunch and thinking through it, I just have this feeling that the ball is going to overshoot. Please move the cup just half an inch further out.
The Unnoticed: The traditional last-minute gut-level adjustment. Not always right, but a good idea more often than not.
*Robert redeclares his official prediction. Guitars ring out, fireworks explode. The ball is dropped from the top of the ramp before the First Lady of the previous presidential administration, three Nobel-winning chemists, Dolly Parton, one guy who thinks he’s a prophet, the best man from Robert’s wedding, and one bored and frankly surly teenager who will definitely not admit to being related to Robert in any way. The ball rolls down the ramp…
… and goes into the cup…
… and hits the back of the cup and knocks it over. But that counts as success!*
Robert: Heyo! Robert M.D. de-livers, once again!
The Unnoticed: … *snaps fingers again*
The Back To Being Noticed: Ok, Alison, what did we learn from this?
Alison: That medical puns are the worst?
The Noticed: Yes. And?
Alison: Ugh well, I feel a little silly now because I was probably too confident before. I guess, the main thing is that Robert figured out that the key was the speed of the ball coming off the ramp and that he could get a good read on that value based on pixel measurements of the video… which he discovered by just kind of… looking at the system?
The Noticed: Bingo!
Alison: But…but surely there’s more to it than that?? I mean… I don’t love saying this, but maybe Robert is just smarter than me? I don’t know if I would have noticed the speed-calculation-from-pixels trick or the load-bearing nature of the final speed even if I had taken a video of it!
The Noticed: Your math already implied that the only thing you needed to know from the ramp part of the calculation was the ball’s velocity as it left the ramp.
Alison: …I guess that’s true. Argh. If I’m being fully honest with myself, maybe I just didn’t try as hard. When Robert noticed the ramp-bending problem he tried to figure out how it worked while I just ignored it, and it was in trying to figure that out that he had his most important insight. I suppose it’s possible I could have had the same thing happen if I’d dug deeper on the places I felt most confused.
The Noticed: Don’t be so hard on yourself. You were streetlamping a bit, but… that’s not quite the same as just not trying hard enough. Most people have an instinct to pour additional effort into the things they know how to do, rather than spend time finding some way to tackle things they don’t know yet how to handle. Getting the ball in the cup does require some work—estimating velocity from pixels in a video is more work than an equation, especially if you know the math—but it’s more about applying the right kind of effort, rather than applying more of it.
Here’s an analogy. How would you get a piece of code to work on the very first try? First, assume there are bugs. There are always bugs. Some particular parts of your calculations may be bug-free, but there will be bugs somewhere. And not just one—you caught the thing about rotational kinetic energy, but that wasn’t the whole story.
Second, in order to find the bugs, you need to go empirically test the parts of the system. You couldn’t do a full end-to-end test in this case, but you could check each of the pieces of the system separately. See if the key numbers you’re relying on—like speed—match your calculations, but also generally look for anything Weird. (Like the ramp bending.) Heck, I don’t even know if the bending was actually the main issue here, but looking at the velocity of the ball at the end of the ramp it’s obvious that something is off with using the standard equations and you need to do something different.
Alison: Well. I wish I had gotten this one right on the first try, but at least I feel like I’ve learned something! Got any other similar problems I can try again on?
The Noticed: Well there’s this thing with AI, see…
This post helped nudge me toward Feedbackloop Rationality, but the stated solution feels kinda cheating – it wouldn’t have occurred to me that doing a complete run-to-the-end-of-the-ramp was allowed (it seems like the rules explicitly imply they can’t?).
I haven’t actually done this exercise, nor used it in my workshops. I think it’s probably a mistake that I haven’t actually tried it at all. As I sit and think about it now, it doesn’t seem that hard to patch the exercise so it doesn’t feel like cheating, and a fixed version seems worth trying at Cognitive Bootcamp.
(I once asked John if he thought the exercise was possible without doing a complete “roll the ball to the end of the ramp”, and he said “I dunno that does seem way harder.”)
It seems like the current implementation of the exercise requires you to figure out you can do a single complete run, and then taking advantage of that. If I were try and fix the instructions, I’d want to make that feel “allowed” without “directly drawing their attention to it.
Some options:
State “some approaches of rolling the ball down are cheating, some are not. You are encouraged to think of options, and ask if they are cheating.”
State “you only get one run from top-to-bottom-of-ramp, but as many partial runs as you want.” I think this will make it too easy for some people, but many still probably won’t think of it.”
State “you get more points the smaller-percentage-of-the-track you use for your tests, and you get 0 points if you don’t get it into the cup after 1 hour.” This changes a vague feeling of dishonor into a rule to measure and evaluate.
Actually figure out if there’s a tractable way of solving it without doing a full-ramp-test. idk.
I would generally either be clear cut about the rules, or clear cut at the meta-level about how to ask what counts as cheating (and be clear internally about what counts).