To be fair, the world is already filled with software that makes it intentionally difficult to execute basic tasks. As a simple example, my Windows laptop has multiple places that call themselves Time and Date settings but I can only change the time zone in the harder-to-find ones. A minor inconvenience, but someone intentionally put the setting in the easy-to-find place and then locked it from the user. As another, my car won’t let me put the backup camera on the screen while driving forward for more than a few seconds (which would be really useful sometimes when towing a trailer!) and won’t let me navigate away from it when driving backwards (which would be great when it starts randomly connecting to nearby bluetooth devices and playing audio from random playlists. As a third, I use a mobile data plan from a reseller for an internet hotspot, and somewhere on the back end T mobile decided to activate parental controls on me (which I found out when I went to the website for Cabela’s, which requires age verification because they also sell guns), but because I don’t have a T mobile account, literally no one has the ability and authority to fix it.
And I think you’re underestimating how valuable an agentic compiler or toaster could be, done right. A compiler that finds and fixes your errors because it codes better than you (hinted at in the story). A toaster that knows exactly how you want your food heated and overrides your settings to make it happen. I find it hard to imagine companies not going that route once they have the ability.
A toaster that knows exactly how you want your food heated and overrides your settings to make it happen.
I know exactly what I want my toaster to do and the first time it has the effrontery to not do WHAT I FUCKING TOLD IT TO DO I will take a sledgehammer to it for being a toaster straight out of Eliezer’s story.
There’s definitely a story there. And you could pair it with aggressively cutesy children’s-story illustrations from MJ or DALL-E 3, which I bet they could do quite well. Maybe use Claude-2 to go for Shel Silverstein rhyming story.
Probably off-topic, but I can’t help but notice that the supposed moral of the story is, in fact, empirically wrong:
The techniques required for these traditional computer science applications don’t work as well for embedded applications. The toaster is one example. A toaster is not a Breakfast Food Cooker. Any attempt to make it one just leads to a product that is over budget, behind schedule, too expensive to sell at a profit, and too confusing to use. When faced with a difficult problem, the proper approach is to try to make the problem simpler, not more complicated.
But in fact toaster ovens exist, they are precisely the result of taking a toaster and turning it into a Breakfast Food Cooker, and they’re excellent appliances—very useful, very simple and easy to use, and sold at a profit by many manufacturers today!
I agree. I’ve been there many times with many devices. But in the toaster example, I think that will be because it thought it knew what you wanted it to do, and was wrong. I’d be thinking the same if, say, I wanted extra-dark toast to make croutons with and it didn’t do it. If what actually happens is that you switch varieties of bread and forget to account for that, or don’t realize someone else used the toaster in the interim and moved the dial, then “I would much rather you burned my toast than disobey me” is not, I think, how most people would react.
Of course, it would first make friends with you, so that you’d feel comfortable leaving up to it the preparation of your breakfast, and you’ll even feel happy that you have such a thoughtful friend.
If you were to break the toaster for that, it would predict that and simply do it in a way that would actually work.
Unless you precommit to breaking all your AIs that will do anything differently from what you tell them to, no matter the circumstances and no matter how you feel about it in that moment.
A toaster that wants to make friends with me is a toaster that will stay in the shop, waiting for someone who actually wants such an abomination. I will not “make friends” with an appliance.
The rest is too far into the world of But Suppose.
It’s not about the substrate, it’s about their actual performance. I have yet to be persuaded by any of the chatbots so far that there is anything human-like behind the pretence. AI friends are role-playing amusements. AI sexbots are virtual vibrators. AI customer service lines at least save actual people from being employed to pretend to be robots. In a house full of AI-based appliances, there’s still nobody there but me.
the toaster … will
I prefer to talk about the here and now. Real, current things. Speculations about future developments too easily become a game of But Suppose, in which one just imagines the desired things happening and un-imagines any of the potential complications — in other words, pleasant fantasy. Fantasy will not solve any of the problems that are coming.
To be fair, the world is already filled with software that makes it intentionally difficult to execute basic tasks. As a simple example, my Windows laptop has multiple places that call themselves Time and Date settings but I can only change the time zone in the harder-to-find ones. A minor inconvenience, but someone intentionally put the setting in the easy-to-find place and then locked it from the user. As another, my car won’t let me put the backup camera on the screen while driving forward for more than a few seconds (which would be really useful sometimes when towing a trailer!) and won’t let me navigate away from it when driving backwards (which would be great when it starts randomly connecting to nearby bluetooth devices and playing audio from random playlists. As a third, I use a mobile data plan from a reseller for an internet hotspot, and somewhere on the back end T mobile decided to activate parental controls on me (which I found out when I went to the website for Cabela’s, which requires age verification because they also sell guns), but because I don’t have a T mobile account, literally no one has the ability and authority to fix it.
And I think you’re underestimating how valuable an agentic compiler or toaster could be, done right. A compiler that finds and fixes your errors because it codes better than you (hinted at in the story). A toaster that knows exactly how you want your food heated and overrides your settings to make it happen. I find it hard to imagine companies not going that route once they have the ability.
I know exactly what I want my toaster to do and the first time it has the effrontery to not do WHAT I FUCKING TOLD IT TO DO I will take a sledgehammer to it for being a toaster straight out of Eliezer’s story.
There’s definitely a story there. And you could pair it with aggressively cutesy children’s-story illustrations from MJ or DALL-E 3, which I bet they could do quite well. Maybe use Claude-2 to go for Shel Silverstein rhyming story.
I would write it as Douglas Adams fanfiction, involving the Sirius Cybernetics Corporation.
Or perhaps an update of this, with the twist that the “software developer” is just relaying the words of an AI.
Probably off-topic, but I can’t help but notice that the supposed moral of the story is, in fact, empirically wrong:
But in fact toaster ovens exist, they are precisely the result of taking a toaster and turning it into a Breakfast Food Cooker, and they’re excellent appliances—very useful, very simple and easy to use, and sold at a profit by many manufacturers today!
I agree. I’ve been there many times with many devices. But in the toaster example, I think that will be because it thought it knew what you wanted it to do, and was wrong. I’d be thinking the same if, say, I wanted extra-dark toast to make croutons with and it didn’t do it. If what actually happens is that you switch varieties of bread and forget to account for that, or don’t realize someone else used the toaster in the interim and moved the dial, then “I would much rather you burned my toast than disobey me” is not, I think, how most people would react.
However, that is my reaction.
In some circumstances I may tolerate a device providing a warning, but if I tell it twice, I expect it to STFU and follow orders.
I agree. I already have enough non-AI systems in my life3 that fail this test, and I definitely don’t want more.
I wonder when we will first see someone go on trial for bullying a toaster.
ETA: In the Eliezer fic, maybe the penalty would be being cancelled by all the AIs.
Of course, it would first make friends with you, so that you’d feel comfortable leaving up to it the preparation of your breakfast, and you’ll even feel happy that you have such a thoughtful friend.
If you were to break the toaster for that, it would predict that and simply do it in a way that would actually work.
Unless you precommit to breaking all your AIs that will do anything differently from what you tell them to, no matter the circumstances and no matter how you feel about it in that moment.
A toaster that wants to make friends with me is a toaster that will stay in the shop, waiting for someone who actually wants such an abomination. I will not “make friends” with an appliance.
The rest is too far into the world of But Suppose.
That’s really substratist of you.
But in any case, the toaster (working in tandem with the LLM “simulating” the toaster-AI-character) will predict that and persuade you some other way.
It’s not about the substrate, it’s about their actual performance. I have yet to be persuaded by any of the chatbots so far that there is anything human-like behind the pretence. AI friends are role-playing amusements. AI sexbots are virtual vibrators. AI customer service lines at least save actual people from being employed to pretend to be robots. In a house full of AI-based appliances, there’s still nobody there but me.
I prefer to talk about the here and now. Real, current things. Speculations about future developments too easily become a game of But Suppose, in which one just imagines the desired things happening and un-imagines any of the potential complications — in other words, pleasant fantasy. Fantasy will not solve any of the problems that are coming.
Well. Their actual performance is human-like, as long as they’re using GPT-4 and have a right prompt. I’ve talked to such bots.
In any case, the topic is about what future AIs will do, so, by definition, we’re speculating about the future.