I’ve been notified that this post was nominated as a finalist for the Less Wrong 2023 Review! This is fantastic news, and I’m deeply honored! As part of the notification I was encouraged to write a self-review, with some example prompts like “Do you still endorse this?” and “What further work do you think should be done exploring the ideas here?”
Fiction is pretty Out Of Distribution for Less Wrong posts. I almost didn’t post it, because what is the point? I mean that literally… what IS the point of fiction on Less Wrong? Most often it’s to help demonstrate a different point that’s the real purpose of the post. That doesn’t make for good fiction, any more than “Bobby has three apples, John steals two of his apples, now Bobby has one apple” makes for good fiction. Sure there’s action and maybe even emotion, but the goal was to teach math.
The goal of a Less Wrong post is to teach. To impart some intellectual insight or real-world observation or new (or updated) technique. That is not the goal of fiction. The goal of fiction is to create an emotion experience or to entertain. Truly skilled authors can achieve both a LW and a Fiction goal in the same work, but I am not that skilled. My goal is a story that entertains and creates emotion before anything else, and for me that means sacrificing the ability for a story to teach rationality.
When I workshopped the first draft of the story I was asked ‘Why would the AI work in that way? It’s impossible for it to have come to such a utility function on accident, and literally no one has the motivation to make it have this utility function.’ ‘Why do these two aspects of the AI act like this, that’s not how a functional entity would act given X-Z.’ The answer to all questions of this type were “Yes, I know. This isn’t meant to be a realistic portrayal of what could happen. This is supposed to be fun. And this personality, and these problems that this personally runs into, are actually really fun! Can you point me to the parts that reduce fun or emotion? I don’t care about the story actually being rational.”
This puts the story directly at odds with what a Less Wrong post is supposed to be. I had severe reservations about posting it. But maybe fiction can have good secondary effects in the rationalist world even if its not at all rational. Terminator 2 is extremely non-rational, and IIRC is often cited by Eliezer as a particularly egregious movie because it primed people to think of AIs as dumb things that would use robot-soldiers we can defeat. It sacrificed rational depiction of AI danger on the alter of making an exciting story that an average person could be a hero within.
And yet, T2 probably has done more to bring the mass of the populace on board with worrying about AI risk than any other single thing in our lifetimes. It was easy to understand, very emotionally compelling (which gives it a long lifespan in our memory and a lot of weight in our pre-deliberative intuitions), and extremely popular (which spread its impact wide and gave everyone a common cultural touchstone). Despite being non-rational (or anti-rational) it was a huge boon to the cause of rationality.
So maybe it’s ok to post a story that isn’t actually rational and doesn’t do any of the things a Less Wrong post is supposed to do. Maybe just sharing a fun story on its own can be worth it if it plants an emotional seed that aligns with the rationalist cause. I do think The Real Fanfic is a fun story. I also think it contains the seed of “even a god that superficially looks like us and seems to have our interests at heart can be incredibly creepy and bad, actually, for reasons that aren’t at all obvious.”
So, if my goal is to give something fun to my fellow rats, this is the best place to do so. And if it can advance rationality in some small way it’s probably even worth doing so, despite being a very non-central post. I figured it was worth a shot. A decent amount of post karma, plus a nomination to the 2023 Review, leads me to suspect some people agreed.
So “Do I still endorse this?” It’s story I’m still proud of. :) Whether you enjoyed it or very much didn’t, I’m excited to read any review.
I’ve been notified that this post was nominated as a finalist for the Less Wrong 2023 Review! This is fantastic news, and I’m deeply honored! As part of the notification I was encouraged to write a self-review, with some example prompts like “Do you still endorse this?” and “What further work do you think should be done exploring the ideas here?”
Fiction is pretty Out Of Distribution for Less Wrong posts. I almost didn’t post it, because what is the point? I mean that literally… what IS the point of fiction on Less Wrong? Most often it’s to help demonstrate a different point that’s the real purpose of the post. That doesn’t make for good fiction, any more than “Bobby has three apples, John steals two of his apples, now Bobby has one apple” makes for good fiction. Sure there’s action and maybe even emotion, but the goal was to teach math.
The goal of a Less Wrong post is to teach. To impart some intellectual insight or real-world observation or new (or updated) technique. That is not the goal of fiction. The goal of fiction is to create an emotion experience or to entertain. Truly skilled authors can achieve both a LW and a Fiction goal in the same work, but I am not that skilled. My goal is a story that entertains and creates emotion before anything else, and for me that means sacrificing the ability for a story to teach rationality.
When I workshopped the first draft of the story I was asked ‘Why would the AI work in that way? It’s impossible for it to have come to such a utility function on accident, and literally no one has the motivation to make it have this utility function.’ ‘Why do these two aspects of the AI act like this, that’s not how a functional entity would act given X-Z.’ The answer to all questions of this type were “Yes, I know. This isn’t meant to be a realistic portrayal of what could happen. This is supposed to be fun. And this personality, and these problems that this personally runs into, are actually really fun! Can you point me to the parts that reduce fun or emotion? I don’t care about the story actually being rational.”
This puts the story directly at odds with what a Less Wrong post is supposed to be. I had severe reservations about posting it. But maybe fiction can have good secondary effects in the rationalist world even if its not at all rational. Terminator 2 is extremely non-rational, and IIRC is often cited by Eliezer as a particularly egregious movie because it primed people to think of AIs as dumb things that would use robot-soldiers we can defeat. It sacrificed rational depiction of AI danger on the alter of making an exciting story that an average person could be a hero within.
And yet, T2 probably has done more to bring the mass of the populace on board with worrying about AI risk than any other single thing in our lifetimes. It was easy to understand, very emotionally compelling (which gives it a long lifespan in our memory and a lot of weight in our pre-deliberative intuitions), and extremely popular (which spread its impact wide and gave everyone a common cultural touchstone). Despite being non-rational (or anti-rational) it was a huge boon to the cause of rationality.
So maybe it’s ok to post a story that isn’t actually rational and doesn’t do any of the things a Less Wrong post is supposed to do. Maybe just sharing a fun story on its own can be worth it if it plants an emotional seed that aligns with the rationalist cause. I do think The Real Fanfic is a fun story. I also think it contains the seed of “even a god that superficially looks like us and seems to have our interests at heart can be incredibly creepy and bad, actually, for reasons that aren’t at all obvious.”
So, if my goal is to give something fun to my fellow rats, this is the best place to do so. And if it can advance rationality in some small way it’s probably even worth doing so, despite being a very non-central post. I figured it was worth a shot. A decent amount of post karma, plus a nomination to the 2023 Review, leads me to suspect some people agreed.
So “Do I still endorse this?” It’s story I’m still proud of. :) Whether you enjoyed it or very much didn’t, I’m excited to read any review.