Using karma for like/dislike and having a separate Arbital-style probability meter for every post and comment (plus an option to insert a probability meters into a specific part of your post) might get you a lot of the value of this, and it seems like a different genre from “buttons” in the sense Zvi is talking about.
Specifically, I would have every post and comment automatically come with a probability meter at the bottom for “How confident are you that all the key claims in this (post/comment) are true?”, which can maybe be disabled on a case-by-case basis. I’d expect people to normally just intuit what the key claims are, but people could also be encouraged by the existence of this feature to include an explicit tl;dr at the bottom or top of their post listing what they see as their key claims. (I’d consider it valuable to have the ‘whole-post/comment’ probability meter consistently ask this question, so that people can easily scroll quickly down a page and eyeball the probability meter for each comment/post without the extra mental overhead of needing to worry about whether the ‘whole-post/comment’ probability meter represents something fundamentally different in each case.)
Separately, I would also include a feature for inserting an arbitrary number of probability meters into the body of a post or comment, explicitly associated with particular claims.
(When people have multiple “key claims” and choose to explicitly list them, it can also be standard to include a probability meter for each one, in which case the whole-post/comment probability meter may not be useful (unless some people feel comfortable weighing in on the conjunction but not the conjuncts, which I could imagine happening for a variety of reasons). I see this redundancy as more or less harmless, and would still consider it useful in this case to make each-post/comment-gets-a-single-top-level-probability-meter a site-wide standard, both because it would make it possible to see the probabilities different entire posts or comments got in indexes of those posts and comments (e.g., “All posts” or “Featured posts”), and because maintaining this norm would make it more normal and cognitively available to do a lot of probability assignments.)
I would suggest making people’s probability judgments non-anonymous by default (to encourage accountability and conversation, and to keep the site from devaluing globally unpopular opinions that are held by high-status contrarians), but allowing them to switch to anonymous judgments if they want. I would also suggest marking when each probability judgment was made, and encouraging people to make periodic follow-up comments in the case of important/interesting things they’ve updated on, in which they give their new probability judgments for the exact same claim, without overwriting the record of their old judgment.
(I think overwriting is OK too, though, provided that there’s some way for users to access their own old judgment in case it’s important later (or for mods to find them), and provided that the probability judgments receive dates. The ability to “remove” an old probability judgment seems generally good insofar as it makes people less worried that they’ll have an embarrassing mistake immortalized on the Internet forever, and therefore encourages them to try assigning probabilities to things more. Ideally people would never remove old judgments, even if they misunderstood the thing they were answering—the probability you’re misunderstanding a question should always be factored into your probability assignments—but I don’t think this is a case where we want to force people to do the ideal thing.)
I assume a lot of people would abuse probability meters for all sorts of trivial/weird/meta things, once they became commonplace. E.g., “Proposition: I did a good job summarizing my key claims above.” “Proposition: Most people will assign <50% probability to this proposition.” “Proposition: You found OP confusing.” “Proposition: If you threw a dart at a random point on a dartboard whose target was as large as a proportion of dartboard as the amount you liked the new Blade Runner movie is high compared to how much you like your favorite movies, then you would hit the target.” “Proposition: You’ll join me at the MIRIxLA meetup tomorrow at noon.” I consider this an actively good thing and want to be in a culture where probability assignment is so ubiquitous that it gets incorporated into trifles and in-jokes and gets appropriated for a variety of uses.
Most of these ideas seem good in isolation (or pointing in the direction of good things). I think they’d add up to significant complexity cost for the page, so figuring out to what degree they are worth adding to the overall cognitive load for the site will be an issue.
I agree that we should be very careful about adding complexity, especially when the complexity is added to every single post and comment. I can think of a few things that might reduce the complexity:
1. Instead of having a big eye-catching Arbital-style visualization of the probabilities displayed for every post and comment on the site, display an aggregate probability in boring grey text that matches the other text, and have people hover/click on that text to view/predict. E.g., your comment header could look like this:
Raemon +2 votes ∧ ∨ 𝗣 ≈ 0.9 6h
You could still include the full Arbital-style visualization within posts and comments, but it would be a deliberate choice by the post/comment author, rather than being a default. In cases where not enough people have assigned probabilities to the post/comment for the system to think it’s worth displaying an aggregate probability, the default visualization will be 𝗣 = ?.
2. Use a functionally similar (though visually distinct) click-and-drag sliding scale for the voting system’s karma-weighting that we use (see above) and for assigning probabilities, so some of the basic habits and motor intuitions people build up with karma can also be used for probability.
In general: My vague understanding of Oliver and co.’s vision for LessWrong is that LessWrong is to be a site where probability assignments, predictions, cruxes, bets, etc. play a huge role. Having easy infrastructure for making and comparing probability assignments might be more of a core feature than the full range of “buttons”, particularly if some of the key buttons can themselves be implemented as probability assignments.
More specifics of how I might implement a probability system like this:
Before you can assign probabilities, you need a “level-1 calibration” badge, achieved via hitting a certain calibration level in a LW/CFAR game/app. (You can then get level-2, level-3, etc. calibration badges for even better performance, maybe unlocking other site features like karma-betting markets.)
Everyone’s probability assignments (which are non-anonymous by default) can always be viewed by clicking or hovering on a “collapsed” probability (i.e., one that look like 𝗣 = foo instead of like an Arbital probability distribution image).
Collapsed probabilities will display as 𝗣 = ? until, e.g., some number of users with at least 2000 karma between them have assigned probabilities to that comment/post. Some aggregated probability will then be displayed for “foo” in 𝗣 = foo, with better-calibrated users (according to badge count) receiving more weight in the aggregation.
The main purpose of hiding the probabilities behind 𝗣 = ? until enough high-karma users have weighed in is to discourage low-karma users from getting really excited and wasting time running around and assigning probabilities to every comment on the site. If someone feels like doing that, that’s totally fine — maybe they’ll learn something from the process — but those probabilities shouldn’t be prominently displayed, because a lot of comments on the site are things like “I agree!” or “Woah.” where it doesn’t really matter if someone decides to waste their time adding silly probability assignments, but it does start carrying a cost if this makes silly probability assignments distracting and visible to anyone visiting the page.
(Note that users’ karma totals do not affect the weight users’ assignments receive in the probability aggregation at all, even though it affects how prominently the probabilities are displayed. On the other hand, it might be fine to weight users more if they have badges for things other than calibration, e.g., a “general knowledge” badge reflecting that you’re unusually good at answering Jeopardy questions or what-have-you.)
Using karma for like/dislike and having a separate Arbital-style probability meter for every post and comment (plus an option to insert a probability meters into a specific part of your post) might get you a lot of the value of this, and it seems like a different genre from “buttons” in the sense Zvi is talking about.
Specifically, I would have every post and comment automatically come with a probability meter at the bottom for “How confident are you that all the key claims in this (post/comment) are true?”, which can maybe be disabled on a case-by-case basis. I’d expect people to normally just intuit what the key claims are, but people could also be encouraged by the existence of this feature to include an explicit tl;dr at the bottom or top of their post listing what they see as their key claims. (I’d consider it valuable to have the ‘whole-post/comment’ probability meter consistently ask this question, so that people can easily scroll quickly down a page and eyeball the probability meter for each comment/post without the extra mental overhead of needing to worry about whether the ‘whole-post/comment’ probability meter represents something fundamentally different in each case.)
Separately, I would also include a feature for inserting an arbitrary number of probability meters into the body of a post or comment, explicitly associated with particular claims.
(When people have multiple “key claims” and choose to explicitly list them, it can also be standard to include a probability meter for each one, in which case the whole-post/comment probability meter may not be useful (unless some people feel comfortable weighing in on the conjunction but not the conjuncts, which I could imagine happening for a variety of reasons). I see this redundancy as more or less harmless, and would still consider it useful in this case to make each-post/comment-gets-a-single-top-level-probability-meter a site-wide standard, both because it would make it possible to see the probabilities different entire posts or comments got in indexes of those posts and comments (e.g., “All posts” or “Featured posts”), and because maintaining this norm would make it more normal and cognitively available to do a lot of probability assignments.)
I would suggest making people’s probability judgments non-anonymous by default (to encourage accountability and conversation, and to keep the site from devaluing globally unpopular opinions that are held by high-status contrarians), but allowing them to switch to anonymous judgments if they want. I would also suggest marking when each probability judgment was made, and encouraging people to make periodic follow-up comments in the case of important/interesting things they’ve updated on, in which they give their new probability judgments for the exact same claim, without overwriting the record of their old judgment.
(I think overwriting is OK too, though, provided that there’s some way for users to access their own old judgment in case it’s important later (or for mods to find them), and provided that the probability judgments receive dates. The ability to “remove” an old probability judgment seems generally good insofar as it makes people less worried that they’ll have an embarrassing mistake immortalized on the Internet forever, and therefore encourages them to try assigning probabilities to things more. Ideally people would never remove old judgments, even if they misunderstood the thing they were answering—the probability you’re misunderstanding a question should always be factored into your probability assignments—but I don’t think this is a case where we want to force people to do the ideal thing.)
I assume a lot of people would abuse probability meters for all sorts of trivial/weird/meta things, once they became commonplace. E.g., “Proposition: I did a good job summarizing my key claims above.” “Proposition: Most people will assign <50% probability to this proposition.” “Proposition: You found OP confusing.” “Proposition: If you threw a dart at a random point on a dartboard whose target was as large as a proportion of dartboard as the amount you liked the new Blade Runner movie is high compared to how much you like your favorite movies, then you would hit the target.” “Proposition: You’ll join me at the MIRIxLA meetup tomorrow at noon.” I consider this an actively good thing and want to be in a culture where probability assignment is so ubiquitous that it gets incorporated into trifles and in-jokes and gets appropriated for a variety of uses.
Most of these ideas seem good in isolation (or pointing in the direction of good things). I think they’d add up to significant complexity cost for the page, so figuring out to what degree they are worth adding to the overall cognitive load for the site will be an issue.
I agree that we should be very careful about adding complexity, especially when the complexity is added to every single post and comment. I can think of a few things that might reduce the complexity:
1. Instead of having a big eye-catching Arbital-style visualization of the probabilities displayed for every post and comment on the site, display an aggregate probability in boring grey text that matches the other text, and have people hover/click on that text to view/predict. E.g., your comment header could look like this:
Raemon +2 votes ∧ ∨ 𝗣 ≈ 0.9 6h
You could still include the full Arbital-style visualization within posts and comments, but it would be a deliberate choice by the post/comment author, rather than being a default. In cases where not enough people have assigned probabilities to the post/comment for the system to think it’s worth displaying an aggregate probability, the default visualization will be 𝗣 = ?.
2. Use a functionally similar (though visually distinct) click-and-drag sliding scale for the voting system’s karma-weighting that we use (see above) and for assigning probabilities, so some of the basic habits and motor intuitions people build up with karma can also be used for probability.
In general: My vague understanding of Oliver and co.’s vision for LessWrong is that LessWrong is to be a site where probability assignments, predictions, cruxes, bets, etc. play a huge role. Having easy infrastructure for making and comparing probability assignments might be more of a core feature than the full range of “buttons”, particularly if some of the key buttons can themselves be implemented as probability assignments.
More specifics of how I might implement a probability system like this:
Before you can assign probabilities, you need a “level-1 calibration” badge, achieved via hitting a certain calibration level in a LW/CFAR game/app. (You can then get level-2, level-3, etc. calibration badges for even better performance, maybe unlocking other site features like karma-betting markets.)
Everyone’s probability assignments (which are non-anonymous by default) can always be viewed by clicking or hovering on a “collapsed” probability (i.e., one that look like 𝗣 = foo instead of like an Arbital probability distribution image).
Collapsed probabilities will display as 𝗣 = ? until, e.g., some number of users with at least 2000 karma between them have assigned probabilities to that comment/post. Some aggregated probability will then be displayed for “foo” in 𝗣 = foo, with better-calibrated users (according to badge count) receiving more weight in the aggregation.
The main purpose of hiding the probabilities behind 𝗣 = ? until enough high-karma users have weighed in is to discourage low-karma users from getting really excited and wasting time running around and assigning probabilities to every comment on the site. If someone feels like doing that, that’s totally fine — maybe they’ll learn something from the process — but those probabilities shouldn’t be prominently displayed, because a lot of comments on the site are things like “I agree!” or “Woah.” where it doesn’t really matter if someone decides to waste their time adding silly probability assignments, but it does start carrying a cost if this makes silly probability assignments distracting and visible to anyone visiting the page.
(Note that users’ karma totals do not affect the weight users’ assignments receive in the probability aggregation at all, even though it affects how prominently the probabilities are displayed. On the other hand, it might be fine to weight users more if they have badges for things other than calibration, e.g., a “general knowledge” badge reflecting that you’re unusually good at answering Jeopardy questions or what-have-you.)
Is there an existing CFAR/LW calibration app that we consider good? (I know there have been attempts but haven’t actually used them myself)
New thing: https://www.openphilanthropy.org/blog/new-web-app-calibration-training
This one broke for me a few times :/
I actually also think the UI design and feedback mechanisms are a lot worse, so I would recommend that people still use the old one.