toner
We could do! Will you be here? We’ll at least repeat this meetup on December 21.
Notes from discussion about finding better metrics to measure CfAR’s effectiveness
[Discussion about what CfAR is trying to teach.]
Have to avoid CfAR just measuring how much people are paying attention in their courses
Beforehand: write down goals
Six months later: measure them against their pre-camp goals
Control group
Just attending a camp might make it feel like you’re making more progress towards your goals.
Some goals hard to measure
How much throughput are they expecting? 3 camps/month.
Measuring income: a lot of people might decide during the camp to change career, or devote their life to effective charity, so income may well drop when your rationality goes up.
Also income has a long lag before interventions take effect
epicwin: gamify your life.
We’re going to stop proposing solutions for a while
Suggestion to generate ideas individually.
Desirable qualities: low stat noise, objective, quantitative etc.
What sorts of measures: e.g. based on other’s opinions, performance at a task, other classes
Could talk about what goals people have? Then we could think about how to measure those particular goals. Money, happier, healthier, etc.
Whose perspective are we taking? The consumer of the minicamps.
In self-help field, typically sell books or lecture tours based on how good your stuff sounds. Most of it doesn’t come with metrics. If there’s some of that works (e.g. GTD), then CfAR should be teaching it.
[Tangent about quality of scientific literature.] Problem with finding material to teach by looking for stuff whose effectiveness has already been well measured in the literature is that you’re setting the bar too high. We don’t need to published, so can make do with messier non-laboratory non-ethics-committee-approved measures.
Come up with a test you like. Try to teach to that.
Break up to discuss:
Desirable qualities of metrics
Types of test
Goals that people might have
[We break into groups]
I’m with yurifury (in hammock) + patrick (in bean bag)
Types of tests
Maybe have an interactive test that heaps you make decisions (like it includes a checklist of things to think about and biases to avoid) and it would generate metrics along the way. An iterative process, so you could track how people’s goals change. And if it’s a process to help people, people more likely to use it. Blurring line between self-help tool and pure metric. Or not a tool just for significant decisions, but a weekly thing. People’s goals might change from month to month and a tool that let you say “i’m doing well on this one” “i don’t care about this one anymore” would be helpful and would generate metrics.
An agile version opposed to the current waterfall 1-year-apart surveys pre- and post-minicamp.
Difference between quantified selfers, who measure things to detect correlations, and people who just measure things they’re trying to improve.
They should run a course specifically for quantified selfers. Gwern could be the Matt of the course. Then the could observe what changes about them and they might get less obsessive people to measure those specific things.
Goals
Big problem with motivation: how to get people to take the tests. 1. Have measurements be side-effects of something otherwise useful. 2. Social pressure.
X, what would you like to change about yourself? Answer: spend more time reading. Akrasia.
People who talk regularly following minicamp (already) could evaluate each other. People at meetups could evaluate each other. 5 minute quiz at meetups.
Have an award for the person who changes the most. ← hard to make honest
We seem to be generating more ideas/unit time in our small group, than the group as a whole. Same for other groups? Yes.
We’ll focus on goals that are easiest to measure. Health. Measures like resting heart rate not BMI Facebook friends/klout. Happiness Grades ← not so good Memory/IQish tests
[We decide that the notes above are a really good example of how useless a 12-person group discussion can be. AVOID AVOID AVOID]
The table was observed to be red-shifted.
Good suggestions from other groups:
Possible tests:
Idealogical Turing test as a measure. People learning to model each other. Tests things including: are you able to model other people well, do you understand other people’s ideas. rather labour intensive
Search skill: able to use google. measure how well people are able to find information (with time constraints). Apparently google has a daily challenge.
Get people to do a three minute video sales pitch.
Game playing. Useful metrics in how well you play poker, Wits and Wagers.
Might be worth having preprepared list of goals that are easy to achieve and/or measure and have people select goals from that list
Look at course material and try to come up with tests inspired by course material, but being sure to ask “Why are we delivering this material?” too
[Tangent/derailment]
[Splitting up again, given what we learned. Other group is dancing.]
Summary of what we want to tell CfAR:
Summarize desirable characteristics of goals. Need as common knowledge.
Directions to go in towards gathering metrics
What we learned about how to have this discussion. Talking about goals seemed too general and not useful; discussion about transferable skills seemed more useful. Things that people wanted that came up frequently: Calibrated, appraise evidence, socially competent, ability to persuade others, motivation esp. across different time scales.
Some suggestions of metrics (filter above)
Also we want to post in the meetup organizers group what we learned about group discussions. Benefit of small groups is it’s okay to interrupt.
[At this point my battery about to run out.]
Quixey is incredibly successful. Also, LessWrong is still young. Give it time! There may be a bunch of startups out there we haven’t heard of yet. For example, I’m doing a startup with 3 other LWers, but we need a little longer before we’re successful ;-)
I can come and I’d be happy to do the exercise I suggested last month
http://lesswrong.com/lw/bym/meetup_melbourne_practical_rationality/6gcz
but I’m thinking we’d probably prefer to hear from James and Scott about the minicamp (if they’re willing).
Edit: Let’s save the updating activity below for another night or for next month, when Aubrey de Grey isn’t speaking.
We decided that we’d take it in turns to prepare activities for the practical rationality meetups, and I volunteered for this meetup.
Here’s the plan. I have a bunch of trivia questions with numerical answers. For each question: Everyone records either 50% or 90% confidence intervals. Open discussion. Everyone may alter their intervals. Record initial and updated calibration. The aim is to test our calibration (is it true that approximately half your answers lie within your 50% confidence intervals?), ability to update (are your post-discussion guesses better?), and ability to efficiently persuade a group about something you are relatively certain about.
I propose we do both the initial estimation and discussion under moderate time pressure (i) because when you need to update in real life you often don’t have much time, and (ii) so the whole thing doesn’t take too long. We’ll have to work out exactly how much time that should be but let’s start with say 30 seconds to write down intervals and 1 min for discussion.
I’ll reveal each answer immediately after and we can then take as long as we need to discuss how we’re doing, so hopefully we get better as we go along. We’ll be able to measure this, so at the end of the night we’ll have data that will tell us if the activity was useful or not.
Suggestions are welcome.
You just missed it.
It’s not a zero-sum game!
Sadly I can’t make it tonight. I find myself in the wrong timezone and I’ll be asleep.
Thanks for organizing this Adam, I enjoyed it!
There was a strong consensus that we wanted to keep meeting, and I made a google group for us to plan activities, discuss what sorts of things we might like to do in the future, let each other know of interesting events, and so on.
At the moment the group is semi-private, meaning that only members can view content but anyone can join. There is no moderation. We can change this if there is a consensus to do so or as it becomes necessary.
I’ll make AdamBell, Patrick (who organized the previous meetups) and matt admins, unless they object.
If you’re in Melbourne, please join and perhaps introduce yourself!
http://groups.google.com/group/melbourne-less-wrong/
I’ll come but will probably be a bit late.
Many jobs, including almost all of those that people would do on a working holiday, have Award rates higher than the minimum wage. Effectively, in Australia the minimum wage depends on the job. $21 is probably the minimum allowed by whatever Award governs shokwave’s employment, either the Fast Food Industry Award or KFC might have their own enterprise agreement.
If anyone cares, the place to learn about this is here.
I’ll come.
I agree, start with no limit hold’em because there’s an awful lot of good learning material about it and the games at low limits are pretty good, but at some point consider switching to pot limit omaha.
I think most online sites are 18+.
Most people use databases and heads-up displays, but to calculate and present statistics about your own and your opponents’ play, not to calculate odds (calculating odds is easy). I like Poker Tracker.
Read some of the books published by Two Plus Two for solid beginner information that’s mostly a little out of date; then sign up at a video training site (I like Deuces Cracked) for up-to-date information; finally go, e.g., here and accept one of the offers where they give you free money to play with and then use their money to practice at 1 cent/2 cent games.
Also, maybe ask this question at the Two Plus Two forums for a better response.
Go ahead! But it’s hard.
I hear that everywhere too. It’s a selection effect: most of the population aren’t smart and rational enough to be long-term winning players and it’s these people you hear complaining, while the good players go on quietly winning.
It’s definitely true that the games are getting tougher every year, because the community is learning to play better, so the threshold of ability you need to be a winning player is constantly increasing. But it’s not that high yet.
Now let’s talk about your two bugbears, bots and collusion.
1. Bots
You never ever have to worry about bots. The goal in poker is to seek out and play against bad players, while tolerating the presence of good players. It’s completely irrelevant whether these players are controlled by humans, machines, or some combination. (In practice, except possibly for heads-up limit hold’em, good players are still better than the best bots published in the academic literature anyway.)
2. Collusion
This is something you have to worry about, but in practice it’s not that big a deal, especially if you play at low limits, where it’s not going to be worth the bother for competent players to collude. There have been only a handful of times when I’ve suspected collusion online, in which case, the obvious response was to stop playing against those players. Sometimes collusion can be detected statistically, but if some collusion does go undetected, as long as you’re winning, what does it matter?
If you’re truly smart, truly rational, and with the goal function you describe in your post, an obvious answer is to play poker on the internet. But beware: if it turns out you’re not actually as rational as most of us on Less Wrong think we are, it probably won’t work out.
Thank you for organising this. I’ll come. It’s a good choice of venue.
We’ll organize something. I’ll talk to people about it at this coming meetup. There are some other non-local Less Wrong people in Melbourne right now too.