Generally, single experiments are small data points in exploring a larger question. They don’t resolve it one way or the other, but many experiments might. Was there some larger scientific question you had in mind that you hoped this experiment would shed light on? If so, how did the outcome influence your thinking?
The larger scientific question was related to Factored Cognition, and getting a sense of the difficulty of solving problems through this type of “collaborative crowdsourcing”. The hope was running this experiment would lead to insights that could then inform the direction of future experiments, in the way that you might fingertip feel your way around an unknown space to get a handle on where to go next. For example if it turned out to be easy for groups to execute this type of problem solving, we might push ahead with competitions between teams to develop the best strategies for context-free problem solving.
In that regard it didn’t turn out to be particularly informative, because it wasn’t easy for the groups to solve the math problems, and it’s unclear if that’s because of the problems selected, the team compositions, the software, etc. So re: the larger scientific question I don’t think there’s much to conclude.
But personally I felt that by watching relay participants I gained a lot of UX intuitions around what type of software design and strategy design is necessary for factored strategies—what I broadly think of as problem solving strategies that rely upon decomposition—to work. Two that immediately come to mind:
Create software design patterns that allow the user to hide/reveal information in intuitive ways. It was difficult, when thrown into a huge problem doc with little context, to know where to focus. I wanted a way for the previous user to only show me the info I needed. For example, the way workflow-y / Roam Research bullet points allow you to hide unneeded details, and how if you click on a bullet point you’re brought into an entirely new context.
When designing strategies try focusing on the return signature: When coming up with new strategies for solving relay problems, at first it was entirely free form. I as a user would jump in, try pushing the problem as far as I could, and leave haphazard notes in the doc. Over time we developed more complex shorthand and shared strategies for solving a problem. One heuristic I now use when developing strategies for problem solving that use decomposition is to prioritizing thinking about what each sub part of the strategy will return to the top caller. That clarifies the interface, simplifies what the person working on the sub strategy needs to do, and promotes composability.
These ideas are helpful because—I posit—we’re faced with Relay Game like problems all the time. When I work on a project, leave it for a week, and come back, I think I’m engaging in a relay between past Ben, present Ben, and future Ben. Some of these ideas informed my design of templates for collaborative group forecasting.
All projects are forms of learning. I find that much of my learning time is consumed by two related tasks:
Familiarizing myself with the reference materials. Examples: reading the textbook, taking notes on a lecture, asking questions during a lecture.
Creating a personalized meta-reference to distill and organize the reference materials so that it’ll be faster and easier to re-teach myself in the future. Examples: highlighting textbook material that I expect I won’t remember and crossing out explanations I no longer need, re-formatting concepts learned in a math class into a unified presentation format, deciding which concepts need to be made into flash cards.
Those steps seem related to the challenges and strategies you encountered in this project.
We know that students forget much of what they learn, despite their best efforts. I think it’s wiser not to try hard to remember everything, but instead to “plan to forget” and create personalized references so that it’s easy to re-teach yourself later when the need arises.
I wish that skill were more emphasized in the school system. I think we put too much emphasis on trying to make students work harder and memorize better and “de-stress,” and too little on helping students create a carefully thought-out system of notes and references and practice material that will be useful to them later on.
The process of creating really good notes will also serve as a useful form of practice and a motivating tool. I find myself much more inclined to study if I’ve done this work, and I do in fact retain concepts much better if I’ve put in this work.
Your project sounds like an interesting approach to tackle a related challenge. I’d be especially interested to hear about any efforts you make to tease out the differences between work that’s divided between different people, and work that’s divided between different “versions of you” at different times.
Generally, single experiments are small data points in exploring a larger question. They don’t resolve it one way or the other, but many experiments might. Was there some larger scientific question you had in mind that you hoped this experiment would shed light on? If so, how did the outcome influence your thinking?
The larger scientific question was related to Factored Cognition, and getting a sense of the difficulty of solving problems through this type of “collaborative crowdsourcing”. The hope was running this experiment would lead to insights that could then inform the direction of future experiments, in the way that you might fingertip feel your way around an unknown space to get a handle on where to go next. For example if it turned out to be easy for groups to execute this type of problem solving, we might push ahead with competitions between teams to develop the best strategies for context-free problem solving.
In that regard it didn’t turn out to be particularly informative, because it wasn’t easy for the groups to solve the math problems, and it’s unclear if that’s because of the problems selected, the team compositions, the software, etc. So re: the larger scientific question I don’t think there’s much to conclude.
But personally I felt that by watching relay participants I gained a lot of UX intuitions around what type of software design and strategy design is necessary for factored strategies—what I broadly think of as problem solving strategies that rely upon decomposition—to work. Two that immediately come to mind:
Create software design patterns that allow the user to hide/reveal information in intuitive ways. It was difficult, when thrown into a huge problem doc with little context, to know where to focus. I wanted a way for the previous user to only show me the info I needed. For example, the way workflow-y / Roam Research bullet points allow you to hide unneeded details, and how if you click on a bullet point you’re brought into an entirely new context.
When designing strategies try focusing on the return signature: When coming up with new strategies for solving relay problems, at first it was entirely free form. I as a user would jump in, try pushing the problem as far as I could, and leave haphazard notes in the doc. Over time we developed more complex shorthand and shared strategies for solving a problem. One heuristic I now use when developing strategies for problem solving that use decomposition is to prioritizing thinking about what each sub part of the strategy will return to the top caller. That clarifies the interface, simplifies what the person working on the sub strategy needs to do, and promotes composability.
These ideas are helpful because—I posit—we’re faced with Relay Game like problems all the time. When I work on a project, leave it for a week, and come back, I think I’m engaging in a relay between past Ben, present Ben, and future Ben. Some of these ideas informed my design of templates for collaborative group forecasting.
Thanks for that thorough answer!
All projects are forms of learning. I find that much of my learning time is consumed by two related tasks:
Familiarizing myself with the reference materials. Examples: reading the textbook, taking notes on a lecture, asking questions during a lecture.
Creating a personalized meta-reference to distill and organize the reference materials so that it’ll be faster and easier to re-teach myself in the future. Examples: highlighting textbook material that I expect I won’t remember and crossing out explanations I no longer need, re-formatting concepts learned in a math class into a unified presentation format, deciding which concepts need to be made into flash cards.
Those steps seem related to the challenges and strategies you encountered in this project.
We know that students forget much of what they learn, despite their best efforts. I think it’s wiser not to try hard to remember everything, but instead to “plan to forget” and create personalized references so that it’s easy to re-teach yourself later when the need arises.
I wish that skill were more emphasized in the school system. I think we put too much emphasis on trying to make students work harder and memorize better and “de-stress,” and too little on helping students create a carefully thought-out system of notes and references and practice material that will be useful to them later on.
The process of creating really good notes will also serve as a useful form of practice and a motivating tool. I find myself much more inclined to study if I’ve done this work, and I do in fact retain concepts much better if I’ve put in this work.
Your project sounds like an interesting approach to tackle a related challenge. I’d be especially interested to hear about any efforts you make to tease out the differences between work that’s divided between different people, and work that’s divided between different “versions of you” at different times.