Michael Simm

Karma: 13

Michael Simm 26 Nov 2022 21:59 UTC
2 points
2
in reply to: Lao Mein’s comment on: Introducing The Logical Foundation, A Plan to End Poverty With Guaranteed Income
We need to take a bit of a step back here. I am just as keen as you are to get good unbiased data that can be relied upon to know precisely how impactful cash transfers—and other interventions—are at helping people. And I’d like to acknowledge that you’re correct that we should not rely solely on unchecked survey data when trying to figure out impact results.
So I did a bit of a deep dive into meta-analyses and systematic reviews of cash transfer studies. It turns out that while surveys are generally one part of the data collected, researchers have been able to use objective measures and collect data that participants can’t lie about (such as government income reporting, and school attendance rates among others). So I figure you might find it useful to check out the systematic reviews, and the organizations with tons of statisticians that have likely accounted for the very real problems you’ve pointed out with surveys.
https://odi.org/en/publications/cash-transfers-what-does-the-evidence-say-a-rigorous-review-of-impacts-and-the-role-of-design-and-implementation-features/
We all know GiveWell, they do extremely rigorous analyses of studies, and came to this conclusion:
Bottom line: Cash transfers have the strongest track record we’ve seen for a non-health intervention, and are a priority program of ours.
Their focus is primarily on cash transfers in developing nations, but I think their point about non-health interventions is especially interesting when considering how effective health interventions are much harder to come by in developed nations. There are several research institutions entirely dedicated to sussing out the effectiveness of cash transfers. They have tons of statisticians, and I’m sure they actively account for the issues with surveys versus other data types.
https://as.nyu.edu/departments/cash-transfer-lab/faq.html
https://basicincome.stanford.edu/research/ubi-visualization/
https://www.penncgir.org/research
I would like to see more research comparing cash directly to other interventions, I think there are a surprisingly large amount of current interventions going on that are either marginally helpful or actively harmful to beneficiaries. The most important thing in philanthropy is diverting as much funding as possible, at a global scale, to the interventions that we know are the most highly effective.
You’ve mentioned a few times that I need a statistician for what we’re doing, and I fully intend on going even further. When we have a large enough experiment to run we will not only hire a statistician but have unbiased external scientific institutions operate the research alongside our program (such as the research labs linked above). That way we cannot manipulate their results. I am very interested in falsifiable studies that can either indicate we’re going in the right direction or prevent us from wasting a lot of time on something that isn’t impactful.
Finally, I’m well aware that guaranteed income regarding homelessness is far smaller and much less rigorous than cash transfers in general or globally. However we’ve spoken to dozens of homeless aid workers as well as homeless people, and we’ve seen promising anecdotal results from all of the (imperfect) small to medium-scale studies done so far. As long as we’re conducting good large-scale research, and (good of you to point out) not relying entirely on surveys or other potentially biased metrics, I think doing larger-scale and higher-quality research is well worth EA dollars.
If the larger studies prove what we think we see happening on a small scale, we could solve the homelessness crisis with ~10% of the current amount we spend on the problem. If the large-scale studies show less impact, but are still in the same range as general cash transfer research, then guaranteed income would still be the most cost-effective way to help the homeless, just not 5-10X better than the other methods as it currently looks.
Overall, better research is needed, and we fully intend to produce highly consequential studies once we have the funding to launch a statistically relevant program. If we could get an EA statistician to look over the current homelessness research, they would be far more qualified than you or I to know just how much trust we should be putting in the research done so far. Do you know any?

Michael Simm 22 Nov 2022 23:17 UTC
2 points
2
in reply to: Lao Mein’s comment on: Introducing The Logical Foundation, A Plan to End Poverty With Guaranteed Income
Our organization is not big enough to hire a statistician, although we will for sure get one when we are able to build a sufficiently large study / program. I’d be happy to refer you to the people that do have a ton of statisticians:
https://www.givedirectly.org/research-at-give-directly/
https://basicincome.stanford.edu/research/ubi-visualization/
https://www.penncgir.org/research
Let’s use a different analogy. Let’s say that you are in exactly the same situation you are in right now, and some random organization decided to start giving you $1,000 checks every month for one year. All they want, is periodic updates on how you’re doing, and they tell you that your answers are anonymized and will not affect the payments. Would you go out of your way to lie to them?
We are not trying to be anyone’s parents, and have no desire for the weird inter-personal shame dynamics that would be going on in your analogy.

Michael Simm 22 Nov 2022 23:01 UTC
2 points
2
in reply to: Lao Mein’s comment on: Introducing The Logical Foundation, A Plan to End Poverty With Guaranteed Income
Good thing I’m not particularly interested in being a charity, I’m interested in building a tool for funders and fundraisers that maximizes their impact against poverty. So bring on the criticism.
That said, I think you’re being excessively negative towards surveys in general. There are two primary reason why surveys are (seemingly) the only way to go about getting decent data, and not nearly as unreliable as you’re suggesting.
1. I can’t think of another way to go about collecting more accurate & less biased information. The only way 100% know exactly what even one person is spending their money on and doing every day is to have a researcher follow them around, watch exactly what they do, and record every change in their life over 6 months. Doing that would likely result in way bigger behavior changes than any survay ever would. It would also be expensive, extraordinarily undignified. Can you think of a single way to actually go about getting the detailed factual information that would not cause far larger behavior changes or be horrifically undignified (like a tiny bug drone that flies around watching people, and inviting lawsuits)?
2. While people may feel an obligation to give good answers about drug use, for example, funding has not been determinant on the answers to the questions. Why would anyone go out of their way to lie about an easily checkable questions like: Where are you living at? Do you have a job? Or what is your net worth? People have no incentive to missstate answers to those questions.
It’s also important to not that it is not just survays. The New Leaf Project survay, and many of GiveDirectly’s research includes in-person interviews. Interviews that often take place in the homes that participants have attained because of the program. I’ve never seen a single example of a random in-person check-in discovering that a participant is actually still homeless, having lied about attaining a stable place to live.
With regards to the database, that is all of the research out there across all time with anything to do with cash transfers. I cannot say that guaranteed income is 100% proven against homelessness because there are only a handful of studies on that. Simmilarly, you should know better than to pick out two random studies. We should both be relying on the big studies looking into tons of other studies, like the one from this graph: Graph adapted from Bastagli, Zanker, et al (2016)
In conclusion
It seems to me that you have created a completely unfalsifiable position that survay recipients will systematically misrepresent their ideas. This isn’t an argument about guaranteed income (correct me if I’m misunderstanding ofc), it seems to be suggesting that all survays, especially those where participants have been helped in any way, are fundamentally unreliable because participants are inherently untrustworthy. Any study to analyze this argument would inevitably cause greater misrepresentations compared to the counterfactual.
Finally, and I actually very much enjoy the discourse, but insight can be gained by looking from a rational angle as well as research-based. What would you do if given a guaranteed income ($1,000 a month for a year)? Would you randomly start reporting demonstrable falsehoods to researchers on survays even if there was no reason to? If not, why assume that many (if not most) other participants would?

Michael Simm 22 Nov 2022 7:00 UTC
3 points
7
in reply to: Lao Mein’s comment on: Introducing The Logical Foundation, A Plan to End Poverty With Guaranteed Income
I do not agree that hospital visits and incarceration statistics, (although I’d love to have those numbers), are foundational to measuring the impactfulness of a homeless intervention:
Overwhelming demographic data as well as medical analysis make it evident that living on the streets directly accounts for most, though not all, of the massive mortality rate increase. There is a causal relationship between living on the streets and high death rates, especially in Arizona due to the high summer heat.
I’d be happy to hear why you might disagree, but I believe it’s well-established ^ that having to live on the streets directly results in almost all of the complications, hospital visits, incarceration, and other service costs that are incurred trying to keep homeless people alive. Not the other way around.
Having stable housing is by far the most important objective measure, followed in my opinion by:
1. Housing Situation (the major determinant of hospitalizations & incarceration)
2. Employment situation (are they going to fall back on the streets)
Under these, there are several other significant, although not foundational, metrics:
1. Net Worth (how much of a cash cushion do they keep over time)
2. Quality of Life
3. hospitalization & incarceration metrics
4. Spending Metrics
To be honest, I hadn’t even thought of adding hospitalization and/or incarceration numbers until you brought them up. That said, I will add them to our pool of desired data points.
While I do see your point about surveys maybe not resulting in accurate drug usage numbers, I don’t think they are all that important to the effectiveness of the intervention for the reasons I stated in the last post. They are the only subjective parts of these surveys that might be skewed due to cognitive bias.

Michael Simm 21 Nov 2022 23:51 UTC
1 point
1
in reply to: Slider’s comment on: Introducing The Logical Foundation, A Plan to End Poverty With Guaranteed Income
Those people are trying to persuade the whole public at first and then moving. With this apporoach we first move and then show it was a good thing. Sure need to get funders on board but private money pushing ahead of public policy is a “shoot first, ask permission later” approach.
You’re pretty much right. I started my journey writing about UBI policy and its potential to improve society, but I got fed up with politics. I do think the danger isn’t quite as bad at first. We basically have permission to ‘shoot’ because guaranteed income pilots are very common nowadays and tons of cities and private organizations have launched guaranteed income pilots. We will have to ensure that our platform cannot violate IRS rules about who qualifies as ‘in need’.
Sure more things become possible but pushing an old system aside means everybody invested in the old way will have their survival instincts triggered. You don’t want to be on the business end of capitalism defending itself. So knowing which structures are (felt) existentially central is key to letting sleeping bears lay.
So basically, watch our backs when we get big enough for entrenched interests to start losing their grip on exploited labor. That’s a good piece of advice, although, I think we may be able to get wealthy & powerful people to at least claim to support guaranteed income. I think it will be important for us to leave the more political side of it all to other organizations such as Income Movement and the Economic Security Project
With messaging, I see it as framing guaranteed income as the most powerful possible way to help almost any group of people in any location, and back that up with tons of falsifiable research. It’ll be pretty tough to attack us I think. At the same time, lots of technologists and powerful people think that guaranteed income is the only thing that can save capitalism and ensure prosperity during a time of automation.
A 20 M program would fund only 3K participants for 1 year at $500 a month. For perspective, there are 13.5K people homeless in Arizona.
And 13.5K people are approximately 0.02% of the population in Arizona. We would have to be throwing around billions (and that is the goal between 5-10 years) to make any noticeable macroeconomic impacts on the labor force.
Say you have a 20 M program running in a country. Yay, people are free to self-improve. But it also means they are not doing their previous income activity. Will there be a search for all those task that these people were doing? Probably. And because they were the bottom 20 M the replacements will not do it for cheaper (this is somewhat unique condition for this setup (normal economic thinking assumes that others step up)). Sure some tasks are not worth the new cost and the increased demands might not be that much. But the switchers will reduce laborers for the next wage class (and this recourses as much as it needs to). Doing it at scale means firms feel the statistical impact. Or is the bid that they could full-time self-improve but will statistically significantly choose not to?
The data we’ve seen from basic income pilots at all scales around the world indicate that guaranteed income may actually increase labor force participation. There are examples of people using it to get an education or do socially valuable care work, but overall RTC study examinations have seen statistically insignificant, slightly increased rates of working.
The major problem most homeless people encounter is that they cannot get jobs, because of various obstacles such as having clean interview clothes, a home address, a bank account, stolen IDs, etc… From the limited prior studies on guaranteed income for homeless and housing insecure people, guaranteed income seems to enable them to gain employment. Unlike conventional benefits, guaranteed income doesn’t force people to stay unemployed or refuse promotions by having a benefits cliff.

Michael Simm 21 Nov 2022 20:53 UTC
2 points
1
on: Introducing The Logical Foundation, A Plan to End Poverty With Guaranteed Income
Thanks for reading this 30-minute thing. I first wanted to make a short 5-minute read but I realized that many of you would probably really want all of the evidence laid out clearly, and our plan explained in excruciating detail. - so you can point out the super obvious reason why this has a 0% chance of success, that I’ve completely overlooked - despite my search for fundamental issues since I came up with the idea, and asking all of the experts I can find.
The EA community is probably the most knowledgeable community in the world about helping people. Considering the world-changing impact potential I’ve outlined, even a 1% chance of success would justify spending millions to make this happen. Unless of course, someone can find a problem.
Please find a fundamental, first-principles problem with my plan, and, if you can’t, please help us succeed. My challenge for you: Find an insurmountable problem, or help us change the world.
I think our odds are more like 40% without support from EA organizations and as high as 75% with your support. The faster we can grow (but not too fast), the faster we may be able to end poverty. Time is very much of the essence.

Michael Simm 21 Nov 2022 20:49 UTC
1 point
0
in reply to: Slider’s comment on: Introducing The Logical Foundation, A Plan to End Poverty With Guaranteed Income
Taking out at scale the most extractive labour globally will both do a lot of good and draw the ire of the most aggressive economic players. Making a chair taller by chopping of a leg for building materials is not a repeatable strategy. One may not like what is going on in the kitchen, but serving unprocessed ingredients will improve nutrition and will drive down patreonage.
I think I’m missing your point. UBI is a long way off, but there are a lot of (mostly economic theory) writings about how guaranteed income at a large scale would drastically shift power in our economy/society in a positive way for people across the wealth spectrum. It might draw ire, but It’s totally worth doing. I just hope we don’t get assassinated
For the logistics, I don’t see what service is being provided for the help direction. Sure, using a develped country means that whatever personal need the receipient has there is a specialised industry to serve that. And for undeveloped countries an existing industry is proven to be viable and possible instead of illfitted outside injection. But money used in undeveloped country will suffer from the local infrastructure not being, well, developed. Using a definition of dollars for value creates a big distortion there. Sure, if I need car in developed country and pay $1000 for it and pay in undeveloped country £3000 in one sense both are “100% money to need fulfillment”. But on another view the other option is three times less efficient. Providing a $2000 car in the poor area is in one view half as efficient as a rich area car but on another view significantly more efficient than the local poor infrastructure option.
I think it could be important to take a look at the many examples of people in developed nations sending stuff to developing countries that they don’t actually need, or want, the most. A car might be cheaper to deliver than a local car would be to buy, but most of the lime locals are not looking to buy a car, they need something else that is specific to them or their village. You have to be comparing the counterfactual value gained by beneficiaries if they were given $2,000 versus the $2,000 cost (including all logistical and administrative costs) of delivering a certain item such as a car.
I also can’t think of any other metric than dollars, and it does account for examples wherien not giving people money could be more efficient. A case point would be GiveWell’s Top Charities, they don’t give cash ‘value’ but instead, they give cheap, easily distributable items that have incredibly large life-saving impact ‘value’. Money value, lives saved value, economic multiplier effects value… all of these things can be most easily converted to dollars for comparison—although none of this is easy at all.
Yeah, hello Pascal’s mugger. It was atleast interesting when there were numbers about the amount of rooms in the castle making up the deficiencies in building materials. In the wake of FTX big payouts on tail probabilities might be a bit of a challenging sell.
Hi Pascal! But in all seriousness, I don’t think that there’s a near-zero chance. Given the research, I’ve outlined, and the input that I’ve received from a bunch of unbiased professionals in the nonprofit industry, I think the likelihood is closer to 40% if we do not get any support from EA, and over 75% if we do get tons of support. Unlike FTX’s BSBF I’m not interested in a microscopic chance of massive impact, I’m interested in a likely chance of massive, within a decade, impact. The comment was basically meant to ask,” How low would the likelihood of these impacts happening have to be before not dropping everything and working hard on this problem would not be worth it”. Perhaps an unhelpful question.
What I’m looking for is people like you to please find a fundamental first-principles problem with the plan.
You’re right I probably should make our actual chances clearer though. I don’t want to argue about 0-1% chance, I’d like people to argue that our chances are not 40%, but 0 instead.

Michael Simm 21 Nov 2022 19:56 UTC
2 points
2
in reply to: Lao Mein’s comment on: Introducing The Logical Foundation, A Plan to End Poverty With Guaranteed Income
First, thank you for your rigor in analyzing the homelessness part of the post! I most certainly agree with you that cash transfers—explicitly relating to homeless individuals—need more studies and more rigorous RTCs from independent sources.
According to surveys given to the participants. Even if you tell them that the payments aren’t contingent on positive results, they don’t necessarily believe you. And even if they do, they’ll still feel obliged to give you the results you’re after. This is a commonly known effect in psychology and sociology, but it was never addressed.
Thanks for pointing this out, I completely agree. I’ll add this to the document as an important disclaimer.
That said, I also want to add a modifier talking about how the general public is also full of drug users (have you had beer/wine/cannabis in the last year?) and we don’t judge them at all. There’s a big difference between drug use and drug addiction, and there’s a very strong argument that we probably shouldn’t care about drug use metrics at all. It’s incarceration rates & hospital visits that we need more data on. It’s also kinda impossible to accurately determine who does how many drugs without extremely invasive & expensive oversight.
Secondly, there are no comparisons with other programs, like directly providing housing. What exactly is the relative benefit of direct cash transfers over conventional welfare programs like food stamps? Less overhead? It seems marginal at best.
I did try to compare with housing first, especially about the budgets spent per ‘unit’. I think that producing academic studies on these topics is essential over the next few years, I want to keep our work as falsifiable as possible.
Comparisons differ from intervention to intervention, but it comes back to the snake charts that I made right near the top.
It’s not just overhead. Food stamps are pretty similar to guaranteed income because neither requires infrastructure or supply chains (like food banks do), but you still have to account for the loss caused by.
1. Beneficiary Paperwork: Paperwork is usually an undignified hassle within conventional welfare programs. Wasting beneficiaries’ time is costly, and hiring a ton of bureaucrats to scrutinize people is also costly.
2. Inefficient Value transfer: Beneficiaries do not get exactly what they need most, so the counterfactual impact would (almost) always be larger if they had cash. I used the example below, which would probably waste ~20% of the value vs the counterfactual.
“Fundamentally, poor people are way better than nonprofit ‘experts’ at knowing what they need and getting it. For example, if someone needs a car loan and the aid program provides food stamps, they have to launder their food stamps to get the cash for their car loan. Guaranteed income empowers poor people with liquid aid.”
Thirdly, most of your links are to your own foundation, and you’re explicitly here to ask for money. Everything is optimized to make your hypothesis look good without a counterargument.
So I primarily linked to my website’s research page because it has all of the links to all of the studies on it. It has a database (not curated by us) of 320+ studies, so I see your point but I’m far more interested in people accessing the research put together in one spot. For homelessness, I linked to the website because you can’t click on the links in the table picture. You can click on the links, and see the source data on the website, and no one else has a list like that. I’m explicitly asking for attention -
And as for your last sentence, yes I’m looking for 3 things.
1. Someone such as yourself to come up with a fundamental problem with any part of our plan (not just homelessness, as that’s just a tiny part with greater risk but also reward), and save me a bunch of wasted time doing something that won’t lead anywhere.
2. You’ve hit the nail on the head, after looking at every single piece of research into “What happens when you provide guaranteed income to homeless individuals,” I can’t find any strong counter-arguments, other than there haven’t been enough studies done yet. And considering the unparalleled research into cash transfers on other populations (global poverty, impoverished people in cities, even formerly incarcerated individuals) I’m really not optimizing the links. I’d be happy to see if you can find pretty much any strong evidence that guaranteed income isn’t simply effective.
3. I did state explicitly that I used a drastic underestimate for all of the impact estimates because of the lack of homelessness RTCs. Luckily there are several on the way, I’m especially looking forward to the one in Denver.
In every experiment to date, guaranteed income has helped at least 66% of participants regain economic stability. We are using an unreasonably low estimate of 50% for two reasons.
1. There have only been three experiments completed as of November 2022; it’s not the biggest data set.
2. Because of the logarithmic rate of the success cost curve, we could have miscalculated the optimal duration and/or the monthly amount needed. If we set the bar too high for ourselves and miss slightly, our pilot could look like a failure even if it’s extremely impactful. If only a third or quarter of participants regained housing stability, our pilot would still be an order of magnitude more cost-effective compared to other homelessness interventions

Michael Simm 21 Nov 2022 3:45 UTC
1 point
1
in reply to: Mitchell_Porter’s comment on: Introducing The Logical Foundation, A Plan to End Poverty With Guaranteed Income
I think at first, definitely not. I see it playing out like this:
1. While other forms of assistance are going about things like normal, someone (us) builds a big enough guaranteed income program to provide half or most of the homeless population in an area with guaranteed income.
2. When that program happens, hopefully, most homeless people attain far better situations within a year, and the existing assistance services find themselves with more resources available to assist fewer people (the ones in highly bad mental states/addiction).
3. Using guaranteed income to help people in danger of falling into homelessness drastically reduces the rates of people falling into homelessness, and leaves the existing service providers with less and less to do over time, twiddling their thumbs.
4. Peace and quiet.
5. Only then, will funding slowly get transitioned to guaranteed income (not all, some 20-30% will probably still be needed for intensive addiction, psychiatric services, and disaster relief). At this point, there will probably need to be major policy changes.
Is that ambitious, yeah, but it all relies on #2. Somebody needs to try it.

Michael Simm 20 Nov 2022 19:43 UTC
2 points
2
in reply to: Shmi’s comment on: Introducing The Logical Foundation, A Plan to End Poverty With Guaranteed Income
So we’ve got two major types of guaranteed income experiments, those on homeless individuals, and those on the general (impoverished) public.
I agree with you that the New Leaf Project experiment (that I cited quite a bit) was quite positive, and also that the problem is still getting worse. Although It’s important to note that that experiment only had 115 participants, so it covered only a drop in the bucket.
The real question is, “If guaranteed income was scaled up to cover the entire population in homelessness or in danger of becoming homeless, would the results of the experiment stay true?”. Would it eradicate (mostly at least) homelessness? This has never been done so far, but considering the evidence, it seems to at least be worth trying. Especially because guaranteed income is way less expensive than all other forms of assistance. We spend ~$80,000 on average servicing each homeless person, giving them even $1,000 a month, which is <1/5 the cost. And it looks so far to be substantially more effective.
I am guessing that there is a subset of the population for whom it will work, verkeepingrsus keep neutral or potentially harm.
The important question, as you accurately point out, is, ” How big is that subset of society”.
Luckily, this is the part where there is infinitely more research and data available, including large-scale experiments to look at. There’s a map with dozens of studies, the results of a large random experiment in Stockton California, and the results from the most comprehensive UBI experiment ever.
If anything, please watch this 15-minute TED Talk. I think you’ll find it fascinating.

Michael Simm 19 Nov 2022 7:34 UTC
1 point
1
in reply to: Shmi’s comment on: Introducing The Logical Foundation, A Plan to End Poverty With Guaranteed Income
to be successful one would need to guarantee UBI to everyone, without means testing, for credibly unlimited duration, linked to a reasonable index to protect against inflation.
I would like to see a reason stated for any of these assertions. We’re not doing UBI, or unlimited durations (at first at least). We also will be forced to do some means testing because of IRS limitations on 501(c)(3)s, but it will be only a 30-minute application. Our initial experiments will not be indexed to inflation because we still need to figure out the optimal amount to help people.
What do you mean by expensive? It seems clear we’re currently spending a lot more money a lot less impactfully. Our Pilot Budget is $182,000.
The US seems like a poor testing ground for it.
The second half of the post describes in detail why Arizona & homelessness is likely the highest-impact opportunity anywhere in the developed world. I’d like to hear the reasoning.

Michael Simm 18 Nov 2022 23:50 UTC
3 points
2
in reply to: Dagon’s comment on: Introducing The Logical Foundation, A Plan to End Poverty With Guaranteed Income
Thanks for the comment! I also do not think cash will be more effective for—every—impoverished person. I do think, however, that most are a lot closer to almost all with the only exceptions being people that are suffering from extreme drug addiction and/or mental illness. Actually, there’s some spotty evidence (we need a good RTC study on this) that cash transfers could actually be a cost-effective way to reduce drug usage. (check out Simon, a case study who was a 20-year homeless heroin addict).
individuals come and go from populations, and there’s a ton of overlap/leakage between a served and an unserved population for any given program
I 100% agree with your comment that we should make sure to look at the individuals as well as the larger data. I may be misunderstanding your point, but I think you might be misunderstanding how guaranteed income programs work. Some are household-based, but I vastly prefer individual-based (it means larger families get more support and individuals are empowered to, for example, escape abusive relationships).
Either way, the individuals or households in the population selected do not leak between served and unserved during the duration of their transfers. The whole point of guaranteed income is its consistency for every person in the program.
very small pilots are often successful not because of the underlying mechanisms, but because of observation and selection effects
This is an argument I see pretty frequently, and I agree that (specifically for homeless people) more research is needed. That said, in the largest GI Randomized Controlled Trial experiment for homeless people, the (115) pilot participants were not carefully overseen. “Participants completed questionnaires at 1 month and then every 3 months. To better understand individual experiences, participants also completed open-ended qualitative interviews after 6 and 12 months.”
Nonetheless, spending on drugs & alcohol went down by 39%, and the pilot was a massive success. It appears like larger, better pilots with fewer observation and selection biases don’t have worse results than the small ones.

Michael Simm

In conclusion