The JPAL and IPA Dataverses have data from 200+ RCTs from development economics and the 3ie portal has 500+ studies with datasets available (and you can further filter by study type if you want to limit to RCTs). I can't point you to particular studies that having missing or mismeasured covariates, but from personal experience, a lot of them have lots of missing data.
Can you explain more why the bootstrapping approach doesn't give a causal effect (or something pretty close to one) here? The aggregate approach is clearly confounded since questions with more answers are likely easier. But once you condition on the question and directly control the number of forecasters via bootstrapping different sample sizes, it doesn't seem like there are any potential unobserved confounders remaining (other than the time issue Nikos mentioned). I don't see what a natural experiment or RCT would provide above the bootstrapping approach.
Side note: a Cohen's d of .31 is not small. My opinion is that the rules of thumb used to interpret effect sizes in psychology are messed up, because so much p-hacking in the past produced way overinflated effect sizes. Regardless, 0.3 is typically seen as a moderate effect size. A 0.3 standard deviation increase in IQ would be 4.5 points which would lead to economically meaningful differences in income.
Within 3 days of departing the UK to return to the US, take another COVID test. This is required by the US CDC according to this link, and both PCR and Rapid Antigen tests are acceptable. I am planning to walk into an NHS location near the EA conference venue (like this) and get a free test. You don’t have to be a UK citizen to get free tests from the NHS (link).
My understanding is that you should not be using the free NHS test for travel and should instead book a private test, which is possible across London and at airports on the day of your flight. See the travelling abroad section of this NHS page. More practically, I think you only get a text message confirming your result from the NHS tests and this is not sufficient documentation for the CDC requirements.
What information must be included on the test result?
A test result must be in the form of written documentation (paper or electronic copy). The documentation must include:
1 . Type of test (indicating it is a NAAT or antigen test)
2. Entity issuing the result (e.g. laboratory, healthcare entity, or telehealth service)
3. Specimen collection date. A negative test result must show the specimen was collected within the 3 days before the flight. A positive test result for documentation of recovery from COVID-19 must show the specimen was collected within the 3 months before the flight.
4. Information that identifies the person (full name plus at least one other identifier such as date of birth or passport number)
5. Test Result
Our funder was interested in How Asia Works, presumably from positive reviews it's received from people like Bill Gates and Noah Smith, and asked us to check the land section in more detail. We had a comparative advantage here given my background in development economics.
I wouldn't be particularly interested in more land redistribution research, given that there don't seem to be any clear funding opportunities in this space. If someone could find decent opportunities then that would make it a bit more interesting. But given the ambiguous results on the relationship between farm size and yield, I imagine research on other unexplored development interventions would have higher value of information.
I would be interested to read a deep dive into tenure reform, but this is just my personal opinion. A bunch more work, both policy and academic, seems to have been done on tenure reform so there would probably be more literature and case studies to work with. We link a couple of systematic reviews (Gignoux et al. 2014 and Lawry et al. 2017) but didn't look into them ourselves.
If a whole book is too much, you could also try their article, Economic Lives of the Poor - https://www.aeaweb.org/articles?id=10.1257/jep.21.1.141 - but this is explicitly focused on people living below the extreme poverty line, who are an order of magnitude poorer than the global median.
For example, David and Jason's report on charter cities was completed in 100 hours, a reasonable fraction of which was extra legwork for external writeup/following up with affected parties, after the original report was delivered to Open Phil. My impression is that the bulk of the work was done on a fairly short calendar time cycle too, in ways that may be hard for external parties to replicate. But naively the report would still be useful to Open Phil and cost-effective to fund if it took 200 hours to complete and 3x the calendar time.
Just to clarify, the 100 hours was actually just for the original report and doesn't include any of the extra leg work for the public version, because I forgot to update that time taken estimate in the public version. The extra work for the public version was an additional 10-15 hours of work from the two of us, but there was also work from others reviewing the report. This extra work took place over 5 weeks of calendar time.
I know you were explicit about these being your views and not Founders Pledge's, but is there anyone better placed to think through those implications than Founders Pledge? And similarly, it seems like Founders Pledge would be one of the most natural organisations to advocate against limits on patient philanthropy, given the work on the long-term investment fund.
I'm not convinced that our CEA is particularly useful for more generalised interventions. All we really do is assume that the intervention causes some growth increase (a distribution rather than a point estimate) and then model expected income with the intervention, with the intervention 10 years later and with no intervention. The amount the intervention increases growth is the key parameter and is very uncertain so further research on this will have the highest VoI, but this will be different for each intervention. We treat how the intervention increases growth as a black box so I think looking inside the box and trying to understand the mechanisms better would shed some light on how robust the assumed growth increase is and how we might expect it to generalise to other contexts.
Furthermore, we only model the direct benefits of the growth intervention. In general, I'd expect the indirect effects to be larger and our modelling approach doesn't say anything about these so I expect looking into these indirect benefits, perhaps via an alternative model, to have higher VoI than further modelling of the direct benefits.
For charter cities in particular, we could probably further tighten the bounds on the direct benefits by getting more rigorous information on city population growth rates and the correlation between population growth and income growth.
Thanks Mark, both for your time and feedback while we were writing the report and your comments now.
On 1, I agree that charter cities sit somewhere between neartermist and longtermist so thinking about them as mid/mediumtermist makes sense. I imagine Rethink Priorities’ future work in this space will be a mixture of traditionally neartermist and mediumtermist topics. However, most of the current arguments for charter cities, especially Mason (2019), have an explicitly neartermist flavour, given the direct comparisons to GiveWell charities and a focus on the direct benefits. I'm keen to see robust medium/longtermist arguments for charter cities being made more explicitly.
On 2 & 3, there's some tension between the claims that (1) Chinese growth is a result of SEZs, (2) the charter cities movement is trying to replicate the success of China, and (3) that SEZs are not the right comparison for charter cities.
To simplify the argument somewhat, we are taking the position that the more useful currently existing empirical analogue for charter cities is all SEZs, whereas your position is that it is Shenzhen. I totally accept your points about the important differences between SEZs and charter cities, however I am still concerned that focusing solely on the Shenzhen SEZ is cherry picking and an unrepresentative sample of how we might expect charter cities to perform. I think the ideal empirical analogue would be the subset of all SEZs that were large, had relatively high autonomy and multiple industries, however we couldn't find any analysis of the performance of this subset.
On 4, I think the report is clear about why we are currently skeptical of the tractability of charter cities despite recent history (although I recognise that you have inside knowledge that might cause us to update more positively). I’d also highlight that regardless of what you think of the absolute tractability of charter cities, it seems intuitive that the relative tractability is lower than alternatives such as special reform zones, which aim at delivering the same benefits as charter cities without having to set up and build a brand new city. That said, I'm happy you and CCI are still working on this and I would love for you to prove us wrong!
The JPAL and IPA Dataverses have data from 200+ RCTs from development economics and the 3ie portal has 500+ studies with datasets available (and you can further filter by study type if you want to limit to RCTs). I can't point you to particular studies that having missing or mismeasured covariates, but from personal experience, a lot of them have lots of missing data.
Can you explain more why the bootstrapping approach doesn't give a causal effect (or something pretty close to one) here? The aggregate approach is clearly confounded since questions with more answers are likely easier. But once you condition on the question and directly control the number of forecasters via bootstrapping different sample sizes, it doesn't seem like there are any potential unobserved confounders remaining (other than the time issue Nikos mentioned). I don't see what a natural experiment or RCT would provide above the bootstrapping approach.
Side note: a Cohen's d of .31 is not small. My opinion is that the rules of thumb used to interpret effect sizes in psychology are messed up, because so much p-hacking in the past produced way overinflated effect sizes. Regardless, 0.3 is typically seen as a moderate effect size. A 0.3 standard deviation increase in IQ would be 4.5 points which would lead to economically meaningful differences in income.
My understanding is that you should not be using the free NHS test for travel and should instead book a private test, which is possible across London and at airports on the day of your flight. See the travelling abroad section of this NHS page. More practically, I think you only get a text message confirming your result from the NHS tests and this is not sufficient documentation for the CDC requirements.
Hi Edo!
Our funder was interested in How Asia Works, presumably from positive reviews it's received from people like Bill Gates and Noah Smith, and asked us to check the land section in more detail. We had a comparative advantage here given my background in development economics.
I wouldn't be particularly interested in more land redistribution research, given that there don't seem to be any clear funding opportunities in this space. If someone could find decent opportunities then that would make it a bit more interesting. But given the ambiguous results on the relationship between farm size and yield, I imagine research on other unexplored development interventions would have higher value of information.
I would be interested to read a deep dive into tenure reform, but this is just my personal opinion. A bunch more work, both policy and academic, seems to have been done on tenure reform so there would probably be more literature and case studies to work with. We link a couple of systematic reviews (Gignoux et al. 2014 and Lawry et al. 2017) but didn't look into them ourselves.
If a whole book is too much, you could also try their article, Economic Lives of the Poor - https://www.aeaweb.org/articles?id=10.1257/jep.21.1.141 - but this is explicitly focused on people living below the extreme poverty line, who are an order of magnitude poorer than the global median.
Just to clarify, the 100 hours was actually just for the original report and doesn't include any of the extra leg work for the public version, because I forgot to update that time taken estimate in the public version. The extra work for the public version was an additional 10-15 hours of work from the two of us, but there was also work from others reviewing the report. This extra work took place over 5 weeks of calendar time.
I know you were explicit about these being your views and not Founders Pledge's, but is there anyone better placed to think through those implications than Founders Pledge? And similarly, it seems like Founders Pledge would be one of the most natural organisations to advocate against limits on patient philanthropy, given the work on the long-term investment fund.
I'm not convinced that our CEA is particularly useful for more generalised interventions. All we really do is assume that the intervention causes some growth increase (a distribution rather than a point estimate) and then model expected income with the intervention, with the intervention 10 years later and with no intervention. The amount the intervention increases growth is the key parameter and is very uncertain so further research on this will have the highest VoI, but this will be different for each intervention. We treat how the intervention increases growth as a black box so I think looking inside the box and trying to understand the mechanisms better would shed some light on how robust the assumed growth increase is and how we might expect it to generalise to other contexts.
Furthermore, we only model the direct benefits of the growth intervention. In general, I'd expect the indirect effects to be larger and our modelling approach doesn't say anything about these so I expect looking into these indirect benefits, perhaps via an alternative model, to have higher VoI than further modelling of the direct benefits.
For charter cities in particular, we could probably further tighten the bounds on the direct benefits by getting more rigorous information on city population growth rates and the correlation between population growth and income growth.
Thanks Mark, both for your time and feedback while we were writing the report and your comments now.
On 1, I agree that charter cities sit somewhere between neartermist and longtermist so thinking about them as mid/mediumtermist makes sense. I imagine Rethink Priorities’ future work in this space will be a mixture of traditionally neartermist and mediumtermist topics. However, most of the current arguments for charter cities, especially Mason (2019), have an explicitly neartermist flavour, given the direct comparisons to GiveWell charities and a focus on the direct benefits. I'm keen to see robust medium/longtermist arguments for charter cities being made more explicitly.
On 2 & 3, there's some tension between the claims that (1) Chinese growth is a result of SEZs, (2) the charter cities movement is trying to replicate the success of China, and (3) that SEZs are not the right comparison for charter cities.
To simplify the argument somewhat, we are taking the position that the more useful currently existing empirical analogue for charter cities is all SEZs, whereas your position is that it is Shenzhen. I totally accept your points about the important differences between SEZs and charter cities, however I am still concerned that focusing solely on the Shenzhen SEZ is cherry picking and an unrepresentative sample of how we might expect charter cities to perform. I think the ideal empirical analogue would be the subset of all SEZs that were large, had relatively high autonomy and multiple industries, however we couldn't find any analysis of the performance of this subset.
On 4, I think the report is clear about why we are currently skeptical of the tractability of charter cities despite recent history (although I recognise that you have inside knowledge that might cause us to update more positively). I’d also highlight that regardless of what you think of the absolute tractability of charter cities, it seems intuitive that the relative tractability is lower than alternatives such as special reform zones, which aim at delivering the same benefits as charter cities without having to set up and build a brand new city. That said, I'm happy you and CCI are still working on this and I would love for you to prove us wrong!