why is Tetlock-style judgmental forecasting so popular within EA, but not that popular outside of it?
The replies so far seem to suggest that groups outside of EA (journalists, governments, etc) are doing a smaller quantity of forecasting (broadly defined) than EAs tend to.
This is likely correct but it is also the case that groups outside of EA (journalists, governments, etc) are doing different types of forecasting than EAs tend to. There is less "Tetlock-style judgmental" forecasting and more use of other tools such as horizon scanning, scenario planning, trend mapping, etc, etc.
(E.g. see the UK government Futures Toolkit, although note the UK government also has a more Tetlock-style Cosmic Bazaar)
So it also seems relevant to ask: why does EA focuses very heavily on "Tetlock-style judgmental forecasting", rather than other forecasting techniques, relative to other groups?
I would be curious to hear people's answers to this half of the question too. Will put my best guess below.
– –
My sense is that (relative to other futures tools) EA overrates "Tetlock-style judgmental" forecasting a lot and that the world underrates it a bit.
I think "Tetlock-style" forecasting is the most evidence based, easy to test and measure the value of, futures technique. This appeals to EAs who want everything to be measurable. Although it leads to it being somewhat undervalued by non-EAs who undervalue measurability.
I think the other techniques have been slowly developed over decades to be useful to decision makers. This appeals to decision makers who value being able to make good decisions and having useful tools. Although it leads to them being significantly undervalued by EA folk who tend to have less experience and a "reinvent the wheel" approach to good decision making to the extent that they often don’t even notice that other styles of forecasting and futures work even exist!
On deforestation. Just to be clear the result of our prioritisation exercise was our top recommendations (ideas 1-2) on subscription models for new antibiotics and stopping dangerous dual use research. The ideas 4-7 (including the deforestation one) did well in our early prioritisation but ultimately we did not recommend them. I have made a minor edit to the post to try to make this clearer.
The stopping deforestation report idea was originally focused on limiting the human animal interface to prevent zoonotic pandemics (which did well in our prioritisation). Then in the report we prioritise between the ways one might go about stopping zoonoses. The summary is:
Approach
Scale of issue
Impact of approach
Tractability
Neglected
Avoids risk of increasing pandemic risk
Externalities for other cause areas, e.g., climate, Animal welfare
Overall sense of how promising
Reduce deforestation
High
High
Moderate
Moderate
High
Unclear for animals
Positive for climate
High
Wild animal supply chain and trade regulation
High
Moderate
Low
Moderate
High
Neutral to slightly positive
Moderate
Education and/or regulation or biosecurity on farms
Moderate
Moderate
Low
High
High
Neutral to slightly positive
Moderate
Vaccination of animals
Low
Low
Moderate
High
High
Neutral to slightly positive
Low
Better detection at high-risk human-animal interface
Moderate
Low
Low- Moderate
Low
Moderate
Slightly negative
Low
Unfortunately the full report is not quite ready for publication. Hopefully the full report will be available soon.
We didn’t look into gene synthesis risks so might have missed something there. Although potentially a charity working on reducing dual use research could play a role in limiting these risks.
I would have to check this with Akhil, the lead author, but my understanding is that this CEA compares a case where the PASTEUR act passes with a business as usual case where very few (but not zero) new anitbiotics are developed.
I agree this is probably overly-optimistic as we can and probably should assume that someone is likely to do something about antibiotic resistance in the next few decades. Good spot!
And thank you for the great questions and for looking over things in such detail.
Hi Ben, happy to justify this. I was responsible for the alternate estimate of 10%-17.5%
– –
These numbers here are consistent with our other estimates of policy change. Other estimates were easier (but not easy) to justify as they were in better evidenced areas and tended to range between 5% and 40% (see 2022 ideas here). Areas with the best data were road safety policy, where we looked at looked at 84 case studies finding a 48% chance of policy success, and food fortification policy, where we looked at 62 case studies (in Annex) with a 47% chance of success – we scaled these numbers downwards but they are not low.
Other EA estimates are also in this ballpark – e.g. Nuno's estimates on OpenPhil criminal justice reform are in the same ball park and give 7-50% and 1-10% chance of policy change success. Also personal experience suggests reasonably high numbers too – CE has one long-running policy charity (LEEP) and it seems to have been pretty successful, driving policy change within 6 months. My own experience was also fairly successful. I think this allows us to put relatively high priors on policy change of 5 to 20 percentage points increase.
That said my 7.5 percentage points increase here was guesswork. I wish I had more time to look into it but it was driven by intuitions based of case studies and experience, that is mostly not US based. I would be open to hearing from experienced US policy advocates that it is too high (or too low).
– –
On the articles you link to:
The informational lobbying post estimates that lobbying raises the chance of policy change from "very low" to 2.5%. I think this is consistent with a raise from 10% to 17.5%.
My view on policy (based on working in the field) is that the most impact per $ comes when advocating for a policy issue that might just happen anyway – e.g. advocating for something where the political door is ajar and it just needs a nudge. I think moving the needle from 15%-40% is about as easy as moving the needle from 1%-5%.
I am unfamiliar with the LSE study and will have to have a look at it. Maybe it will lead me to be more pessimistic.
– –
Note 1: I work for CE. Note 2: I think policy change in the animals welfare space is a bit different so assume I am not talking about animal welfare work in any of the above.
(Disclaimer the following is my initial impressions based on 2 minutes of Googling, cannot promise accuracy)
Scale – 400k-1million people are in slavery in the DRC. They lead horrendous lives suffer a myriad of terrible health conditions and are not free. The number is huge, more than die of Malaria each year, more than die of AIDs each year. EAs have looked into US criminal justice but there might be nearly as many slaves in the DRC as there are prisoners in the US and ALL of them are being held unjustly and likely suffer in many more ways than US prisoners.
Tractability – the animal welfare movement has over the last decade, developing a host of evidence based tools that have lead to win after win for animal welfare. In particular we have a playbook for targeted corporate campaigns and have been immensely successful at driving corporates to commit to ethical practices. Most of the products of slavery in the Congo are used by Western companies that could be pressured to change. In many ways this should be even easier than the case for animals as people care more about humans than animals.
Neglectedness – No body seems to be doing this (based on my 1min of Googling). The anti-slavery space seems very very focused on a slavery in HICs (like trafficking to the US or the UK) and not on the Congo. It is talked about but I did not find any targeted campaigns.
30 second BOTEC – Number of years a corporate campaign program would need to run that could end 75% of slavery in the Congo x cost per year / 75% x number of slaves * a best guess DALY burden of life in slavery = ( 10 x $2,000,000 ) / ( 75% x 700,000 x 12.5 ) = $3/DALY
Curious if you have a sense of the geographic scope of these needs / talent gaps?
(Policy development work can be extremely country dependent. The same person could be highly qualified to do this work in the UK and highly underqualified to do it for the US, or India, or China or Finland, etc).
Huge thanks to Austin and the Manifold Markets team for making this collaboration possible. It has been a pleasure to work with you and your support has been invaluable.
Is there info about why grantors didn't give more funding to HH?
I don’t have this info. I think it is possible that funders are not interested in Africa (HH was working in Kenya) or that funders don’t value this kind of work as they see it as incremental welfare improvements that they don’t lead to long run change, but I'm mostly honestly speculating ...
Animal charities. Most suffering in the world happens in farms. See recommendations for where to donate by Animal Charity Evaluators.
To address extreme suffering in humans then consider:
Mental health charities. See this page the Happier Lives Institute (HLI) recommend Strong Minds. Note there has been a lot of debate on the EA Forum about them. You could also donate to HLI or to other mental health charities like Vida Plena* or Friendship Bench, etc.
Pain prevention charities. I rate Organisation for the Prevention of Intense Suffering (OPIS) although I don’t know of any independent evaluations. (There is also Lifting The Burden on headaches and some anti-torture charities like Aegis although I have not looked into them so cannot vouch for effectiveness so would do your research.)
Bednets do also prevent suffering and are very well evidenced so are still an option. See HLI's research on this here.
There are reasons to not be a 100% negative utilitarian. See Toby Ord's essay on this here.
* Indicates where I have been involved in orgs/research. Although honestly I have met folk at most of the orgs listed and so just assume I am bias and do your own research.
The replies so far seem to suggest that groups outside of EA (journalists, governments, etc) are doing a smaller quantity of forecasting (broadly defined) than EAs tend to.
This is likely correct but it is also the case that groups outside of EA (journalists, governments, etc) are doing different types of forecasting than EAs tend to. There is less "Tetlock-style judgmental" forecasting and more use of other tools such as horizon scanning, scenario planning, trend mapping, etc, etc.
(E.g. see the UK government Futures Toolkit, although note the UK government also has a more Tetlock-style Cosmic Bazaar)
So it also seems relevant to ask: why does EA focuses very heavily on "Tetlock-style judgmental forecasting", rather than other forecasting techniques, relative to other groups?
I would be curious to hear people's answers to this half of the question too. Will put my best guess below.
– –
My sense is that (relative to other futures tools) EA overrates "Tetlock-style judgmental" forecasting a lot and that the world underrates it a bit.
I think "Tetlock-style" forecasting is the most evidence based, easy to test and measure the value of, futures technique. This appeals to EAs who want everything to be measurable. Although it leads to it being somewhat undervalued by non-EAs who undervalue measurability.
I think the other techniques have been slowly developed over decades to be useful to decision makers. This appeals to decision makers who value being able to make good decisions and having useful tools. Although it leads to them being significantly undervalued by EA folk who tend to have less experience and a "reinvent the wheel" approach to good decision making to the extent that they often don’t even notice that other styles of forecasting and futures work even exist!
Hi John.
Thank you for the feedback and comments.
On deforestation. Just to be clear the result of our prioritisation exercise was our top recommendations (ideas 1-2) on subscription models for new antibiotics and stopping dangerous dual use research. The ideas 4-7 (including the deforestation one) did well in our early prioritisation but ultimately we did not recommend them. I have made a minor edit to the post to try to make this clearer.
The stopping deforestation report idea was originally focused on limiting the human animal interface to prevent zoonotic pandemics (which did well in our prioritisation). Then in the report we prioritise between the ways one might go about stopping zoonoses. The summary is:
Unclear for animals
Positive for climate
High
Unfortunately the full report is not quite ready for publication. Hopefully the full report will be available soon.
Or $ per tCO2 were from two sources:
Sorry that we missed your estimate.
We didn’t look into gene synthesis risks so might have missed something there. Although potentially a charity working on reducing dual use research could play a role in limiting these risks.
I would have to check this with Akhil, the lead author, but my understanding is that this CEA compares a case where the PASTEUR act passes with a business as usual case where very few (but not zero) new anitbiotics are developed.
I agree this is probably overly-optimistic as we can and probably should assume that someone is likely to do something about antibiotic resistance in the next few decades. Good spot!
And thank you for the great questions and for looking over things in such detail.
Hi Ben, happy to justify this. I was responsible for the alternate estimate of 10%-17.5%
– –
These numbers here are consistent with our other estimates of policy change. Other estimates were easier (but not easy) to justify as they were in better evidenced areas and tended to range between 5% and 40% (see 2022 ideas here). Areas with the best data were road safety policy, where we looked at looked at 84 case studies finding a 48% chance of policy success, and food fortification policy, where we looked at 62 case studies (in Annex) with a 47% chance of success – we scaled these numbers downwards but they are not low.
Other EA estimates are also in this ballpark – e.g. Nuno's estimates on OpenPhil criminal justice reform are in the same ball park and give 7-50% and 1-10% chance of policy change success. Also personal experience suggests reasonably high numbers too – CE has one long-running policy charity (LEEP) and it seems to have been pretty successful, driving policy change within 6 months. My own experience was also fairly successful. I think this allows us to put relatively high priors on policy change of 5 to 20 percentage points increase.
That said my 7.5 percentage points increase here was guesswork. I wish I had more time to look into it but it was driven by intuitions based of case studies and experience, that is mostly not US based. I would be open to hearing from experienced US policy advocates that it is too high (or too low).
– –
On the articles you link to:
The informational lobbying post estimates that lobbying raises the chance of policy change from "very low" to 2.5%. I think this is consistent with a raise from 10% to 17.5%.
I am unfamiliar with the LSE study and will have to have a look at it. Maybe it will lead me to be more pessimistic.
– –
Note 1: I work for CE. Note 2: I think policy change in the animals welfare space is a bit different so assume I am not talking about animal welfare work in any of the above.
My entry:
Modern slavery
(Disclaimer the following is my initial impressions based on 2 minutes of Googling, cannot promise accuracy)
Scale – 400k-1million people are in slavery in the DRC. They lead horrendous lives suffer a myriad of terrible health conditions and are not free. The number is huge, more than die of Malaria each year, more than die of AIDs each year. EAs have looked into US criminal justice but there might be nearly as many slaves in the DRC as there are prisoners in the US and ALL of them are being held unjustly and likely suffer in many more ways than US prisoners.
Tractability – the animal welfare movement has over the last decade, developing a host of evidence based tools that have lead to win after win for animal welfare. In particular we have a playbook for targeted corporate campaigns and have been immensely successful at driving corporates to commit to ethical practices. Most of the products of slavery in the Congo are used by Western companies that could be pressured to change. In many ways this should be even easier than the case for animals as people care more about humans than animals.
Neglectedness – No body seems to be doing this (based on my 1min of Googling). The anti-slavery space seems very very focused on a slavery in HICs (like trafficking to the US or the UK) and not on the Congo. It is talked about but I did not find any targeted campaigns.
30 second BOTEC – Number of years a corporate campaign program would need to run that could end 75% of slavery in the Congo x cost per year / 75% x number of slaves * a best guess DALY burden of life in slavery = ( 10 x $2,000,000 ) / ( 75% x 700,000 x 12.5 ) = $3/DALY
Curious if you have a sense of the geographic scope of these needs / talent gaps?
(Policy development work can be extremely country dependent. The same person could be highly qualified to do this work in the UK and highly underqualified to do it for the US, or India, or China or Finland, etc).
Thanks :-)
Thank you for the views and feedback!
It is an honour to work with you too!!
Huge thanks to Austin and the Manifold Markets team for making this collaboration possible. It has been a pleasure to work with you and your support has been invaluable.
[Added a thanks to the post]
I don’t have this info. I think it is possible that funders are not interested in Africa (HH was working in Kenya) or that funders don’t value this kind of work as they see it as incremental welfare improvements that they don’t lead to long run change, but I'm mostly honestly speculating ...
Animal charities. Most suffering in the world happens in farms. See recommendations for where to donate by Animal Charity Evaluators.
To address extreme suffering in humans then consider:
Some caveats:
Bednets do also prevent suffering and are very well evidenced so are still an option. See HLI's research on this here.
There are reasons to not be a 100% negative utilitarian. See Toby Ord's essay on this here.
* Indicates where I have been involved in orgs/research. Although honestly I have met folk at most of the orgs listed and so just assume I am bias and do your own research.