Note: I found out that there have been detailed expert reviews of Founders Pledges work in the past. See Johannes Ackvas comment below. Also, despite the focus on Founders Pledge, this doesn't mean that I think that they are especially under-reviewed. More generally, I don't mean to criticise FP - I just wanted to share this argument and see what people think about it.
Last time I checked, I couldn't find any in-depth, expert reviews of the cost-effectiveness estimates of Founders Pledge. I know that there isn't one for their evaluation of the Clean Air Task Force. GivingGreen and SoGive have looked at some of them, but not in-depth. (They told me this in correspondence). So, assuming I didn't miss anything, there are two possibilities:
(i) they didn't have any such reviews or
(ii) they had reviews, but didn't publish them
Lets first assume (i) is the case.
There seem to be large potential benefits of getting reviewed. If such an expert would find out that FP is significantly off, then this is valuable information, because it might lead investors to change the amount they donate. If FP underestimated e.g. CATFs cost-effectiveness, they might shift funding from less effective opportunities towards the CATF and if they overestimated it's cost-effectiveness the reverse might happen. Either way, if an expert review uncovers that the size of the error is sufficiently large, it is not implausible that this would improve large funding decisions.
If, however, such an expert verifies FPs models, then this is valuable information too. In that case, their research seems much more trustworthy from the outside, which plausibly attracts more investors. This is especially true, for cost-effectiveness estimates that seem spectacular. Take the claim that CATF averts 1 ton of CO2e for less than a dollar. Many people outside of EA that I talked to were initially skeptical of this. (I'm not making any claim as to the reasonableness of that estimate, I am merely commenting on its public perception.)
So it seems like there are large potential benefits of getting an expert review for organisation like FoundersPledge (all I'm saying here might well apply to many other similar organisation that I'm not as familiar with).
The remaining question is then: are the expected benefits of an independent analysis justifying its costs?
I assume that you can hire an expert researcher for less than 100$/hour and that such an analysis would take less than 4 full work weeks. At 40 hours/week the whole thing would cost less than 16.000 $. That seems unrealistically high, but let’s assume it's not.
Estimating that tens to hundreds of millions of dollars are allocated on the basis of their recommendations, it still seems worth it, doesn’t it?
Edit: I have probably overestimated this amount quite a lot. See Linch's comment
Now lets assume that (ii) is the case - they had independent expert reviews, but didn't publish them. In that case, for the exact reasons given above, it would be important to make them public.
What do you make of this argument?
I hope to have time for a longer comment on Monday, but for now some quick clarifications (I work at FP and currently lead the climate work):
1. Linch's comment on FP funding is roughly right, for FP it is more that a lot of FP members do not have liquidity yet (it is a pledge on liquidity events).
2. I spent >40h engaging with the original FP Climate Report as an external expert reviewer (in 2017 and 2018, before I joined FP in 2019). There were also lots of other external experts consulted.
3. There isn't, as of now, an agreed-to-methodology on how to evaluate advocacy charities, you can't hire an expert for this. Orgs like FP, SoGive, OPP, try to evaluate methodologies to do this kind of work, but there is no single template. The approach we chose is, roughly, cluster-based reasoning and integrating lots of different considerations and models. I discuss this in more detail here (second part of the comment).
4. As discussed in that comment, lots of expert-judgments and scientific results are reflected in the recommendation, but the ultimate cost-effectiveness estimates come from combining them. Crucially, something like a CATF recommendation, does not depend on a single cost-effectiveness estimate that we describe as highly uncertain where ever we talk about it, but on (a) different ways to estimate cost-effectiveness, (b) most importantly, crucial considerations about the climate space and solutions (e.g. the main reason we think CATF is very high impact is because they focus on neglected solutions that can have large leverage for global decarbonization, see e.g. discussed here, with our view of what's important / effective / neglected driven by years of research + lots of conversations with experts), (c) have a long successful track record, and (d) continuous investigations over time (> 80h this year alone) investigating CATF's current programs, funding margins, and plans.
5. Skepticism from external people on cost-effectiveness of philanthropy in climate comes, I think, in a large part from the false promises of cheap offsets that lack environmental integrity.
Because, when offsets cost < 1 USD/t they are not credible (and indeed, they aren't) whereas credible offsets are much more expensive (> 100 USD/t). So the fact that you can be much more cost-effective when you are risk-neutral and leverage several impact multipliers (advocacy, policy change, technological change, increased diffusion) is hard to explain and not intuitively plausible.
Hi Johannes!
I appreciate you taking the time.
"Linch's comment on FP funding is roughly right, for FP it is more that a lot of FP members do not have liquidity yet"
I see, my mistake! But is my estimate sufficiently off to overturn my conclusion?
" There were also lots of other external experts consulted."
Great! Do you agree that it would be useful to make this public?
"There isn't, as of now, an agreed-to-methodology on how to evaluate advocacy charities, you can't hire an expert for this."
And the same ist true for evaluating cost-effectiveness analyses of advocacy charities (e.g. yours on CATF)?
"So the fact that you can be much more cost-effective when you are risk-neutral and leverage several impact multipliers (advocacy, policy change, technological change, increased diffusion) is hard to explain and not intuitively plausible."
Sure, thats what I would argue as well. Thats why its important to counter this skepticism by signalling very strongly that your research is trustworthy (e.g. through publishing expert reviews).