I think "red teaming" GiveWell (or other people aiming to do good) is a great idea and I encourage more people to try it. However, I think this critique aims a bit too broadly and shallowly and misses a lot of the detail. Generally speaking, GiveWell has put a lot of time and effort into their thoughts and are generally unlikely to be overturned by a few hours of thought.

I'm not here to "defend" GiveWell. I think GiveWell is a very sold, thoughtful organization, though they are capable of making mistakes (indeed, they openly list some). However, I think your analysis of GiveWell is woefully incomplete and would benefit from a lot more depth.

I've left individual comments here about what I think you've gotten wrong.

"Givewell’s defends their use of DALYs by citing that they are commonly used."

The post you link to is from 2008 - 9 years ago. GiveWell subsequently became somewhat disillusioned with DALYs (see http://www.givewell.org/research/DALY) and has developed their own models of the ultimate effects on human welfare of a variety of different benefit flows from AMF, deworming and cash transfers. For example, you can see their cost-effective analysis for 2017 here (http://www.givewell.org/how-we-work/our-criteria/cost-effectiveness/cost-effectiveness-models) which I think is closer to what you're looking for.

Even on the page you link, Holden says of DALYs: "It is also true that we personally find the metric to be unhelpful."

"They erroneously (or hyperbolically) identify that they are identical to QALYs."

I can't see anyone from GiveWell saying this on the pages you linked (though perhaps they do somewhere else). You may be referring to a comment from 'Ron' who mentions "the essentially identical metric of QALYs" - but Ron is a random commenter who is not associated with GiveWell.

I suspect a post critiquing GiveWell will have to cover a smaller range of issues, with substantial care applied to each one, in order to help them do better.

“GiveDirectly households reported a ... increase … on an index measuring psychological well-being. This improvement was largely driven by increases in happiness and life satisfaction, and reductions in stress and depression.”

However, "increases in neighbours' wealth strongly decrease life satisfaction and moderately decrease consumption and asset holdings''. Giving What We can also describe how neighbors of cash transfers recipients are less happy after their neighbors enrichment. This should come as no surprise: Participants in many foreign aid programs are better off but at the cost of those who aren't involved in the program.

The evidence for this is minor, but it has been enough that GiveDirectly now gives on the village and not the individual level. So this should no longer be a large concern. This is also an area of active RCT study and GiveDirectly also experiments with different targeting techniques.

Donations undermine the accountability and feedback inherent in free enterprise. Further, freely distributed bed nets could undercut sustainable free market solutions.

GiveWell has written about this too, stating that "[e]vidence implies that charging a fee has significantly reduced demand for the product, without leading to corresponding increases in utilization rates".

One might think that this misses the point, and the benefits of the bednets are outweighed by the greater cost to the African economy. This is a tricky philosophical point and likely depends on your values. GiveWell does push back against the "savior" mentality and views health interventions as less likely to be parternalistic. I would personally think health interventions are also likely to be underserved by the free market and thus would not be disruptive, just as public hospitals usually are not taken to displace "free market" hospitals.

While a good point in general, there are important differences in dumping clothes in Africa versus giving out bednets and deworming treatments. GiveWell has pointed out that aid critics like Easterly tend not to be critical of the interventions that GiveWell supports. GiveWell even had a conversation with Easterly where they point out areas of agreement and disagreement.

-

Charity can be ineffective or potentially harmful without sufficient evidence to the contrary. Do GiveWell’s methods generate sufficient evidence to the contrary?

Yes, I think so. Why would you disagree?

However, GiveWell's AMA calculations do not include or infer psychological well-being.

This is also a good point. The Oxford Prioritization Project recently looked at StrongMinds, an organization focused on treating depression in the developing world, and found the cost-effectiveness lower than GiveWell top charities, but this does not mean that other organizations might be more effective or that the cost-effectiveness of current orgs might change once psychological effects are better taken into account (e.g., grieving parents over a dead child). I'd like to see more research into this, but I'm not sure how it would affect actual decisions about top charities.

Note that some GiveWell staff gave to StrongMinds in 2016.

-

What of wild animal well-being?

This is also a good point, though admittedly outside of the self-defined scope of GiveWell, which (as far as I know) aims to only consider costs and benefits to humans. This is speciesist, so I hope other organizations will aim to address this issue.

Quick question: The paper on the PAHO-HANLON approach doesn't include the term equity. What does it mean in this context, and where did you find an expansion that might be helpful for me as well? Positioning is very relevant to something in writing at the moment, and so might be this equity thing.

I love having so many directions of improvement summarized in one post. In some cases my critiques would soon in the opposite direction. Not too little evidence for our against something but too great risk-aversion, perhaps as a concession to donors who are biased by considerations from the area of private investments, which don't carry over to commons.

Sorry, very compressed because I'm typing on my phone.

While I applaud the idea of playing devil's advocate, I find the style of this post to be quite snide (eg liberal use of sarcastic rhetorical questions), which I think is problematic. Efforts to red team the community should be aimed at pointing out errors to be fixed, and I don't see how this helps. On the contrary, it can decrease morale and also signal to outsiders a lack of a sense of community within EA. It would be no more difficult to bring up potential problems in a simple, matter of factual manner.

Someone once asked why Bill Gates doesn’t close the funding gap for GiveWell recommended charities? The most demonstrable cost-effective interventions are going to funded by the international community anyway, even if we do nothing.

This is likely true, but even if the international community takes the most cost-effective interventions, there still are demonstrably cost-effective interventions that don't get funded by the international community anyway. For example, AMF is very cost-effective and it still has a funding gap.

Did they reinvent the wheel? Their approach to cause prioritisation is remarkably similar to the PAHO-HANLON method.

It's a bit hard to track down how the INT (importance, neglectedness, tractability) method started... it's first mentioned by GiveWell in May 2012 and given the INT names by 80,000 Hours in December 2013. Because GiveWell and OpenPhil don't elaborate on how they came to the INT criteria, it's hard to know who or what they were inspired by (if anything) or what wheels they reinvented.

To avoid personally missing the forest for the trees here, I would like to say that I agree there is a lot to be learned from existing prioritization work done by world governments and pan-governmental bodies, and I'd really like to see someone effectively summarize these, draw comparisons and contrasts with the INT framework, and make conclusions.

-

The PAHO-HANLON approach incorporates two additional elements: positioning and equity, the former being a common topic of discussion in the community. The latter being a topic of criticism from other communities. By adopting the World Health Organisation’s framework, could the associated problems have been dulled?

Which problems are you referring to?

-

To illustrate the relevance of ‘positioning’, choosing a controversial cause may be tractable, but undermine the movement in the long run because one side of politics, business, the general public or the philanthropic community may be alienated or disdain the Effective Altruism community. For instance, there is a current of disdain for effective altruism (or Singer specifically) from the disability rights community which may have been averted with more careful marketing.

This is a good point, though I disagree. I think there is an important separation between a framework for evaluating causes and a framework for making sure your chosen interventions and marketing methods don't alienate people. I'm skeptical combining the two frameworks would make for a better approach than considering the two separately.

-

One opportunity I foresee arising from our short-sightedness of positioning is the fact that the general public thinks self-inflicted ill-health (e.g. cigarette smoking? attempted suicide?) and higher socioeconomic status (if they choose to sabotage their health, do they prefer ill-health?) deserve less help. Could we maximise the impact of our donation recommendation by factoring in the variables that drive the general population of donors.

Do you mean we should follow along with donor interests and avoid focusing on self-inflicted ill-health in order to maybe get more donations?

Or do you mean we should go against donor interests because it means the causes are more likely to be neglected?

GiveWell's initially selected causes by looking through a book on philanthropy's greatest hits. Selection bias anyone?

This claim is highly misleading.

First, it's not correct that the selected causes came solely from considering the history of philanthropy. OpenPhil says they have "cast a wide net for potential focus areas, incorporating information on the history of philanthropy as well as what today’s foundations focus on" and their process implies a lot more than consulting just the history of philanthropy.

Second, it's not clear this is selection bias. GiveWell had selected causes prior to studying the history of philanthropy and used the history of philanthropy to -- correctly, in my opinion -- inform itself of what philanthropy can accomplish. It makes sense to consider past track records when deciding what it is that is best to do. This also involves learning from some failures.

Also, OpenPhil mentions the drawback: "At the same time, I thought the Casebook’s history had important shortcomings - in particular, not putting successes in context along with failures - and I see a lot of room for improvement in the amount of information available about the history of philanthropic successes and failures."

GiveWell spinoff, the Open Philanthropy Project states it is uncertain about dealing with the Disability Paradox. They too use DALY’s rather than quality of life metrics. Givewell’s defends their use of DALYs by citing that they are commonly used. Status quo bias anyone?

GiveWell uses DALYs for some measures so they can be compared to other measures, but they have critiqued DALYs a lot (see here, here, here, here, and here), and don't exclusively rely on it. Indeed, they seem to more prominently publish information about "cost per life saved".

-

They erroneously (or hyperbolically) identify that they are identical to QALYs. I do not believe that evidence for disability adjusted life years (daly’s) is an adequate proxy for quality of life due to the disability paradox: that disability does not necessarily equate to suffering.

As Rob mentioned, I don't see anywhere GiveWell mentioning QALYs and DALYs are identical, but in practice I do personally (not speaking for GiveWell) think they are hard to tell apart.

The disability paradox is an aspect worth bringing up, though I'd want to hear more about how it would actually change decisions about current top charities. OpenPhil has noted it as an issue but neither OpenPhil nor GiveWell have talked about it as far as I can tell.

Schistosomiasis Control Initiative (SCI) and Evidence Action are deworming programs. GiveWell states there is no evidence deworming improves quality of life. GiveWell treats this absence of evidence as non-evidence of absence (of improving the lives of those in dewormed communities).

But what GiveWell says is:

"We find it highly plausible that deworming has subtle but significant developmental effects that improve quality of later life. The studies we know of on this topic each have substantial limitations, and we do not consider them conclusive evidence for the presence (much less for the size) of these effects even in conditions similar to those in the trials. We do consider them suggestive evidence, enough to take the possibility of developmental effects seriously."

They also say:

"Most GiveWell staff members would agree that deworming programs are more likely than not to have very little or no impact, but there is some possibility that they have a very large impact. (Our cost-effectiveness model implies that most staff members believe there is at most a 1-2% chance that deworming programs conducted today have similar impacts to those directly implied by the randomized controlled trials on which we rely most heavily, which differed from modern-day deworming programs in a number of important ways.)"

These posts articulate a very nuanced view -- not to mention that GiveWell also produced one of the most thorough reviews of deworming evidence currently published. Moreover, the view articulated here is inconsistent with your assessment.

-

Deworming is controversial. GiveWell admits deworming may have no impact. In academia, a crude division formed between epidemiologists and economists about whether mass deworming makes sense. This debate was called the 'worm wars' and was visible on Twitter with this and that hashtag. #Wormwars is over. Guess who ‘won’? GiveWell was founded by economists...

Likewise, GiveWell is very aware of the "worm wars" and have written about it.

Also, technically, GiveWell was not founded by economists. Elie has a religion degree and no other schooling. Holden also has no advanced degrees beyond his undergraduate degree from Harvard.