The correspondence between theoretical and practical efficiency is definitely not perfect. Theoretical efficiency guarantees that individuals are properly incentivized. Practical efficiency may not follow because of things like computational costs, and the extent to which this will be a problem will depend on the specific mechanism and the situation in question. For example, in the computational cost case, the actions of large companies would probably be closer to optimal behavior than individual actions.
My hunch would be that proving theoretical efficiency is generally a relatively good proxy for practical efficiency in most cases, but these other practical considerations should be considered in addition to it, as further constraints that one is trying to satisfy. But this is an empirical question, and I'm also relatively uncertain here.
I guess I feel like I'm less compelled by analyses of optimal performance and more like, what system will work in practice.
My impression is that these theoretical properties are the main reason why people are excited about QF. For example, you would prefer it over 1:1 donation matching because it is a more "principled" matching rule, which should lead to an allocation that's closer to the efficient level than 1:1 donation matching. So if not for these properties, I don't see why people should expect this mechanism to work particularly well in practice.
More generally, I agree that full on efficiency shouldn't be thought of as a strictly necessary condition mechanisms to be useful in practice. For example, majority voting works relatively well in practice, despite not being efficient. But efficiency is nevertheless still a central concept, as it is the very motivation behind the public goods provision property (it's basically the only problem with providing public goods privately). The framing I would use here is that increasing efficiency, while satisfying other (context-dependent) considerations, should be considered a key goal of public good provision mechanisms.
As this post explains, the main study that people cite when saying that "superforecasters are better than experts" comes from a competition where the aggregation methods for the two groups was different (Good Judgment Project's aggregation algorithm versus prediction market with low liquidity for amateur forecasters and experts, respectively). Prediction markets for forecasters and experts had similar performance.
Thanks for the comment! On the point of making this information more well-known, is there an easy way to do so, given that I have very little familiarity with these communities?
Showing endogenous CQF is (in)efficient under complete information sounds relatively easy, right? I would love it if someone did this or explained why my intuition about hardness is wrong!
I haven't tried it, and it could turn out to be quite easy, but I think it's probably not so trivial to prove the result either way.
Thanks for these thoughts! I agree with most of what you said. Some replies to specific points:
1b: The post I mentioned discusses this point. I think it's plausible that that's a factor, but even if it were a major one, it still doesn't explain the lack of demand for forecasting consultancies, which could presumably do an even better job at forecasting questions which don't require company-specific information.
2: This matches my intuitions as well. Though I think it doesn't say much about whether forecasting is actually useful or not, as this could mean "EAs are more keen to pay the initial (large) fixed cost to learn how to use and integrate this tool", or "EAs use this tool because they like it, even though it isn't actually that helpful for decisionmaking"
3: I agree with your point if we're talking about forecasting in general! I think that all of the actors I mention make extensive use of forecasting. However, in this question, I tried to restrict my attention to Tetlock-style judgmental forecasting, as I mentioned in the first paragraph (this was my bad, I should have been clearer when specifying the question). The fact that these agents do use various forms of forecasting makes the question more intriguing for me, given that judgmental forecasting is very general and seemingly really promising.
Thanks for the comment, John! I agree with your point about preference aggregation as a main drawback, and I wish that EAs would appreciate this point more. The reason why I chose not to make it a drawback is because this criticism applies to most of the public goods provision literature, as opposed to applying specifically in the case of QF. But hopefully my points in the discussion about potential applications and your comment will bring more attention to this issue.
Thanks a lot for sharing your experience, Austin! I've added a link to your comment in the post. I'm not surprised that it didn't do great for getting more donations to the charities (as the post suggests), but I'm intrigued by your impression that it didn't do well in allocating money to different charities. What was your expectation regarding how allocations would be made, and how were they actually made instead?
And it's really interesting to know that Gitcoin is also de-emphasizing quadratic funding. Their website still mentions quadratic funding quite a lot; do you know if they have written this down somewhere?
As for fancy funding methods, I agree that the s-process looks interesting on the face of it. But I don't think my opinion here is more valuable than anyone else's, and I don't know how it compares to other mechanisms in this space. It would be great if someone thought through the theoretical considerations in that case, and try to get a sense for how participants/funders feel about it (like this testimony). This feels relevant given how much money the SFF has moved to date.
When I wrote the post, the framing I used was focused on the differences in direct impact of focusing on local causes. This meant not mentioning a number of other important considerations, both positive and negative. Since writing the post, I've had the opportunity to talk more to others, and to reflect some more on what these other considerations are. Below, I highlight some that I consider particularly important (see Sjir's comment for some others).
Getting people involved with effective interventions for local problems can serve as a way to get more people involved with effective global interventions. This can happen because it will be easier for some people to get involved with a local problem first before getting involved with a global problem, and also because working on local-level interventions can increase the popularity of EA in the region. I've recently learned that GiveDirectly's US cash transfer program appears to have led to a really large increase to their international donations to people in extreme poverty, due to these two channels. They claim that this was the case, and a quick look at their funding from previous years shows no signs of such a large increase in (international) donations in 2020. If this increase in international donations was indeed not caused by something else, then this example makes me a lot more optimistic about this indirect benefit of doing LPR. Relevantly, I should note that GiveDirectly (1) first consolidated their international transfers program before starting their US transfer program, and (2) tried to emphasize international transfers following the media attention they obtained for US transfers.
A second point in favor of LPR is that, in practice, it doesn't need to be the case that resources directed to local causes come at the expense of resources for the global cause. If an EA group already has a well-developed core group, and can already do outreach to those interested in the global cause, then the group might be able to start dedicating resources to LPR at a relatively low cost. As long as this doesn't compromise the group's ability to do outreach for global issues properly, I think that LPR might become a valuable activity for EA groups to engage in.
It's also worth highlighting some downsides:
Value drift: if research and outreach of local problems comes at the expense of global ones, then this could be a reason for the group to lose most of their impact. And, as the number of people working on local priorities increases, this could attract even more people with a local focus, in a gradual drifting process that might result in a group that's much more focused on local problems than the ideal.
Research costs: high-quality research requires a lot of time, as well as specific research skills. There's already a lot of research available on the most relevant global issues, but for the local problems the group's countries would have to mostly do this themselves. The opportunity cost is doing direct work, doing context-specific GPR[1] that's not LPR, or doing outreach for global problems, which is quite a high bar.
Low-quality research has a really large cost in expectation. If the recommended charities aren't among the best local opportunities, then this further increases the gap in effectiveness of local vs global interventions.[2] Furthermore, it opens up space for large damage to the group's reputation, such as negative media coverage.
These considerations highlight a timing aspect of the overlap between LPR and context-specific GPR. They suggest that LPR is generally better suited for more mature EA groups, which have already consolidated their outreach structure for people willing to work on global problems. Depending on how high the benefits of LPR turn out to be, this could mean that at some point we would want to do LPR even in the US. But there's still a lot of uncertainty, and I think that more information on the benefits and costs of LPR can be quite valuable. I would feel excited about attempts to get a sense for the magnitude of some effects mentioned here, such as by evaluating the GiveDirectly US transfers case more carefully, and looking for other related cases. And I'm excited about EA doing some work on LPR for the information value, particularly in the countries where it's most likely to be the best local opportunity, such as India.
Given a heavy-tailed distribution of impact at the local level, redirecting donations from the average charity to a better-than-median but not great charity is also likely to have negative impact in counterfactual terms. I won't explain why here, but you can read about properties of heavy-tailed impact in this paper.
First off, there is a point worth clarifying here. Scope insensitivity makes it impossible to have feelings that adequately scale with the number of beings affected, and I don't think that there is much that can be done here (sidenote: this Wait But Why post and its sequel are the best ways I'm aware of to try to get an intuitive sense for the magnitude of large numbers). On the other hand, we can get a very good sense for the intensity of suffering in situations that are presented to us, which is what you point out as being the problematic part.
My sense is that it would be a really bad idea to try to get people to have a very intuitive grasp of intense suffering, for all of the problems you point out. I think that maybe the idea here is to try to give people some sense for it, but in a very small dose, which is sufficient to allow people to relate to other's suffering but not nearly enough to cause them harm. Of course, there isn't a right level for everyone (I think that watching the White Christmas/Black Museum episodes of Black Mirror was an adequate level of this for me, in that sense), but I think that a small enough dose here would be beneficial, and my claim is that this dose can be higher than simply "hundreds of thousands of people die of malaria every year". Make-a-Wish does make some effort into describing the children's situation, but they don't go as far as describing the details of their suffering in a way that could be traumatizing, even though that's possible to do in many cases.
And yes, there are definitely ways to frame this in a more positive and inspirational way, which I strongly favor!
Great point!
The correspondence between theoretical and practical efficiency is definitely not perfect. Theoretical efficiency guarantees that individuals are properly incentivized. Practical efficiency may not follow because of things like computational costs, and the extent to which this will be a problem will depend on the specific mechanism and the situation in question. For example, in the computational cost case, the actions of large companies would probably be closer to optimal behavior than individual actions.
My hunch would be that proving theoretical efficiency is generally a relatively good proxy for practical efficiency in most cases, but these other practical considerations should be considered in addition to it, as further constraints that one is trying to satisfy. But this is an empirical question, and I'm also relatively uncertain here.
Thanks, JP!
My impression is that these theoretical properties are the main reason why people are excited about QF. For example, you would prefer it over 1:1 donation matching because it is a more "principled" matching rule, which should lead to an allocation that's closer to the efficient level than 1:1 donation matching. So if not for these properties, I don't see why people should expect this mechanism to work particularly well in practice.
More generally, I agree that full on efficiency shouldn't be thought of as a strictly necessary condition mechanisms to be useful in practice. For example, majority voting works relatively well in practice, despite not being efficient. But efficiency is nevertheless still a central concept, as it is the very motivation behind the public goods provision property (it's basically the only problem with providing public goods privately). The framing I would use here is that increasing efficiency, while satisfying other (context-dependent) considerations, should be considered a key goal of public good provision mechanisms.
As this post explains, the main study that people cite when saying that "superforecasters are better than experts" comes from a competition where the aggregation methods for the two groups was different (Good Judgment Project's aggregation algorithm versus prediction market with low liquidity for amateur forecasters and experts, respectively). Prediction markets for forecasters and experts had similar performance.
Thanks for the comment! On the point of making this information more well-known, is there an easy way to do so, given that I have very little familiarity with these communities?
I haven't tried it, and it could turn out to be quite easy, but I think it's probably not so trivial to prove the result either way.
Thanks for these thoughts! I agree with most of what you said. Some replies to specific points:
Thanks for the comment, John! I agree with your point about preference aggregation as a main drawback, and I wish that EAs would appreciate this point more. The reason why I chose not to make it a drawback is because this criticism applies to most of the public goods provision literature, as opposed to applying specifically in the case of QF. But hopefully my points in the discussion about potential applications and your comment will bring more attention to this issue.
Thanks a lot for sharing your experience, Austin! I've added a link to your comment in the post. I'm not surprised that it didn't do great for getting more donations to the charities (as the post suggests), but I'm intrigued by your impression that it didn't do well in allocating money to different charities. What was your expectation regarding how allocations would be made, and how were they actually made instead?
And it's really interesting to know that Gitcoin is also de-emphasizing quadratic funding. Their website still mentions quadratic funding quite a lot; do you know if they have written this down somewhere?
As for fancy funding methods, I agree that the s-process looks interesting on the face of it. But I don't think my opinion here is more valuable than anyone else's, and I don't know how it compares to other mechanisms in this space. It would be great if someone thought through the theoretical considerations in that case, and try to get a sense for how participants/funders feel about it (like this testimony). This feels relevant given how much money the SFF has moved to date.
When I wrote the post, the framing I used was focused on the differences in direct impact of focusing on local causes. This meant not mentioning a number of other important considerations, both positive and negative. Since writing the post, I've had the opportunity to talk more to others, and to reflect some more on what these other considerations are. Below, I highlight some that I consider particularly important (see Sjir's comment for some others).
Getting people involved with effective interventions for local problems can serve as a way to get more people involved with effective global interventions. This can happen because it will be easier for some people to get involved with a local problem first before getting involved with a global problem, and also because working on local-level interventions can increase the popularity of EA in the region. I've recently learned that GiveDirectly's US cash transfer program appears to have led to a really large increase to their international donations to people in extreme poverty, due to these two channels. They claim that this was the case, and a quick look at their funding from previous years shows no signs of such a large increase in (international) donations in 2020. If this increase in international donations was indeed not caused by something else, then this example makes me a lot more optimistic about this indirect benefit of doing LPR. Relevantly, I should note that GiveDirectly (1) first consolidated their international transfers program before starting their US transfer program, and (2) tried to emphasize international transfers following the media attention they obtained for US transfers.
A second point in favor of LPR is that, in practice, it doesn't need to be the case that resources directed to local causes come at the expense of resources for the global cause. If an EA group already has a well-developed core group, and can already do outreach to those interested in the global cause, then the group might be able to start dedicating resources to LPR at a relatively low cost. As long as this doesn't compromise the group's ability to do outreach for global issues properly, I think that LPR might become a valuable activity for EA groups to engage in.
It's also worth highlighting some downsides:
These considerations highlight a timing aspect of the overlap between LPR and context-specific GPR. They suggest that LPR is generally better suited for more mature EA groups, which have already consolidated their outreach structure for people willing to work on global problems. Depending on how high the benefits of LPR turn out to be, this could mean that at some point we would want to do LPR even in the US. But there's still a lot of uncertainty, and I think that more information on the benefits and costs of LPR can be quite valuable. I would feel excited about attempts to get a sense for the magnitude of some effects mentioned here, such as by evaluating the GiveDirectly US transfers case more carefully, and looking for other related cases. And I'm excited about EA doing some work on LPR for the information value, particularly in the countries where it's most likely to be the best local opportunity, such as India.
This is the same concept as what I refer on the post as contextualization research; see Vaidehi's comment.
Given a heavy-tailed distribution of impact at the local level, redirecting donations from the average charity to a better-than-median but not great charity is also likely to have negative impact in counterfactual terms. I won't explain why here, but you can read about properties of heavy-tailed impact in this paper.
Thanks for the detailed comment, Geoffrey!
First off, there is a point worth clarifying here. Scope insensitivity makes it impossible to have feelings that adequately scale with the number of beings affected, and I don't think that there is much that can be done here (sidenote: this Wait But Why post and its sequel are the best ways I'm aware of to try to get an intuitive sense for the magnitude of large numbers). On the other hand, we can get a very good sense for the intensity of suffering in situations that are presented to us, which is what you point out as being the problematic part.
My sense is that it would be a really bad idea to try to get people to have a very intuitive grasp of intense suffering, for all of the problems you point out. I think that maybe the idea here is to try to give people some sense for it, but in a very small dose, which is sufficient to allow people to relate to other's suffering but not nearly enough to cause them harm. Of course, there isn't a right level for everyone (I think that watching the White Christmas/Black Museum episodes of Black Mirror was an adequate level of this for me, in that sense), but I think that a small enough dose here would be beneficial, and my claim is that this dose can be higher than simply "hundreds of thousands of people die of malaria every year". Make-a-Wish does make some effort into describing the children's situation, but they don't go as far as describing the details of their suffering in a way that could be traumatizing, even though that's possible to do in many cases.
And yes, there are definitely ways to frame this in a more positive and inspirational way, which I strongly favor!
Thanks! :)