I work on biosecurity grantmaking with Effective Giving. I am not aiming to be anonymous/pseudonymous on here but sparingly write my full name because I prefer my Forum activity not to appear in search engines.
Thanks for doing this survey and sharing the results, super interesting!
Regarding
maybe partly because people who have inside views were incentivised to respond, because it’s cool to say you have inside views or something
Yes, I definitely think that there's a lot of potential for social desirability bias here! And I think this can happen even if the responses are anonymous, as people might avoid the cognitive dissonance that comes with admitting to "not having an inside view." One might even go as far as framing the results as "Who do people claim to defer to?"
I would probably frame some things a bit differently (more below), but I think you raise some solid points, and I definitely support the general call for nuanced discussion.
I have a personal anecdote that really speaks to your "do your homework point." When doing research for our 2021 article on dual-use risks (thanks for referencing it!) , I was really excited about our argument for implementing "dual-use evaluation throughout the research life cycle, including the conception, funding, conduct, and dissemination of research." The idea that effective dual-use oversight requires intervention at multiple points felt solid, and some feedback we'd gotten on presentations of our work gave me the impression that this was a fairly novel framing.
It totally wasn't! NSABB called for this kind of oversight throughout the research cycle (at least) as early as 2007, [1] and, in hindsight, it was pretty naïve of me to think that this simple idea was new. In general, it's been a pretty humbling experience to read more of the literature and realise just how many of the arguments that I thought were novel based on their appearance in recent op-eds and tweets can be found in discussions from 10, 20, or even 50 years ago.
Alright, one element of your post that I would've framed differently: You put a lot of emphasis on the instrumental benefits of nuanced discussion in the form of building trust and credibility, but I hope readers of your post also realise the intrinsic value of being more nuanced.
E.g., from the summary
"[what you say] does impact how much you are trusted, whether or not you are invited back to the conversation, and thus the potential to make an impact"
And the very last sentence:
"Always make sure ‘you’re invited back to the table’."
This is a great point, and I really do think it's possible to burn bridges and lose respect by coming across as ignorant or inflammatory. But getting the nuanced details wrong is also a recipe for getting solutions wrong! As you say, proper risk-benefit analysis for concrete dual-use research is almost always difficult, given that the research in question very often has some plausible upside for pandemic preparedness or health more generally.
And even if you know what specific research to draw red lines around, implementation is riddled with challenges: How do you design rules that won't be obsolete with scientific advances? How do you make criteria that won't restrict research that you didn't intend to restrict? How do you avoid inadvertent attention hazards from highlighting the exact kinds of research that seem the most risky? Let's say you've defined the perfect rules. Who should be empowered to make the tough judgment calls on what to prohibit? If you're limiting access to certain knowledge, who gets to have that access? And so on, and so on.
I do think there's value in strongly advocating for more robust dual-use oversight or lab biosafety, and (barring infohazard concerns), I think op-eds aimed at both policymakers and the general public can be helpful. It's just that I think such advocacy should be more in the tone of "Biosecurity is important, and more work on it is urgently needed" and less "Biosecurity Is Simple, I Would Just Ban All GOF."
Bottom line, I especially like the parts of your post that encourage people to be more nuanced, not just sound more nuanced.
From Casadevall 2015: "In addition to defining the type of research that should elicit heightened concern, the NSABB recommended that research be examined for DURC potential throughout its life span, from experimental conception to final dissemination of the results."
For instance, I’m worried people will feel bait-and-switched if they get into EA via WWOTF then do an 80,000 Hours call or hang out around their EA university group and realize most people think AI risk is the biggest longtermist priority, many thinking this by a large margin.
I particularly appreciate your point about avoiding 'bait-and-switch' dynamics. I appreciate that it's important to build broad support for a movement, but I ultimately think that it's crucial to be transparent about what the key considerations and motivations are within longtermism. If, for example, the prospect of 'digital minds' is an essential part of how leading people in the movement think about the future, then I think that should be part of public outreach, notwithstanding how offputting or unintuitive it may be. (MacAskill has a comment about excluding the subject here).
One thing I disagreed with.
MacAskill at times seemed reluctant to quantify his best-guess credences, especially in the main text.
I agree it's good to be transparent about priorities, including regarding the weight placed on AI risk within the movement. But I tend to disagree that it's so important to share subjective numerical credences and it sometimes has real downsides, especially for extremely speculative subjects. Making implicit beliefs explicit is helpful. But it also causes people to anchor on what may ultimately be an extremely shaky and speculative guess, hindering further independent analysis and leading to long citation trails. For example, I think the "1-in-6" estimate from The Precipice may have led to premature anchoring on that figure, and likely is relied upon too much relative to how speculative it necessarily is.
I appreciate that there are many benefits of sharing numerical credences and you seem like an avid proponent of sharing subjective credences (you do a great job at it in this post!), so we don't have to agree. I just wanted to highlight one substantial downside of the practice.
In a nutshell: I agree that caring about the future doesn't mean ignoring the present. But it does mean deprioritising the present, and this comes with very real costs that we should be transparent about.
I think this quote from Piper is worth highlighting:
(...) if the shift to longtermism meant that effective altruists would stop helping the people of the present, and would instead put all their money and energy into projects meant to help the distant future, it would be doing an obvious and immediate harm.That would make it hard to be sure EA was a good thing overall, even to someone like me who shares its key assumptions.
I broadly agree with this, except I think the first "if" should be replaced with "insofar as." Even as someone who works full-time on existential risk reduction, it seems very clear to me that longtermism is causing this obvious and immediate harm; the question is whether that harm is outweighed by the value of pursuing longtermist priorities.
GiveWell growth is entirely compatible with the fact that directing resources toward longtermist priorities means not directing them toward present challenges. Thus, I think the following claim by Piper is unlikely to be true:
My main takeaway from the GiveWell chart is that it’s a mistake to believe that global health and development charities have to fight with AI and biosecurity charities for limited resources.
To make that claim, you have to speculate about the counterfactual situation where effective altruism didn't include a focus on longtermism. E.g., you can ask:
Would major donors still be using the principles of effective altruism for their philanthropy?
Would support for GiveWell charities have been even greater in that world?
Would even more people have been dedicating their careers to pressing current challenges like global development and animal suffering?
My guess is that the answer to all three is "yes", though of course I could be wrong and I'd be open to hear arguments to the contrary. In particular, I'd love to see evidence for the idea of a 'symbiotic' or synergistic relationship. What are the reasons to think that the focus on longtermism has been helpful for more near-term causes? E.g., does longtermism help bring people on board with Giving What We Can who otherwise wouldn't have been? I'm sure that's the case for some people, but how many? I'm genuinely curious here!
To be clear, it's plausible that longtermism is extremely good for the world all-things-considered and that longtermism can coexist with other effective altruism causes.
But it's very clear that focusing on longtermism trades off against focusing on other present challenges, and it's critical to be transparent about that. As Piper says, "prioritization of causes is at the heart of the [effective altruism] movement."
Thanks for researching and writing this!
Thanks for doing this survey and sharing the results, super interesting!
Regarding
Yes, I definitely think that there's a lot of potential for social desirability bias here! And I think this can happen even if the responses are anonymous, as people might avoid the cognitive dissonance that comes with admitting to "not having an inside view." One might even go as far as framing the results as "Who do people claim to defer to?"
Hi Elika,
Thanks for writing this, great stuff!
I would probably frame some things a bit differently (more below), but I think you raise some solid points, and I definitely support the general call for nuanced discussion.
I have a personal anecdote that really speaks to your "do your homework point." When doing research for our 2021 article on dual-use risks (thanks for referencing it!) , I was really excited about our argument for implementing "dual-use evaluation throughout the research life cycle, including the conception, funding, conduct, and dissemination of research." The idea that effective dual-use oversight requires intervention at multiple points felt solid, and some feedback we'd gotten on presentations of our work gave me the impression that this was a fairly novel framing.
It totally wasn't! NSABB called for this kind of oversight throughout the research cycle (at least) as early as 2007, [1] and, in hindsight, it was pretty naïve of me to think that this simple idea was new. In general, it's been a pretty humbling experience to read more of the literature and realise just how many of the arguments that I thought were novel based on their appearance in recent op-eds and tweets can be found in discussions from 10, 20, or even 50 years ago.
Alright, one element of your post that I would've framed differently: You put a lot of emphasis on the instrumental benefits of nuanced discussion in the form of building trust and credibility, but I hope readers of your post also realise the intrinsic value of being more nuanced.
E.g., from the summary
And the very last sentence:
This is a great point, and I really do think it's possible to burn bridges and lose respect by coming across as ignorant or inflammatory. But getting the nuanced details wrong is also a recipe for getting solutions wrong! As you say, proper risk-benefit analysis for concrete dual-use research is almost always difficult, given that the research in question very often has some plausible upside for pandemic preparedness or health more generally.
And even if you know what specific research to draw red lines around, implementation is riddled with challenges: How do you design rules that won't be obsolete with scientific advances? How do you make criteria that won't restrict research that you didn't intend to restrict? How do you avoid inadvertent attention hazards from highlighting the exact kinds of research that seem the most risky? Let's say you've defined the perfect rules. Who should be empowered to make the tough judgment calls on what to prohibit? If you're limiting access to certain knowledge, who gets to have that access? And so on, and so on.
I do think there's value in strongly advocating for more robust dual-use oversight or lab biosafety, and (barring infohazard concerns), I think op-eds aimed at both policymakers and the general public can be helpful. It's just that I think such advocacy should be more in the tone of "Biosecurity is important, and more work on it is urgently needed" and less "Biosecurity Is Simple, I Would Just Ban All GOF."
Bottom line, I especially like the parts of your post that encourage people to be more nuanced, not just sound more nuanced.
From Casadevall 2015: "In addition to defining the type of research that should elicit heightened concern, the NSABB recommended that research be examined for DURC potential throughout its life span, from experimental conception to final dissemination of the results."
Thank you for writing this.
I do not think that this is accurate, I believe that some of their grants are not posted to their website.
Thank you for writing this, I think it's very important.
Oh, and I also quite liked your section on 'the balance of positive vs negative value in current lives'!
Thanks for writing this!
One thing I really agreed with.
I particularly appreciate your point about avoiding 'bait-and-switch' dynamics. I appreciate that it's important to build broad support for a movement, but I ultimately think that it's crucial to be transparent about what the key considerations and motivations are within longtermism. If, for example, the prospect of 'digital minds' is an essential part of how leading people in the movement think about the future, then I think that should be part of public outreach, notwithstanding how offputting or unintuitive it may be. (MacAskill has a comment about excluding the subject here).
One thing I disagreed with.
I agree it's good to be transparent about priorities, including regarding the weight placed on AI risk within the movement. But I tend to disagree that it's so important to share subjective numerical credences and it sometimes has real downsides, especially for extremely speculative subjects. Making implicit beliefs explicit is helpful. But it also causes people to anchor on what may ultimately be an extremely shaky and speculative guess, hindering further independent analysis and leading to long citation trails. For example, I think the "1-in-6" estimate from The Precipice may have led to premature anchoring on that figure, and likely is relied upon too much relative to how speculative it necessarily is.
I appreciate that there are many benefits of sharing numerical credences and you seem like an avid proponent of sharing subjective credences (you do a great job at it in this post!), so we don't have to agree. I just wanted to highlight one substantial downside of the practice.
In a nutshell: I agree that caring about the future doesn't mean ignoring the present. But it does mean deprioritising the present, and this comes with very real costs that we should be transparent about.
Thanks for sharing this!
I think this quote from Piper is worth highlighting:
I broadly agree with this, except I think the first "if" should be replaced with "insofar as." Even as someone who works full-time on existential risk reduction, it seems very clear to me that longtermism is causing this obvious and immediate harm; the question is whether that harm is outweighed by the value of pursuing longtermist priorities.
GiveWell growth is entirely compatible with the fact that directing resources toward longtermist priorities means not directing them toward present challenges. Thus, I think the following claim by Piper is unlikely to be true:
To make that claim, you have to speculate about the counterfactual situation where effective altruism didn't include a focus on longtermism. E.g., you can ask:
My guess is that the answer to all three is "yes", though of course I could be wrong and I'd be open to hear arguments to the contrary. In particular, I'd love to see evidence for the idea of a 'symbiotic' or synergistic relationship. What are the reasons to think that the focus on longtermism has been helpful for more near-term causes? E.g., does longtermism help bring people on board with Giving What We Can who otherwise wouldn't have been? I'm sure that's the case for some people, but how many? I'm genuinely curious here!
To be clear, it's plausible that longtermism is extremely good for the world all-things-considered and that longtermism can coexist with other effective altruism causes.
But it's very clear that focusing on longtermism trades off against focusing on other present challenges, and it's critical to be transparent about that. As Piper says, "prioritization of causes is at the heart of the [effective altruism] movement."