Comments
Epistemic status: Validity of claims - Moderate to strong depending on claim, usefulness of claims - weak to moderate
The notion of an "apocalyptic residual" is described by Nick Bostrom in his paper titled the Vulnerable World Hypothesis. The apocalyptic residual essentially describes people who desire human catastrophe and are willing to work towards it even at harm to themselves.
- Grading, measuring and understanding the apocalyptic residual is probably useful
- Apocalyptic residual has implications for whether systems weilding catastrophic power should be authoritarian or distributed
In this post, I am using the phrase "apocalyptic residual" to only include people who are willing to cause mass death of groups they belong to, including themselves. I am excluding people who only wish to cause destruction of outgroups - be it foreign countries or ethnic groups - but are not willing to accept symmetric destruction of their ingroup to achieve this aim. I'm not sure whether this defintion carves reality at the joints best, perhaps future research can explore that.
There exists literature (1, 2) on why people pursue extremist ideologies - both their self-stated reasons and underlying psychological mechanisms. Some motivations are formed in extremist groups, whereas others are formed by "lone wolves", the latter are harder to profile.
Some conclusions by Randy Borum, Psychology of Terrorism Initiative:
Although early writings on the “psychology of terrorism” were based mostly in psychoanalytic theory (e.g., narcissism, hostility toward parents), most researchers have since moved on to other approaches.
People become terrorists in different ways, in different roles, and for different reasons. It may be helpful to distinguish between reasons for joining, remaining in, and leaving terrorist organizations.
Perceived injustice, need for identity and need for belonging are common vulnerabilities among potential terrorists.
Mental illness is not a critical factor in explaining terrorist behavior. Also, most terrorists are not “psychopaths.”
There is no “terrorist personality”, nor is there any accurate profile – psychologically or otherwise – of the terrorist.
Histories of childhood abuse and trauma and themes of perceived injustice and humiliation often are prominent in terrorist biographies, but do not really help to explain terrorism.
Terrorist ideologies tend to provide a set of beliefs that justify and mandate certain behaviors. Those beliefs are regarded as absolute, and the behaviors are seen as serving a meaningful cause.
Not all extremist ideologies promote violence, nor are all extremists violent. One might ask whether the ideology is driven more by promotion of the “cause” or destruction of those who oppose it.
There is also some literature (1, 2) on extremists groups that explicitly aim for catastrophe. From "The Life Course of Apocalyptic Groups"
Apocalyptic terrorism gained greater attention in the 1990’s.19 The distinction between this type of terrorism and the more traditional terrorism was becoming clearer as the new millennium approached. The greatest distinction seemed to lie in the desire of apocalyptically inspired terrorists to destroy the existing order and to engage in indiscriminate killing to do so. These new terrorists were increasingly dangerous because the sociopolitical restraints that kept the old terrorists in check no longer applied.20 These new terrorists were only accountable to their deity, so the amount of carnage they could theoretically inflict was difficult to estimate but extensive if they were to acquire WMD
Cults that hold apocalyptic beliefs can also be dangerous because they see the world in dichotomous terms, their judgment is blinded, and, to them, the law is relative.22 Apocalyptic cults form the basis of their worldviews by distorting the writings found in the major religions or other systems of belief. For example, People’s Temple and the Branch Davidians distorted the Christian religion, while Heaven’s Gate based its actions on belief in UFO’s. Cults with an apocalyptic vision, and terrorist groups for that matter, have a greater proclivity for violence when led by a charismatic-millenarian leader.2
Whether a person belongs to the apocalyptic residual may not be a binary question. The key factors that determine how strongly a person belongs to the apocalyptic residual are:
- how much and what kinds of personal harm are they willing to endure?
- in return for how much increased likelihood of human extinction?
- do they want catastrophe or extinction?
How much personal harm?
At the lowest end we have people who are willing to anonymously spend economic resources on projects with catastrophic aims. Rich actors can do more damage than poor actors.
At the highest end we have people who are willing to endure torture and painful death of themselves and people they care about, to achieve their ends.
How much increased likelihood?
At the lowest end we have people who are willing to work towards catastrophic aims only if they know their own work is decisive in ensuring catastrophe occurs, i.e., they are not willing to tolerate a failed attempt at catastrophe.
At the highest we have people who are willing to work even to increase the odds of catastrophe by some number of percentage or basis points.
Catastrophe or extinction?
This could depend significantly on the underlying ideology or motivation of the member.
Having quality screening procedures can reduce the likelihood of a selected person pursuing catastrophic aims.
While identification and mitigation are important topics, I will not be covering them much here.
There are multiple reasons to want to measure the statistical distribution of people belonging to various grades of the apocalyptic residual, and to find factors that correlate with them.
Some institutions need to weild catastrophic power for reasons other than causing catastrophe. Examples include nuclear weapons being held for deterrence purposes, or biohazards being held for dual-use research. Such institutions will therefore want to ensure that their processes are not co-opted by members of the apocalyptic residual.
Knowing the distribution helps predict the likelihood of such co-opting, and is therefore relevant to these institutions, as well as to the x-risk reduction community.
Screening procedures are effective, and help constrain the distribution further. Knowing the constrained distribution could also be useful.
Another reason to get statistics regarding the apocalyptic residual is to predict the degree of disincentive an institution may have to apply to get its members to not work towards catastrophic aims. For instance if a person is willing to risk economic resources but not personal life, then threats to life for violating rules of an institution may be more effective at ensuring those institutions are not captured or used by people with catastrophic aims. However if they are willing to risk their life, then these incentives are far less effective.
Measuring the apocalyptic residual among people outside of formal institutions is also relevant for us, however. This is because the Vulnerable World Hypothesis also considers scenarios where people not typically in positions of power, are yet able to obtain the knowledge to work towards catastrophic aims. More specifically this is a Type-1 vulnerability as per Bostrom's terminology.
Unlike inside formal institutions however, there is less of a direct incentive for institutions to study such risks, and therefore we may depend on others to further such research.
There are multiple implications of this research for whether institutions that weild catastrophic power must be more distributed or more authoritarian. This post won't try to settle this debate but it will explore implications of the apocalypitc residual on both.
As Bostrom has noted, greater the apocalyptic residual outside of formal institutions, the more reason there exists for institutions to pre-emptively police and surveil them. Depending on the ease, capability and likelihood of people outside of institutions being able to weild catastrophic power - the optimal amount of surveillance and policicing required sharply shifts from very little to very large - the function is closer to logistic or binary rather than linear.
A greater apocalyptic residual inside of formal institutions could however mean the opposite - that power should be distributed inside of the formal institution.
For instance if the odds that any selected person belongs to the apocalyptic residual is 0.1%, the odds of a catastrophic weapon being fired is significantly higher if one leader gets to control it, versus say a majority (3) of 5 leaders who are independently selected from the same statistical distribution. At the limit it could even make sense to have large democratic vote, to fire catastrophic weapons or control catastrophic tech - if it is indeed known that all members belong to the same distribution and have the same values.
There are however diminishing returns to including more people in the decision-making process, and at some point this benefit will be overriden by other considerations, such as whether the values and competencies of all members with this power are sufficient or optimal.
There is also incentive to have less churn - for instance a leader having this power for 30 years has less risk as compared to rotating the person every 4 years.
Another key factor is levels of competence to achieve their objective. The people trying to wake chuthulu by chanting magic words aren't a problem. The doomsday cult doing cutting edge biotech or AI research is a problem. How many rationality points do these doomsday cultists have?
Yup definitely a very important factor.