Is EA compatible with technopessimism?

[Question]

Is EA compatible with technopessimism?

[anonymous]

2 min readDec 25, 2021

Comments 30

Sorted by

New & upvoted

Darius1

In my experience, many EAs have a fairly nuanced perspective on technological progress and aren't unambiguous techno-optimists.

For instance, a substantial fraction of the community is very concerned about the potential negative impacts of advanced technologies (AI, biotech, solar geoengineering, cyber, etc.) and actively works to reduce the associated risks.

Moreover, some people in the community have promoted the idea of "differential (technological) progress" to suggest that we should work to (i) accelerate risk-reducing, welfare-enhancing technologies (or ideas generally) and (ii) decelerate technologies (or ideas) with the opposite effects. That said, studying the concrete implications of differential progress seems fairly neglected and deserves to be explored in much greater depth. In line with the above idea, it seems common for EAs to argue that technological progress has been very beneficial in some regards—improving human welfare, especially over the last hundreds of years (e.g. here)—while it has been harmful in other regards, such as factory farming having led to greater animal suffering.

[anonymous]

Thanks this is super useful.

Although I guess the question now becomes, should we improve existing institutions or build new ones in ways that allow for differential tech progress, or is it better to prevent all progress.

Pablo Villalobos

The fact that risk from advanced AI is one of the top cause areas is to me an example of at least part of EA being technopessimist for a concrete technology. So I don't think there is any fundamental incompatibility, nor that the burden of proof is particularly high, as long as we are talking about specific classes of technology.

If technopessimism requires believing that most new technology is net harmful that's a very different question, and probably does not even have a well defined answer.

vivek

"risk from advanced AI is one of the top cause areas is to me an example of at least part of EA being technopessimist"
...assuming that particular example is a concern of such an impact primarily on humans, could that be articulated as anthropocentric technopessimism ?

On a broader sidebar, there is discussion around technology (particularly computing) in regards to ecological and other limits - e.g. https://computingwithinlimits.org

mako yass

...assuming that particular example is a concern of such an impact primarily on humans, could that be articulated as anthropocentric technopessimism ?

Why would you want to describe it that way?
On reflection, I don't think it can be called anthropocentric, no. There are four big groups of beings involved here: Humanity, Animals, Transhumanist post-humanity (hopefully without value-drift), and Unaligned AI. Three of those groups are non-human. Those concerned with AI alignment tend to be fighting in favor of more of those non-human groups than they are fighting against.

(It's a bit hard to tell whether we would actually like animals once they could speak, wield guns, occupy vast portions of the accessible universe etc. Might turn out there are fundamental irreconcilable conflicts. None apparent yet, though.)

[anonymous]

animals once they could speak

In my mind animals are defined by their inability to use language. Check out this list of things mostly unique to human language.

P.S. I assume you mean higher animals such as mammals here.

mako yass

I'm referring to https://en.wikipedia.org/wiki/Uplift_(science_fiction) , sort of under the assumption that if we truly value animals we will eventually give them voice, reason, coherence. On reflection, I guess the most humane form of this would probably consist of just aligning an AI with the animals and letting it advocate for them. There's no guarantee that these beings adapted to live without speech will want it, but an advocate couldn't hurt.

[anonymous]

I don't know, a lot of what you're saying still feels anthropocentric to me. (I'll assume mammals again since most animals don't have a brain or spinal chord.)

Language isn't just about speech, it's about cognitive processing. "Do you want speech?" is likely not an object you can input to the algorithm of a mammal's brain and get a meaningful response, and the same applies to most of the mental objects humans manipulate as part of using language.

Also there's no guarantee mammals even "want" consistent things, plenty of algorithms "want" nothing in particular, they just run. If at all they run in specific directions it's because outside forces terminated those running differently. (To some extent this also applies to humans.) Hence I have difficulty understanding what it means to align an AI to a mammal.

mako yass

Yeah, that objection does also apply to humans, which is why, despite it being so difficult to extract a coherent extrapolated volition from a mammal brain, we must find a way of doing it, and once we have it, although it might not produce an agenty utility function for things like spiders or bacteria, there's a decent chance it'll work on dogs or pigs.

[anonymous]

Right, I understand your opinion now.

I do still have some intutions on why CEV shouldn't exist (even for humans), but I'm not sure what the best place to discuss is, it's probably not here.

mako yass

Hm. Well feel free to notify me if you ever write it up.

[anonymous]

Tbvh I'm not sure how to engage with alignment folk because I feel like I'm missing a lot of existing knowledge and mental models that they do. Like I don't get what people think of when they think of "aligned AI", that they go - you know what, we don't even know how to rigorously define this thing yet we're convinced it exists and it is a good thing. But I'll try:

Bostrom mentions stuff like populating the cosmos with digital minds generating positive experiences, that seems like unusual and one of the things we could possibly want, but not necessarily the only thing we want. If it was the only thing we wanted we could actually work to explicitly specify that as the AI's goal, and that's CEV and hence problem solved.

Basically humans want a lot of different things, and we're confused about which things we want more. That doesn't necessarily mean there exists some objective answer as to which of those we want more, that we need to "solve" it. Instead it could just be random - there are neurochemicals firing from different portions of the brain and sometimes one thing wins out, sometimes the other thing wins out. And if you input a sufficiently persuasive sequence of words into this brain it will prioritise some things more, but if you input a different sequence it will prioritise different things more. (An AI with sufficient human manipulation skills can find this easy imo.) Turing machines don't have "values" by default, they have behaviours based on their input.

I also wrote up some stuff here a month back but idk how coherent it is: https://www.lesswrong.com/posts/uswN6jyxdkgxHWi7F/samuel-shadrach-s-shortform?commentId=phwddxuNNuumAbSxC

mako yass

If it was the only thing we wanted we could actually work to explicitly specify that as the AI's goal, and that's CEV and hence problem solved.

This is just an aside, but it might be informative. I actually think that

single alignment: "This specific blob of meat here "is" an "agent". Figure out its utility function and do that"

is going to be simpler to program than

Hard-coded: "make a large number of many "humanlike" things "experience" "happiness""

I think it's clear that.. there seem to be more things in the hard-coded solution that we don't know how to formalize, and we're much further from knowing how to formalize them (we're pretty deep into agency already). And it's fairly likely we'll arrive at a robust account of agency in the process of developing AGI, or as a result of stumbling onto it, it seems to be one of the short paths to it.

I agree about the hmm, murcurial quality of of human agency. There seem to be a lot of triggers that "change" the utility function. Note, I don't know if anyone knows what we mean when we say that a utility function was undermined, by a compelling speech, or by acclimating to the cold water, or by falling in love or whatever. It's strange. And humans generally tend to be indifferent to these changes, they identify with process of change itself. In a sense they all must be part of one utility function (~ "when I am in bed I want to stay in bed, but if I need to pee I would prefer to get out of bed, but once I'm out of bed I will not want to go back to bed. That is my will. That is what it means to be human. If you remove this then it's a dystopia."), but somehow we have to exclude changes like... being convinced by a superintelligence's rhetoric that maximizing paperclips is more important than preserving human life. Somehow we know that's a bad change, even though we don't have any intuitive aversion to reading superintelligences' rhetoric. Even though we (Well I think I would be at least) know we'd be convinced by it, somehow we know that we want to exclude that possibility.

[anonymous]

-1

"This specific blob of meat here "is" an "agent". Figure out its utility function and do that"

For me the very intuitive response to "figure out its utility function" is "this blob of meat doesn't have a utility function". A model that has a utility function is going to be a bad model of human behaviour. The module that causes humans to have desires is the very same module that causes humans to lose pennies, you can't really have one without the other. I can try justifying this - I have a bunch of different sets of intuitions that all point in the same direction - but it isn't that well written up. Maybe I'll do a better job conversing with you :)

https://kroma.substack.com/p/messy-post-intuitions-on-agi

[anonymous]

Thanks, this is a useful distinction.

Although I will note that the default case is you can't pick-and-choose what innovation will or will not happen, current institutions allow for both good and harm.

Maybe another axis would be something like, should we improve institutions ability to pick-and-choose or is this a doomed endeavour and instead should we adopt changes that slow down all tech progress.

Mauricio

This isn't directly answering your question, but one thing I can easily imagine happening is: some techno-pessimists spend lots of effort pulling together evidence and arguments for one premise of techno-pessimism, rush (or leave unmade) arguments for other crucial premises, and then find it frustrating that this doesn't change many people's minds. My guess is there's, say, 5+ substantive disagreements that are all upstream of typical disagreement on techno-pessimism, so I expect the bigger challenge for techno-pessimists in convincing this community will be, not necessarily having watertight arguments for any one background premise, but doing a thorough enough job at addressing the wide range of background disagreements.

I don't know about the community as a whole, so speaking just for myself, I think I'd require at least intuitive arguments (no RCTs needed) for each of the following premises to consider working on this--I'll leave it to you to judge whether that's a reasonable burden of proof:

That a technologically sophisticated future would be bad overall in expectation (i.e. (1)), even under ethical views that are most common in this community
That permanently, globally halting technological progress is tractable (in a way that isn't highly objectionable on other grounds)
That sufficiently improving the trajectory of AI is relatively intractable (or that AI in particular shouldn't be a dominant consideration)
Bonus points: directly addressing heuristics that make techno-pessimism very unintuitive (especially the heuristic that "ban all new technologies" is so un-nuanced/general that it's very suspect)

Mauricio

Other thoughts:

The bar also seems very different depending on precisely what organizations/individuals within the community you're trying to convince, and what you're trying to convince them of (e.g. the global health side of the community seems to be unresponsive to careful, intuitive arguments that are unaccompanied by RCTs, while the AI people are very interested in such arguments).
I think most of the difficulty comes from the generically very high burden of proof for arguing that any given cause area / intervention is the most effective use of resources, not from techno-pessimism-specific disagreements (although those don't help).
Re: "If you feel this is a bad framework, please let me know."--yup, it seems like this framework overlooks/obscures many other potential reactions to (1).

[anonymous]

depending on precisely what organizations/individuals within the community

I can understand why the bar is very different for organisations to react to information - some need to react very fast (startups), some need to be very cautious (judiciaries). But does that also apply to individual's personal opinions? Genuinely curious.

generically very high burden of proof for arguing that any given cause area / intervention is the most effective use of resources

Fair, although I will note most systemic change is not constrained by capital alone. For instance if you want a policy change in the govt, money can help but you also need to genuinely change the minds of some individuals.

yup, it seems like this framework overlooks/obscures many other potential reactions to (1).

Makes sense.

Mauricio

I can understand why the bar is very different for organisations to react to information - some need to react very fast (startups), some need to be very cautious (judiciaries). But does that also apply to individual's personal opinions?

I was thinking that individuals have different background beliefs (and different attentional/incentive environments), and these things create differences in what it will take to change people's minds. E.g. some people in the community are much closer to agreement with techno-pessimists than others--they'll likely have a lower burden of proof.

note most systemic change is not constrained by capital alone

Agreed, I had a broad definition of "resources" in mind (including e.g. labor/careers, connections), but I could have been more clear.

[anonymous]

Makes sense!

[anonymous]

Thanks this is useful.

even under ethical views that are most common in this community

Fair

(in a way that isn't highly objectionable on other grounds)

This shouldn't be too hard if the default case from tech progress is extinction / totalitarianism. But fair.

That sufficiently improving the trajectory of AI is relatively intractable

Short of "solving alignment", some AI alignment researchers already believe it's intractable. And I don't know what a proof of "solving alignment" being impossible looks like, because alignment researchers deliberately state their field is pre-paradigmatic. So there's no universal definition of what stuff like intelligence or values or alignment even mean, no standard approach to attacking the problem, and so on. Maybe this is just a personal qualm I have, but I still haven't understood what it's like inside the mind of someone who believes alignment is possible. So I don't know what would convince them.

heuristic that "ban all new technologies" is so un-nuanced

I find it more intuitive but this is very useful to know that it can be unintuitive to some.

Some intuition: problem is our social institutions, current values, power structures etc might be incapable of weilding very powerful tech responsibly. Our social institutions seem well-made for values that are unnuanced and unchanging, versus having to keep changing them every 50 years because new tech comes out every 50 years. They're already struggling to regulate the internet. So a deliberately unnuanced value like "all new tech is bad" might get encoded better than something more nuanced like differential tech progress, or changing the laws every 50 years.

I'm not actually convinced this is true, but just trying to describe why the option seems a natural one even if it is not the best one.

Mauricio

This shouldn't be too hard if the default case from tech progress is extinction / totalitarianism.

Maybe, although I suspect this assumption makes it significantly harder to argue that a technologically sophisticated future is net negative in expectation (since, at least by ethical views that seem especially common in this community, extinction leads to approximately net zero (not net negative) futures, and it seems plausible to me that a totalitarian future--with all the terrible loss of potential that would involve--would still be better than non-existence, i.e. not net negative).

I don't know what a proof of "solving alignment" being impossible looks like

Just to clarify, I wouldn't demand that--I'd be looking for at least an intuitive argument that solving alignment is intractable. I agree that's still hard.

I still haven't understood what it's like inside the mind of someone who believes alignment is possible

As a tangent (since I want to focus on tractability rather than possibility, although impossibility would be more than enough to show intractability): the main reason I think that alignment (using roughly this definition of alignment) is possible is: humans can be aligned to other humans; sometimes we act in good faith to try to satisfy another's preferences. So at least some general intelligences can be aligned. And I don't see what could be so special about humans that would make this property unique to us.

Returning from the tangent, I'm also optimistic about tractability because:

People haven't been trying for that long, and the field is still very small
At least some prominent, relatively new research directions (e.g. [1], [2], [3]) seem promising

Some intuition: [...]

Yup this seems plausible, you get the bonus points :)

[anonymous]

although I suspect this assumption makes it significantly harder to argue that a technologically sophisticated future is net negative in expectation

I'm not sure I follow. The alternative is to slow down tech progress and try to preserve the status quo. Or maybe even regress technologically, "spend more time with nature" etc. I assume all ethical views prefer status quo to extinction or totalitarianism - it's just the extent to which they prefer it and the amount of work/time/resources they're willing to put in (relative to other opportunities).

humans can be aligned to other humans

Oh I agree, I feel like superintelligence cannot be trusted, atleast the kind that's capable of global power-grabs. So I wouldn't want to give this power to a human being either.

I think it's largely because humans don't want consistent things, and cannot possibly want consistent things, short of neurosurgery. So I wouldn't want to delegate power to an agent that is extremely capable of consistent reasoning.

Mauricio

I'm not sure I follow. [...] I assume all ethical views prefer status quo to extinction or totalitarianism

I wonder if we might be using "net negative" differently? By "net negative" I mean "worse than non-existence," not "worse than status quo." So even though we may prefer a stable status quo to imminent extinction, we might still think the latter leaves us at roughly net zero (i.e. not net negative, or at least not significantly net negative).

I also suspect that, under many ethical views, some forms of totalitarianism would be better than non-existence (i.e. not net-negative). For example, a totalitarian world in which freedoms/individuality are extremely limited--but most people are mostly happy, and extreme suffering is very rare--seems at least a little better than non-existence, by the lights of many views about value.

(A lot of what I'm saying here is based on the assumption that, according to very scope-sensitive views of value: on a scale where -100 is "worst possible future" and 0 is "non-existence" and 100 is "best possible future," a technologically unsophisticated future would be approximately 0, because humanity would miss out on the vast majority of time and space in which we could create (dis)value. Which is why, for a technologically unsophisticated future to be better than the average technologically sophisticated future, the latter has to be net negative.)

Oh I agree, I feel like superintelligence cannot be trusted, at least the kind that's capable of global power-grabs. [...] I think it's largely because humans don't want consistent things, and cannot possibly want consistent things, short of neurosurgery.

I'm not sure if you mean that humans' preferences (a) are consistent at any given time but inconsistent over time, or (b) are inconsistent even if we hold time constant. I'd have different responses to the two.

Re: (a), I think this would require some way to aggregate different individuals' preferences (applied to the same individual at different times)--admittedly seems tricky but not hopeless?

Re: (b), I agree that alignment to inconsistent preferences is impossible. (I also doubt humans can be aligned to other humans' inconsistent preferences--if someone prefers apples to pears and they also prefer pears to apples (as an example of an inconsistent preference), I can't try to do what they want me to do when they ask for a fruit, since there isn't a consistent thing that they want me to do.) Still--I don't know, I don't feel that my preferences are that incoherent, and I think I'd be pretty happy with an AI that just tries to do what I want it to do to whatever extent I have consistent wants.

[anonymous]

Thanks for replying.

I wonder if we might be using "net negative" differently?

Yep I think you're using a stronger assumption in your ethical theories that situations are even comparable, if you ignore when they occur. Because the scores you're assigning to situations seem independent of the sequence of events that led to them or the absolute time they occur.

I guess whether you make such an assumption or not - does not matter to the fundamental statement that:

Status quo in 2021 evolving to a stable society with similar tech in 2100

is better than

Status quo in 2021 evolving to a dictatorship with mass suffering in 2100

Status quo in 2021 evolving to human extinction by 2100

As long as this is true, trying to get the first thing to happen (evolve to stable society) instead of second or third is worth doing if it were the only thing we could do in 2021. Question is whether your ethical theory also demands you to do other things that are even more worth doing (such as building utopia), and because you have finite time and resources you want to do that instead. (Which I'd understand.)

Regarding the AI alignment stuff, I mean (b), but I'm not sure if discussing here would be the best way to go about it. Maybe I should make a separate thread when I'm more confident.

Mauricio

I think you're using a stronger assumption in your ethical theories that situations are even comparable, if you ignore when they occur

Hm I wouldn't endorse that assumption. I avoided specifying "when"s to communicate more quickly, but I had them in mind something like your examples--agree the times matter.

trying to get the first thing to happen (evolve to stable society) instead of second or third is worth doing if it were the only thing we could do in 2021

Agreed but only if we add another condition/caveat: that trying to get the first thing to happen also didn't trade off against the probability of very good scenarios not covered in these three scenarios (which it would mathematically have to do, under some assumptions). As an oversimplistic example with made-up numbers, suppose we were facing these probabilities of possible futures:

20% -- your first scenario (tech stagnation) (10 goodness points)
5% -- your second scenario (mass suffering) (-1,000,000 goodness points)
20%--your third scenario (extinction) (-10 goodness points)
55%--Status quo in 2021 evolving to technologically sophisticated utopia by 2100 (1,000,000 goodness points)

And suppose the only action we could take in 2021 would change the above probabilities to the following:

100% -- your first scenario (tech stagnation) (10 goodness points)
0% -- your second scenario (mass suffering) (-1,000,000 goodness points)
0%--your third scenario (extinction) (-10 goodness points)
0%--Status quo in 2021 evolving to technologically sophisticated utopia by 2100 (1,000,000 goodness points)

Then the expected value of not taking the action is 500,000 goodness points, while the expected value of taking the action is 10 goodness points, so taking the action would be very bad / not worthwhile (despite how technically the action falls under your description of "trying to get the first thing to happen (evolve to stable society) instead of second or third [...] if it were the only thing we could do in 2021").

[anonymous]

Oh okay, I understood you now. You're right.

Linch

I recently convinced myself to be fairly technopessimistic in the short term (at least relative to some people I talk to in EA, unclear how this compares to e.g. online EAs or the population overall), though it's not a very exciting position and I don't know if I should prioritize writing up this argument over other things I can do that's productive.

[anonymous]

Thanks, this is useful.

Ben Millwood

Can you give examples of technopessimists "in the wild"? I'm sure there are plenty of examples of "folk technopessimism" but if you mean something more fleshed-out than that I don't think I've seen it expressed or argued for a lot. (That said, I'm not very widely-read, so I'm sure there's lots of stuff out there I don't hear about.)

Comments

Mauricio

I'm not sure I follow. [...] I assume all ethical views prefer status quo to extinction or totalitarianism

Oh I agree, I feel like superintelligence cannot be trusted, at least the kind that's capable of global power-grabs. [...] I think it's largely because humans don't want consistent things, and cannot possibly want consistent things, short of neurosurgery.

Re: (a), I think this would require some way to aggregate different individuals' preferences (applied to the same individual at different times)--admittedly seems tricky but not hopeless?