I help allocate Open Phil's resources to improve the governance of AI with a focus on avoiding catastrophic outcomes. Formerly co-founder of the Cambridge Boston Alignment Initiative, which supports AI alignment/safety research and outreach programs at Harvard, MIT, and beyond, co-president of Harvard EA, Director of Governance Programs at the Harvard AI Safety Team and MIT AI Alignment, and occasional AI governance researcher. GWWC pledger.
Yep, all sounds right to me re: not deferring too much and thinking through cause prioritization yourself, and then also that the portfolio is too broad, though these are kind of in tension.
To answer your question, I'm not sure I update that much on having changed my mind, since I think if people did listen to me and do AISTR this would have been a better use of time even for a governance career than basically anything besides AI governance work (and of course there's a distribution within each of those categories for how useful a given project is; lots of technical projects would've been more useful than the median governance project).
I just want to say I really like this style of non-judgmental anthropology and think it gives an accurate-in-my-experience range of what people are thinking and feeling in the Bay, for better and for worse.
Also: one thing that I sort of expected to come up and didn't see, except indirectly in a few vignettes, is just how much of one's life in the Bay Area rationalist/EA scene is comprised of work, of AI, and/or of EA. Part of this is just that I've only ever lived in the Bay for up to ~6 weeks at a time and was brought there by work, and if I lived there permanently I'd probably try to carve out some non-EA/AI time, but I think it's a fairly common experience for people who move to the Bay to do AI safety-related things to find that it absorbs everything else unless you make a conscious effort not to. At basically all the social events I attended, >25% of the attendees worked in the same office I did and >25% of the people at any given time are talking about AI or EA. This has not been my experience even while doing related full-time work in Boston, Oxford, and DC.
Again, part of this is that I've been in Berkeley for shorter stints that were more work-focused. But yeah, I think it's not just my experience that the scene is very intense in this way, and this amplifies everything in this post in terms of how much it affects your day-to-day experience.
That is a useful post, thanks. It changes my mind somewhat about EA's overall reputational damage, but I still think the FTX crisis exploded the self-narrative of ascendancy (both in money and influence), and the prospects have worsened for attracting allies, especially in adversarial environments like politics.
I agree that we're now in a third wave, but I think this post is missing an essential aspect of the new wave, which is that EA's reputation has taken a massive hit. EA doesn't just have less money because of SBF; it has less trust and prestige, less optimism about becoming a mass movement (or even a mass-elite movement), and fewer potential allies because of SBF, Bostrom's email/apology, and the Time article.
For that reason, I'd put the date of the third wave around the 10th of November 2022, when it became clear that FTX was not only experiencing a "liquidity crisis" but had misled customers, investors, and the EA community and likely committed massive fraud, and when the Future Fund team resigned. The other features of the Third Wave (the additional scandals and the rise in public interest in AI safety due to ChatGPT, GPT-4, the FLI letter, the CAIS statement, and so on) took a few months to emerge, but that week seems like the turning point.
I don't know how we got to whether we should update about longtermism being "bad." As far as I'm concerned, this is a conversation about whether Eric Schmidt counts as a longtermist by virtue of being focused on existential risk from AI.
It seems to me like you're saying: "the vast majority of longtermists are focused on existential risks from AI; therefore, people like Eric Schmidt who are focused on existential risks from AI are accurately described as longtermists."
When stated that simply, this is an obvious logical error (in the form of "most squares are rectangles, so this rectangle named Eric Schmidt must be a square"). I'm curious if I'm missing something about your argument.
Thanks for this post, Luise! I've been meaning to write up some of my own burnout-type experiences but probably don't want to make a full post about it, so instead I'll just comment the most important thing from it here, which is:
Burnout does not always take the form of reluctance to work or lacking energy. In my case, it was a much more active resentment of my work — work that I had deeply enjoyed even a month or two earlier — and a general souring of my attitude towards basically all projects I was working on or considering. While I was sometimes still, like, medium-productive, it was very bad for several additional reasons:
I endorse the recommendation to take time off when you're feeling burnout-type feelings, but at least in my experience, time off has diminishing returns and a high decay rate. In other words, instead of taking a week off every 3 months, I'm hoping to just build in more time off in my typical week (i.e., making social plans on at least one weekend afternoon, noticing when I'm getting tired at night and stopping work then). If anything, I've found that it's hard to get back into the swing of things after long vacations. But YMMV.
Other things that seem important, for me going forward, include:
The posts linked in support of "prominent longtermists have declared the view that longtermism basically boils down to x-risk" do not actually advocate this view. In fact, they argue that longtermism is unnecessary in order to justify worrying about x-risk, which is evidence for the proposition you're arguing against, i.e. you cannot conclude someone is a longtermist because they're worried about x-risk.
As of May 27, 2023, I no longer endorse this post. I think most early-career EAs should be trying to contribute to AI governance, either via research, policy work, advocacy, or other stuff, with the exception of people with outlier alignment research talent or very strong personal fits for other things.
I do stand by a lot of the reasoning, though, especially of the form "pick the most important thing and only rule it out if you have good reasons to think personal fit differences outweigh the large differences in importance." The main thing that changed was that governance and policy stuff now seems more tractable than I thought, while alignment research seems less tractable.
Edited the top of the post to reflect this.
I love the point about the dangers of "can't go wrong" style reasoning. I think we're used to giving advice like this to friends when they're stressing out about a choice that is relatively low-stakes, even like "which of these all-pretty-decent jobs [in not-super-high-impact areas] should I take." It's true that for the person getting the advice, all the jobs would probably be fine, but the intuition doesn't work when the stakes for others are very high. Impact is likely so heavy-tailed that even if you're doing a job at the 99th percentile of your options, it's probably (?) orders of magnitude worse than the 99.9th percentile — meaning you're giving up more than 90% of your impact.
A corollary is that different roles and projects within each cause area are also likely to be heavy-tailed, and once again, I hear the advice of "can't go wrong" in pretty inappropriate contexts. Picking the second-best option likely means giving up most of your impact, which is measured in expected lives saved. You can definitely go wrong!
Now, we all have limited cognition and for these kinds of choices; we ultimately have to make choices (and doing nothing is also a choice), and we'll inevitably make mistakes, and we should treat ourselves with some compassion. But maybe we should reframe comments like "you can't go wrong" as like, "sounds like you have some really exciting options and difficult choices ahead" — and, if you have the bandwidth to actually do this — "let me know if I can help you think them through!"
Correct that CBAI does not have plans to run a research fellowship this summer (though we might do one again in the winter), but we are tentatively planning on running a short workshop this summer that I think will at least slightly ease this bottleneck by connecting people worried about AI safety to the US AI risks policy community in DC - stay tuned (and email me at trevor [at] cbai [dot] ai if you'd want to be notified when we open applications).