Comments
Is there a reason why organising an AI Alignment conference is not on the list? It seems to be one of the more successful things CEA has done for the EA field.
Is there a reason why organising an AI Alignment conference is not on the list? It seems to be one of the more successful things CEA has done for the EA field.
Why are people downvoting this post? I hope people didn't read my comment as some sort of refutation (it wasn't); I think Akash has proposed some good ideas to discuss.
People sometimes ask me what types of AIS field-building projects I would like to see.
Here’s a list of 11 projects.
But first, a few background points.
Purpose: Raise awareness about AI safety around the world to find highly talented AI safety researchers.
How this reduces P(doom): Maybe there are extremely promising researchers (e.g., people like Paul Christiano and Eliezer Yudkowsky) out in the world who don’t know about AI alignment or don’t know how to get involved. One global talent search program could find them. Alternatively, maybe we need 1000 full-time AI safety researchers who are 1-3 tiers below “alignment geniuses”. A separate global talent search program could find them.
Imaginary example: Crossover between the Atlas Fellowship, old CFAR, and MIRI. I imagine an organization that offers contests, workshops, and research fellowships in order to attract talented people around the world.
Skills needed: Strong models of community-building, strong understanding of AI safety concepts, really good ways of evaluating who is promising, good models of downside risks when conducting broad outreach
Olivia Jimenez and I are currently considering working on this. Please feel free to reach out if you have interest or advice.
Purpose: Provide excellent training, support, internships, and mentorship for junior AI alignment researchers.
How this reduces P(doom): Maybe there are people who would become extremely promising researchers if they were provided sufficient support and mentorship. This program mentors them.
Imaginary example: Something like a big version of SERI-Mats with a strong emphasis on workshops/activities that help people develop strong inside views & strong research taste. (My impression is that SERI-Mats could become this one day, but I’d also be excited to see more programs “compete” with SERI-Mats).
Skills needed: Relationships with AI safety researchers, strong models of mentors, strong ability to attract and assess applicants, insight into how to pair mentors with mentees, good models of AI safety, good models of how to create organizations with epistemically rigorous cultures, good models of downside risks when conducting broad outreach.
Purpose: Provide excellent support for AI alignment researchers in major EA Hubs.
Imaginary example: Something like a big version of Lightcone Infrastructure that runs something like Bell Labs, regularly hosts high-quality events/workshops for AI alignment researchers, or accelerates research progress through alignment newsletters, podcasts, and debates (my impression is that Lightcone or Constellation could become this one day, but I’d be excited to see people try parts of this on their own).
Skills needed: Strong relationships with AI safety researchers, strong understanding of the AI safety community and its needs, and strong understanding of AI safety concepts. Very high context would be required to run a space; medium context would be required to perform the other projects.
I am currently considering starting an AI alignment podcast or newsletter. Please feel free to reach out if you have interest or advice.
Purpose: Identify highly promising people who are already part of the EA community and get them funding/connections/mentorship to do AIS research or launch important/ambitious projects.
How this reduces P(doom): Maybe there are people who would become extremely promising researchers or ambitious generalists who are already part of the EA community but haven’t yet received the support, encouragement, or mentorship required to reach their potential.
Imaginary example: Crossover between the FTX Future Fund’s regranting program, a longtermist incubator, and CEA’s active stewardship vision. I envision a group of “superconnectors” who essentially serve as talent scouts for the EA community. They go to EA globals and run retreats/workshops for new EAs, as well as highly-skilled EAs who aren’t currently doing highly impactful work. They provide grants for people (or encourage people to apply for funding) to skill-up in AI safety or launch ambitious projects.
Skills needed: Strong models of community-building, large network or willingness to develop a large network, strong models of how to identify which people and projects are most promising, strong people skills/people judgment.
Purpose: Identify highly promising researchers in academia and industry, engage them with high-quality AI safety content, and support those who decide to shift their careers/research toward technical AIS.
How this reduces P(doom): Maybe there are extremely talented researchers who can already be identified based on their contributions in fields related to AI alignment (e.g., math, decision theory, probability theory, CS, philosophy) and/or their contributions to messy and pre-paradigmatic fields of research.
Imaginary example: An organization that systematically reads research in relevant fields, identifies promising researchers, and designs targeted outreach strategies to engage these researchers with high-quality sources in AI alignment research. The Center for AI Safety and the AI Safety Field Building Hub may do some of this, though they’re relatively new, and I’d be excited for more people to support them or compete with them.
Skills needed: Strong understanding of how to communicate with researchers, strong models of potential downside risks, strong understanding of AI safety concepts, good models of academia and “the outside world”, good people skills.
Note that people considering this are strongly encouraged to reach out to community-builders and AI safety researchers before conducting outreach to experienced researchers.
People interested in this may also wish to read the Pragmatic AI Safety Sequence and should familiarize themselves with potential risks associated with outreach to established researchers. Note that people disagree about how to weigh upside potential against downside risks, and “thinking for yourself” would be especially important here.
Purpose: Understand the AI scene in China, conduct research about if/how AIS outreach should be conducted in China, deconfuse EA about AIS in China, and potentially pilot AIS outreach efforts in China.
How this reduces P(doom): Maybe there are effective ways to reach out to talented people in China in ways that sufficiently mitigate downside risks. My current impression is that China is one of the leaders in AI, and it seems plausible that China would have a lot of highly talented people who could contribute to technical AIS research. However, I’ve heard that AIS outreach in China has been neglected because EA leaders don’t understand China and don’t understand how to evaluate different kinds of outreach strategies in China (hence the focus on research/deconfusion/careful pilots).
Imaginary example: A think tank-style research group that develops strong models of a specific topic.
Skills needed: Strong understanding of China, fluency in Mandarin, strong ability to weigh upside potential and downside risks.
Purpose: Identify (or develop) subproblems in alignment & turn these into highly-advertised contests.
How this reduces P(doom): Maybe there are subproblems in AI alignment that could be solved by researchers outside of the AI x-risk community. Alternatively, maybe contests are an effective way to get smart people interested in AI x-risk.
Imaginary example: An organization that gets really good at creating contests based on problems like ELK and The Shutdown Problem (among other examples) & then advertising these contests heavily.
Skills needed: Ideally a strong understanding of AI safety and the ability to identify/write-up subproblems. But I think this could work if someone was working closely with AI safety researchers to select & present subproblems.
Purpose: Write extremely clear, engaging, and persuasive explanations of AI safety ideas.
How this reduces P(doom): There are not many introductory resources that clearly explain the importance of AI safety. Maybe there are people who would engage with AI safety if we had better introductory resources.
Imaginary example: A crossover between Nick Bostrom, Will MacAskill, Holden Karnofsky, and Eliezer Yudkowsky. A book or blog that is as rigorous as Bostrom’s writing (Superintelligence), as popular as Will’s writing (NYT bestseller with media attention), as clear as Holden’s writing (Cold Takes), and as explicit about x-risk as Yudkowsky’s writing (e.g., List of Lethalities)
Skills needed: Ideally a strong understanding of AI safety, but I think writing ability is probably the more important skill. In theory, someone with exceptional writing ability could work closely with AI safety researchers to select the most important topics/concepts and ensure that the descriptions/explanations are accurate. Also, strong models of potential downside risks of broad outreach.
I am grateful to Olivia Jimenez, Thomas Larsen, Miranda Zhang, and Joshua Clymer for feedback.
Love seeing posts like this!
At EAG-SF I asked a MATS organizer if we could get other versions of MATS, e.g. a MATS competitor at MIT. Their response was that only one of the two could survive because there are currently only ~15 people capable of doing this kind of mentorship. Mentors are the bottleneck for scaling up programs like MATS, not field builders.
Isn't Vael Gates already mostly focused on this? ("My projects tend to be aimed more at outreach and at older populations– AI researchers, academia and industry.") Curious what the main benefits are of a separate project.
See this comment: Tianxia focuses on building the longtermist community, while Concordia focuses on all things AI, including recruiting STEM undergrads and AI grad students to start working on AI safety. I think you already know this, so I'm wondering why you think it's not enough to focus on scaling up these existing orgs.
Might want to mention CAIS here?
I don't see why people need to be good at management and community building if they end up doing AIS technical research. Maybe you're using "generalists" to mean "people who will start new AIS orgs/projects"?
Ops teams can take care of some of this. AI Safety Support offers a completely free health coach for people working on AI safety. More importantly, I think an executive assistant who works exclusively for Paul Christiano would save him more time than a larger org that can't work with him as closely. MacAskill certainly has assistants, and the top alignment researchers should as well. I think your idea is to have an org that executive assistants can outsource some common tasks to?
I think the implicit assumption is that if you're someone who wants to make the "AI story go well," a necessary pre-requisite is understanding AIS in a fair amount of detail. (See e.g. this elucidation by Owen C-B). I'm not sure I believe this myself, but at the very least it sounds plausible.