Discussion about AI Safety funding (FB transcript)

Akash

Discussion about AI Safety funding (FB transcript)

Akash

7 min readApr 30, 2023

Comments 10

Sorted by

New & upvoted

Chris Leong

I suspect creating funds with clearer guidelines for what would be considered reasonable would help. For example, many people might not apply for an AIS Hub Travel Grant to the Bay Area because when you start looking into how expensive finding somewhere to stay in the Bay Area is, it starts to look ridiculous.

Ryan Kidd

Copying over the Facebook comments I just made.

Response to Kat, intended as a devil's advocate stance:

As Tyler said, funders can already query other funders regarding projects they think might have been rejected. I think the unilateralist's curse argument holds if the funding platform has at least one risk-averse big spender. I'm particularly scared about random entrepreneurs with typical entrepreneurial risk tolerance entering this space and throwing money at projects without concern for downsides, not about e.g. Open Phil + LTFF + Longview accessing a central database (though such a database should be administered by those orgs, probably).
I'm very open to hearing solutions to the risk of a miscommunication-induced unilateralist's curse. I think a better solution than a centralized funding database would be a centralized query database, where any risk-averse funders can submit a request for information to a trusted third party, who knows every proposal that was rejected by all parties and can connect the prospective funders with the funders who rejected the proposal for more information. This reduces the chances that potentially risk-tolerant funders get pinged with every grant proposal but increases the chances that risk-averse funders request information that might help them reject too-risky proposals. I know it's complicated, but it seems like a much better mechanism design if one is risk-averse.
It seems pretty unlikely that small projects will get noticed or critiqued on the EA Forum, but low-quality small projects might be bad en masse. Future Fund gave a lot of money to projects and people that got low visibility but might have contributed to "mass movement building concerns" around "diluted epistemics", "counterfactually driving the wheel of AI hype and progress," and "burning bridges for effective outreach."
Open-sourcing funding analysis is a trade-off between false positives and false negatives for downside risk. Currently, I'm much more convinced that Open Phil and other funders are catching the large downside projects than an open source model would avoid these. False alarms seem safer than downside risk to me too, but this might be because I have a particularly low opinion of entrepreneurial risk tolerance and feel particularly concerned about "doing movement building right" (happy to discuss MATS' role in this, btw).

A few key background claims:

AI safety is hard to do right, even for the experts. Doing it wrong but looking successful at it just makes AI products more marketable but doesn't avert AGI tail-risks (the scary ones).
The market doesn't solve AI risk by default and probably makes it worse, even if composed of (ineffectively) altruistic entrepreneurs. Silicon Valley's optimism bias can be antithetical to a “security mindset." “Deploy MVP + iterate” fails if we have to get it right first on the first real try. Market forces cannot distinguish between AI “saints” and “sycophants" unaided.
Big AI x-risk funders are generally anchoring on the "sign-value" of impact rather than "penny-pinching" when rejecting projects. Projects might sometimes get submaximal funding because the risk/benefit ratio increases with scale.

Greg_Colbourn

-2

Agree with your background claims. But think we should be pivoting toward advocacy for slowing down / pausing / shutting down AI capabilities in general, in the post GPT-4+AgentGPT era. Short timelines means we should lower the bar for funding, and not worry quite so much about downside risks (especially if we only have months to get a moratorium in place).

Ryan Kidd

I think that one's level of risk aversion in grantmaking should depend on the upside and the downside risk of grantees' action space. I see a potentially high upside to AI safety standards or compute governance projects that are specific, achievable, and verifiable and are rigorously determined by AI safety and policy experts. I see a potentially high downside to low-context and high-bandwidth efforts to slow down AI development that are unspecific, unachievable, or unverifiable and generate controversy or opposition that could negatively affect later, better efforts.

One might say, "If the default is pretty bad, surely there are more ways to improve the world than harm it, and we should fund a broad swathe of projects!" I think that the current projects to determine specific, achievable, and verifiable safety standards and compute governance levers are actually on track to be quite good, and we have a lot to lose through high-bandwith, low-context campaigns.

Greg_Colbourn

later, better efforts.

How much later are these efforts happening? I feel like EA leadership is asleep at the wheel here, and the EA community is not cut out for the emergency response we need in terms of how fast it can react (judging by the reaction so far).

Larks

We’re probably going to be setting up channels where funders can discuss applicants. This way if there are concerns about net negativity, other funders considering it can see that. This might even lead to less unilateralist curse because if lots of funders think that the idea is net negative, others will be able to see that, instead of the status quo, where it’s hard to know what other funders think of an application.

I'm not convinced this is a very satisfactory mechanism. It can be very hard to share negative feedback, especially as this is often very personal (e.g. the project idea is good but the applicant is untrustworthy, or they would mess it up very badly and spoil the water for another attempt, etc.) and with a sufficiently large pool of funders viewing your written evaluation the risk that the applicant is informed could become unacceptably high.

MichaelStJules

Why is it hard to share negative feedback? Would it be bad in expectation if the applicant is informed?

With more funders, it could be closer to anonymous. Even some internal discussion could be made anonymous if there are concerns about leaks.

Ofer

The local incentives people face often discourage publicly giving negative feedback that may cause an applicant to not get funding. ("I would gain nothing from giving negative feedback, and that person might hate me.")

Jason

Asking people in the funding network to discuss applications they think are net negative seems like a big ask. At present, I assume that each grantmaker has a pass/fail evaluation function in connection to their own funding bar.

I'm sure this is simplified/imprecise/overquantified, but their thought process might look something like this: As the grantmaker evaluates each application, they implicitly develop a point estimate with a confidence interval (let's say 90% CI for this example). As the grantmaker spends more time on evaluation, the CI shrinks but may remain quite broad. The grantmaker will prematurely abort their evaluation process if the entire CI is below their funding bar at any time. At that point, the risk of erroneous rejecting the application is low enough that devoting further resources to the evaluation is not a good use of their time from the perspective of their grantmaking program.

However, the grantmaker may often reach the conclusion that the entire 90% CI is below the funding bar before they investigate deeply enough to reach a sufficiently firm conclusion that the project is net-negative that they would feel comfortable voicing that. Understandably, many grantmakers will want to be fairly confident that an application is net-negative before making a semi-public statement that could unilaterally torpedo the applicant and/or application.

As can be seen in some of the comments above, a grantmaker who calls out a grant to others is also exposing themselves to reputational risk, which (being human) is another reason they might feel the need to mitigate by conducting additional evaluation on a grant they already decided to reject. ^[1] Plus there's the work of writing up a rationale in enough detail to mitigate reputational risk and actually persuade others. Therefore, it seems that asking grantmakers to identifiably communicate their net-negativity findings would be asking them to do a meaningful amount of extra work. That would detract to some extent from the time they spend evaluating grants they might actually make.

I would be more inclined to rely on a more robust process than mere communication. If we assume all funders are "informed, smart, and value aligned EAs,"^[2] then a random screening panel could be an option. Conditional on all three members selecting best estimate is net harmful, ^[3] what are the odds that asking 100 "informed, smart, and value aligned EAs" would yield a majority finding the proposal net positive? ^[4] Probably low enough to justify refusing to send the application on to the funding network. That will lead to erroneous rejections, but that risk has to be weighed against the probability that the proposal truly is net harmful in expectation and that forwarding it onward risks a unilateralist funding it.

Of course, one could use a randomish-sample-of-evaluators method in other ways, such as asking each funder who evaluates an application to anonymously rate it as likely net beneficial, likely net negative, or declines to opine. The current balance could be displayed with the application, and if the balance ever reveals a high enough probability that a majority of the whole group would vote net harmful, the proposal is removed. That has a lower chance than a screening panel of removing a majority-net-harmful application; maybe the unilateralist will be one of the first few evaluators. But it at least mitigates some of the disadvantages of expecting grantmakers who think the application net negative to take extra time away from their own grants and incur personal reputational risk for the collective good of mitigating unilateralist risk.

^{^}
This is not inconsistent with the idea that a substantial fraction of applications are overall harmful in expectation. It is much easier to be confident on a shallow review that (say) about 25% of all applications one sees would be net harmful than to be confident on a shallow review that a specific application would be net harmful.
^{^}
If not all funders would qualify, that raises other concerns.
^{^}
A 2-1 split in either direction could go to a second panel (although experience might ultimately reveal that a 2-1 split ultimately led to a certain outcome in a high percentage of cases).

^{^}

Moreover, erroneously rejecting a proposal that 51 or even 60 of these EAs would find net positive would be at most a fairly minor error. (I express no opinion here as to whether a percentage of evaluators somewhat under 50% concluding that an application was net harmful should also justify its rejection.)

Sanjay

Seeing this prompted me to post a draft document that I've been meaning to finalise for a while: https://forum.effectivealtruism.org/posts/oLkoKCo3cDzuc996G/coattailing-and-funging-to-learn-strategies-for-non-expert

Comments

Jason

^{^}
This is not inconsistent with the idea that a substantial fraction of applications are overall harmful in expectation. It is much easier to be confident on a shallow review that (say) about 25% of all applications one sees would be net harmful than to be confident on a shallow review that a specific application would be net harmful.
^{^}
If not all funders would qualify, that raises other concerns.
^{^}
A 2-1 split in either direction could go to a second panel (although experience might ultimately reveal that a 2-1 split ultimately led to a certain outcome in a high percentage of cases).

^{^}

Discussion about AI Safety funding (FB transcript)

Discussion about AI Safety funding (FB transcript)

Selected comments from FB thread

Kat Woods (Nonlinear)

Claire Zabel (Program Officer at Open Philanthropy)

Selected portions of Kat's response to Claire:

Larissa's response to Claire:

Kerry's response to Claire:

Caleb Parikh (Executive Director of EA Funds)

Kat's response to Caleb:

Thomas Larsen's response to Caleb

Akash (that's me)