I would suggest that this post probably shouldn't have quotation marks in the title. It makes it look like there is a quote from Sunak that includes the phrase "existential threats", but that isn't the case.
I'll be honest, I'm probably not going to go back through all of the quotations now and give the separate posts they come from. Karnofsky did put the whole series into a single pdf which can function as a single source (but as I say, I haven't gone through again and checked myself).
I do recognize the bit you are quoting though - yes that is indeed where I got my quotations for that part from - and I did think a bit the issue you are bringing up. To me, this seems like a fair paraphrase/description of Karnofsky in that section:
a) He's developed a mindset in which he critically examines things like this with as much rigour as possible;
and
b) He has made a lot of investment in examining this thesis.
So it seemed - and still seems - to me that he is implying that he has invested a lot of time in critically examining this thesis with as much as rigour as possible.
However, I would not have used the specific phrasing that JoshuaBlake used. i.e. I deliberately did not say something like 'Karnofsky claims tohave written these posts with as much rigour as possible". So I do think that one could try to give him the benefit of the doubt and say something like:
'Although he spent a lot of time examining the thesis with as much rigor as possible, it does not necessarily follow that he wrote the posts in a way that shows that. So criticising the writing in the posts is kind of an unfair way to attack his use of rigour.'
But I think to me this just seemed like I would be trying a bit too hard to avoid criticising him. This comes back to some of my points in my post: i.e. I am suggesting that his posts are not written in a way that invites clear criticism, despite his claim that his is one of his main intentions and I suggest that Karnofsky's rhetorical style aims for the best of both worlds: He wants his readers to think both that he has thought about this very rigorously and critically for a long time but also - wherever it seems vague or wrong - to give him the benefit of the doubt and say 'well it was never meant to be taken too seriously or to be 100% rigorous, they're just blog posts etc.'.
(Just a note: Of course you are free to go into more detail in comments but I'm not sure I have much bandwidth to devote to writing long replies.)
I don’t really understand the response of asking me what I would have done. I would find it very tricky to put myself in the position of someone who is writing the posts but who also thinks what I have said I think.
Otherwise, I don’t think you’re making unreasonable points, but I do think that in the piece itself I already tried to directly address much of what you’re saying.
Ie You talk about his ideas being hard to define or hard to precisely explain but that we all kind of get the intention. Among other things, I write about how his vagueness allows him to lump together many versions of his claims under one heading, which hides the true level of disagreement and uncertainty that is likely to exist about any particular precise version of a claim. And about how it means he can focus on persuading you (-and potentially persuading you to act or donate etc) above figuring out what’s true. Etc
(It feels weird to me to have to repeat that we should be skeptical of - or at least very careful with - ideas/arguments that seem to continue to be “hard to explain” but that “we all kinda get what we mean”.)
And I don’t expect to actually be written like analytic philosophy either: eg one of my points here is that it isn’t reasonable to suppose that he is unaware of the standards of academic philosophy and so it doesn’t feel right for him to suggest that he is using a high level of rigour etc
Yeah, it wasn't too clear to me how to think about using the community tag but I decided to go with it in the end. This exchange however makes it look like people tend to disagree and think I shouldn't have used it. Hmmm, I'm not sure.
My mind comes back to points like this very often:
We cannot help but be reminded of Frank H. Westheimer's advice to his research students: “Why
spend a day in the library when you can learn the same thing by working in the laboratory for a
month?"
It's an - or perhaps yet another - example of how sometimes the Bay Area/SE/entrepreneurial mindset is almost diametrically opposed to certain mindsets coming from academia and how this community is trying to balance them or get the best of both worlds (which isn't a stupid thing to try to do per se, it just seems like sometimes it's very tricky). In the spirit of the former you kinda want to move fast (and break things), but the latter wants you to remember the virtues of deliberately taking the time to demonstrate how thorough and methodical you are being (and partly so that you don't, say, squander your resources by running a foreseeably dud experiment)
The case for risk that you sketch isn't the only case that one can lay out, but if we are focussing on this case, then your response is not unreasonable. But do you want go give up or do you want to try? The immediate response to your last suggestion is surely: Why devote limited resources to some other problem if this is the one that destroys humanity anyway?
You might relate to the following recent good posts:
Given the similarities, why are there so many different organizations? How is an outsider >supposed to know what makes each of them unique?
...
What makes these different from each other?
You are not the first person to ask questions like this. And I feel like I often hear people trying to make databases of researchers and orgs or give overviews of the field etc. etc. But the situation you describe just seems...normal, to me. As in, most industries and areas, even pretty niche things will have many different organizations with slightly different outlooks and no central way to navigate them and learn about them. And most research fields in academia are like this; there isn't a great way to get to know everyone in the field, where they work, what they work on etc, it just takes years of talking to people, reading things, meeting them at conferences etc. and you just slowly build up the picture .
I don't think it's silly or bad to ask what you're asking, that's not what I'm saying/I may well be wrong anyway, but to my mind the situation seems like it wouldn't necessarily have a good 'explanation' that someone can just hand to you, and the usual way to find out the sorts of things you want to find out would be to just continue immersing yourself in the field (from what I hear from most people who start figuring out this field, ~15 hours is not very long).
I think this is pretty well said and I too am worried that what I'm seeing is a certain amount of turning inward and hastily figuring out "some kind of consolidated narrative" behind closed doors.
Interesting. Have not read; only skimmed and bookmarked. But one of my initial naive reactions to the framing is that I have more or less always viewed technical safety work that is ‘aimed’ relatively near term as a form of buying time (I’m thinking of pretty much all work that is aimed at AI via the ML/DL paradigm, eg alignment work that is explicitly stuff to do with big transformers etc) .
I would suggest that this post probably shouldn't have quotation marks in the title. It makes it look like there is a quote from Sunak that includes the phrase "existential threats", but that isn't the case.
I appreciate the comment.
I'll be honest, I'm probably not going to go back through all of the quotations now and give the separate posts they come from. Karnofsky did put the whole series into a single pdf which can function as a single source (but as I say, I haven't gone through again and checked myself).
I do recognize the bit you are quoting though - yes that is indeed where I got my quotations for that part from - and I did think a bit the issue you are bringing up. To me, this seems like a fair paraphrase/description of Karnofsky in that section:
a) He's developed a mindset in which he critically examines things like this with as much rigour as possible;
and
b) He has made a lot of investment in examining this thesis.
So it seemed - and still seems - to me that he is implying that he has invested a lot of time in critically examining this thesis with as much as rigour as possible.
However, I would not have used the specific phrasing that JoshuaBlake used. i.e. I deliberately did not say something like 'Karnofsky claims to have written these posts with as much rigour as possible". So I do think that one could try to give him the benefit of the doubt and say something like:
'Although he spent a lot of time examining the thesis with as much rigor as possible, it does not necessarily follow that he wrote the posts in a way that shows that. So criticising the writing in the posts is kind of an unfair way to attack his use of rigour.'
But I think to me this just seemed like I would be trying a bit too hard to avoid criticising him. This comes back to some of my points in my post: i.e. I am suggesting that his posts are not written in a way that invites clear criticism, despite his claim that his is one of his main intentions and I suggest that Karnofsky's rhetorical style aims for the best of both worlds: He wants his readers to think both that he has thought about this very rigorously and critically for a long time but also - wherever it seems vague or wrong - to give him the benefit of the doubt and say 'well it was never meant to be taken too seriously or to be 100% rigorous, they're just blog posts etc.'.
(Just a note: Of course you are free to go into more detail in comments but I'm not sure I have much bandwidth to devote to writing long replies.)
I don’t really understand the response of asking me what I would have done. I would find it very tricky to put myself in the position of someone who is writing the posts but who also thinks what I have said I think.
Otherwise, I don’t think you’re making unreasonable points, but I do think that in the piece itself I already tried to directly address much of what you’re saying.
Ie You talk about his ideas being hard to define or hard to precisely explain but that we all kind of get the intention. Among other things, I write about how his vagueness allows him to lump together many versions of his claims under one heading, which hides the true level of disagreement and uncertainty that is likely to exist about any particular precise version of a claim. And about how it means he can focus on persuading you (-and potentially persuading you to act or donate etc) above figuring out what’s true. Etc
(It feels weird to me to have to repeat that we should be skeptical of - or at least very careful with - ideas/arguments that seem to continue to be “hard to explain” but that “we all kinda get what we mean”.)
And I don’t expect to actually be written like analytic philosophy either: eg one of my points here is that it isn’t reasonable to suppose that he is unaware of the standards of academic philosophy and so it doesn’t feel right for him to suggest that he is using a high level of rigour etc
Thanks for the comment.
Yeah, it wasn't too clear to me how to think about using the community tag but I decided to go with it in the end. This exchange however makes it look like people tend to disagree and think I shouldn't have used it. Hmmm, I'm not sure.
My mind comes back to points like this very often:
It's an - or perhaps yet another - example of how sometimes the Bay Area/SE/entrepreneurial mindset is almost diametrically opposed to certain mindsets coming from academia and how this community is trying to balance them or get the best of both worlds (which isn't a stupid thing to try to do per se, it just seems like sometimes it's very tricky). In the spirit of the former you kinda want to move fast (and break things), but the latter wants you to remember the virtues of deliberately taking the time to demonstrate how thorough and methodical you are being (and partly so that you don't, say, squander your resources by running a foreseeably dud experiment)
The case for risk that you sketch isn't the only case that one can lay out, but if we are focussing on this case, then your response is not unreasonable. But do you want go give up or do you want to try? The immediate response to your last suggestion is surely: Why devote limited resources to some other problem if this is the one that destroys humanity anyway?
You might relate to the following recent good posts:
https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategy
https://www.lesswrong.com/posts/uFNgRumrDTpBfQGrs/let-s-think-about-slowing-down-ai
You are not the first person to ask questions like this. And I feel like I often hear people trying to make databases of researchers and orgs or give overviews of the field etc. etc. But the situation you describe just seems...normal, to me. As in, most industries and areas, even pretty niche things will have many different organizations with slightly different outlooks and no central way to navigate them and learn about them. And most research fields in academia are like this; there isn't a great way to get to know everyone in the field, where they work, what they work on etc, it just takes years of talking to people, reading things, meeting them at conferences etc. and you just slowly build up the picture .
I don't think it's silly or bad to ask what you're asking, that's not what I'm saying/I may well be wrong anyway, but to my mind the situation seems like it wouldn't necessarily have a good 'explanation' that someone can just hand to you, and the usual way to find out the sorts of things you want to find out would be to just continue immersing yourself in the field (from what I hear from most people who start figuring out this field, ~15 hours is not very long).
I think this is pretty well said and I too am worried that what I'm seeing is a certain amount of turning inward and hastily figuring out "some kind of consolidated narrative" behind closed doors.
One reaction when reading this is that you might be kind of eliding the difference between utilitarianism per se and expected value decision analysis.
Interesting. Have not read; only skimmed and bookmarked. But one of my initial naive reactions to the framing is that I have more or less always viewed technical safety work that is ‘aimed’ relatively near term as a form of buying time (I’m thinking of pretty much all work that is aimed at AI via the ML/DL paradigm, eg alignment work that is explicitly stuff to do with big transformers etc) .