Thanks Aaron. Measurement problems were a big issue. We experimented with incentives a bit, particularly offering to randomly select from students who completed the post-program survey, and those selected would receive money to give to a charity of their choice, but that didn't seem to make a difference, or at least we weren't in a place to offer a significant enough incentive to make a noticeable difference.
The other measurement problem that we ran into was knowing that, given the age of workshop participants, in most cases we wouldn't be able to measure actionable impact for another ~5 years.
Could you put together a handbook and/or video that could be sent to all trainees or is it critical that there be interaction between the trainer and trainee?
This would be along the lines of what we would consider doing if we explore this model further. However, I think there would still need to be a vetting process of some kind so that the we can be confident about the quality of the content and delivery.
Definitely true on both counts. I suspect that many answers are signalling intentions, but social desirability certainly has a role to play, as we mentioned above. This is one of the reasons we are now placing less emphasis on the future collection of quantitative survey data.
Great point, and we should have mentioned more about our intention to track things like career or career path changes as a result of the program. We don't currently have data on this because our audience are generally too young to show reliable signs of moving toward an effective career, but part of what we hope to accomplish with extended engagement (detailed in the Plans for Autumn 2018 section above) is to follow high-potential participants more closely so we can monitor changes like that.
We have had several participants state their intentions of taking actions like this to make a bigger impact, but it is uncertain as to whether they will follow through.
We experimented with this model in the early days of SHIC and didn't have much success, but it may have been partly because we didn't have the bandwidth to adequately prepare and support the university students who had volunteered. We are considering a second attempt with a stronger training/support system in place. Some of the limitations we suspect are:
As @Khorton mentioned, we place a lot of value on good presentation skills in order to engage the students. Thus we prefer instructors to have some background in teaching or public speaking.
Better-than-average knowledge of the relevant topics may also be a key component, as participant questions can be fairly complex. We have been slowly compiling a list of FAQs that future instructors can use in training.
As a substantial amount of time and effort goes in to training an instructor, it may only be cost-effective if that instructor is able to commit to running a fair number of workshops. Many university students/local group members are unable to make that commitment.
We believe that these limitations are potentially surmountable, but haven't made any plans as of yet to test this model further.
I see where you're coming from. When we wrote this I think we were referring to the fact that there don't seem to be any clubs or programs at the high school level that have the same goals as SHIC, and that we're feeling like we're in uncharted territory as we've been reaching out to these institutions, teachers and students, mainly because building a curriculum into the network seems like a new approach.
In any case, that's not well explained in the document, and I don't think that sentence gives off the message we want it to, so I'll strongly consider editing. Thanks!
Thanks Aaron. Measurement problems were a big issue. We experimented with incentives a bit, particularly offering to randomly select from students who completed the post-program survey, and those selected would receive money to give to a charity of their choice, but that didn't seem to make a difference, or at least we weren't in a place to offer a significant enough incentive to make a noticeable difference.
The other measurement problem that we ran into was knowing that, given the age of workshop participants, in most cases we wouldn't be able to measure actionable impact for another ~5 years.
Great idea, thank you!
This would be along the lines of what we would consider doing if we explore this model further. However, I think there would still need to be a vetting process of some kind so that the we can be confident about the quality of the content and delivery.
Definitely true on both counts. I suspect that many answers are signalling intentions, but social desirability certainly has a role to play, as we mentioned above. This is one of the reasons we are now placing less emphasis on the future collection of quantitative survey data.
Great point, and we should have mentioned more about our intention to track things like career or career path changes as a result of the program. We don't currently have data on this because our audience are generally too young to show reliable signs of moving toward an effective career, but part of what we hope to accomplish with extended engagement (detailed in the Plans for Autumn 2018 section above) is to follow high-potential participants more closely so we can monitor changes like that.
We have had several participants state their intentions of taking actions like this to make a bigger impact, but it is uncertain as to whether they will follow through.
We experimented with this model in the early days of SHIC and didn't have much success, but it may have been partly because we didn't have the bandwidth to adequately prepare and support the university students who had volunteered. We are considering a second attempt with a stronger training/support system in place. Some of the limitations we suspect are:
As @Khorton mentioned, we place a lot of value on good presentation skills in order to engage the students. Thus we prefer instructors to have some background in teaching or public speaking.
Better-than-average knowledge of the relevant topics may also be a key component, as participant questions can be fairly complex. We have been slowly compiling a list of FAQs that future instructors can use in training.
As a substantial amount of time and effort goes in to training an instructor, it may only be cost-effective if that instructor is able to commit to running a fair number of workshops. Many university students/local group members are unable to make that commitment.
We believe that these limitations are potentially surmountable, but haven't made any plans as of yet to test this model further.
I see where you're coming from. When we wrote this I think we were referring to the fact that there don't seem to be any clubs or programs at the high school level that have the same goals as SHIC, and that we're feeling like we're in uncharted territory as we've been reaching out to these institutions, teachers and students, mainly because building a curriculum into the network seems like a new approach.
In any case, that's not well explained in the document, and I don't think that sentence gives off the message we want it to, so I'll strongly consider editing. Thanks!