Milton Friedman once said: “One of the great mistakes is to judge policies and programs by their intentions rather than their results.” While I don’t generally subscribe to Friedman’s right-wing philosophies, I do think he was onto something here. Few people would argue the good intentions of philanthropic programs that engage and empower young people as key decision-makers and change-makers…but good intentions don’t always add up to meaningful outcomes and long-term impact for society. How can we evaluate the outcomes and impact of our youth philanthropy programs?
As the National Program Manager for the Youth and Philanthropy Initiative (YPI) Canada, this is one of my key challenges. YPI is a public charitable foundation that provides grants to social service organizations across the country through an innovative curricular program that has engaged over 450,000 Grade 9 and 10 students since 2002 in real-world experiences with philanthropy and advocacy. Students participating in our program study social issues in their community, get out of the classroom to interview charity staff, volunteers and beneficiaries, and give pitch presentations to their peers, educating one another and advocating for grants for their chosen charities. Ultimately, the students in each school choose one charity in their community to receive a $5,000 grant from YPI.
This year, I had the pleasure of taking an online certificate course offered by YouthREX, “Program Evaluation for Youth Well-Being”, which is available to those working at youth-serving grassroots organizations in Ontario, Canada. I learned a lot, and with my team, I formalized an evaluation plan for YPI with the evaluation activities we already had in place after completing the course. This is an example of what a small organization can do with limited resources. On behalf of our team, I am pleased to share our full evaluation plan here.
Here are four key suggestions for how to approach evaluation in your youth philanthropy program:
- Evaluation is everywhere. At YPI, we approach evaluation as an integral component of the program – as a program manager, evaluation isn’t just a part of my job, it is my job. We think about evaluation as mission-critical, woven seamlessly throughout every process we undertake. Every year, we conduct annual pre- and post- program questionnaires with students, distribute feedback questionnaires to teachers and charity representatives, and host occasional focus groups and interviews with various stakeholders. Outside of these formal measures, we also use site visits, phone calls, and teacher-submitted data through project management forms to evaluate our success. We capture this data and record our notes in a database called Salesforce and by using other tools like Excel and Google forms. At the end of the year, we revisit this information when considering the outcomes of our program.
- Formalize your program theory. Creating a logic model for your program can be an eye-opening experience. In creating our evaluation plan, our team re-examined YPI’s logic model, and found this to be really helpful in thinking about our program on a macro-level. We dug deep using the theoretical framework of short-term, medium-term, and long-term impact to examine what changes we really want to see as a result of our program. Asking questions about the things we are trying to shift made designing evaluation questions and connecting these questions to evaluation tools (e.g., surveys, interview guides) flow naturally. With this groundwork we were able to think critically about what we already measure and what gaps we should look to fill.
- Don’t reinvent the wheel; look for existing research and frameworks. Ask this important question: “What can we reasonably measure with the resources we have?” In many cases, you’ll likely find that you are able to measure short- and medium-term outcomes, but that it would be challenging to measure the long-term impact of your program. This is okay! At YPI we have approached this challenge by looking to the best research we could find (most current, peer-reviewed, respected in our field) to tell us what short- and medium-term outcomes lead to the societal change related to our mandate. For us that meant finding research that showed what skills young people need in order to demonstrate meaningful civic engagement later in life. We use a framework called “Youth Who Thrive”, published by three respected organizations in Canada to provide the link between outcomes like those in our program and longer-term changes for individuals and society.
- Evaluation is a team effort: We are a small team with big ambitions at YPI. Each of our three core team members is charged with furthering our mission in a different way. This year, each one of us will play a role in analyzing and interpreting the data that comes out of our evaluation program. This is called triangulating, and in this way, we will double and triple check to see data from different perspectives to ensure that we are accurately representing what our data is telling us.
Finalizing our evaluation plan was also a team effort: we discussed in depth what we actually wanted to learn through our evaluation, and how we might best get there. It’s finished for now, but it remains a working document that we will revisit on an ongoing basis.
I hope that these suggestions will initiate conversations in a community of practice for youth philanthropy program staff and evaluators. We’d love to hear what you think! Email us at .
Special thanks to YouthREX, who played a key role in my journey as a program evaluator by supporting me through their online certificate course, Program Evaluation for Youth Well-Being. A version of this blog first appeared on their website here.
