Center for Regional Food Systems

This webinar provides an overview of program evaluation. The training covers a common definition of program evaluation, steps in the process, evaluation questions, evaluation types and feasible methods to consider. The training also offers insights into real world experience with program evaluation and tips for making evaluation practical in small, non-profit settings.

Webinar authors

• Kathryn Colasanti, CRFS Michigan Good Food Coordinator
• Courtney Pinard, Research Scientist for the Gretchen Swanson Center for Nutrition
• Sue Ann Savas, Clinical Assistant Professor at the University of Michigan School of Social Work
• Amanda Edmonds, Executive Director of Growing Hope

Overview of Program Evaluation Webinar Slides

Webinar Transcript:

Kathryn Colasanti: Courtney is joining us from Omaha, Nebraska, and the rest of us are coming to you live from the Office of the [Inaudible] in Ypsilanti, Michigan, and we're showing control. So if we [inaudible], but thank you, everyone, for joining us today. My name's Kathryn Colasanti with the Center for Regional Food Systems at Michigan State University. Welcome to this webinar. [Inaudible] introduce the speakers in just a moment, but, first, I want to give a little bit of context for today's webinar. This is part of our Shared Measurement project, which is part of the larger framework of collected impact that we have been utilizing and thinking about good systems work in Michigan and how we all work together more strategically for stronger outcomes, and Shared Measurement is a way for us to align what data we're collecting, how we're collecting it in ways that will better measure progress towards the goals of the Michigan [inaudible] Charter and also allow us to hold ourselves accountable to, to our goals and to compare and aggregate data across organizations. And this training is in response to a need that was identified through the Shared Measurement interviews conducted by Courtney with the [inaudible] Center for Nutrition, the, the interviews with our stakeholders and partners. Many of you identified an interest and a desire for just greater understanding around data collection. So we are starting off with two trainings, this first one on evaluation and one coming up on research that will just provide a foundational understanding around data collection, and building on those next year we'll be developing more specific trainings around specific data collection tools in line with Share Measurement [Inaudible] areas. So stay tuned for more on those. And now I'd like to introduce today's speakers. First, [inaudible] have Courtney Pinard, who is a research scientist at the Graduate Swanson Center for Nutrition. She holds a special dean's appointment as assistant professor in department of health promotions, social, and behavioral health at the College of Public Health with the University of Nebraska Medical Center. She conducts research focusing on public health outcomes, of policies, programs, and practices that impact nutrition and diet, and her work has an overarching emphasis on measurement and evaluation, which has been a great asset to us in this, the Shared Measurement project. And Courtney has particular interest in [inaudible] food access and how regional food systems can strengthen community food security. Next we'll hear from Sue Ann Savas, who is a clinical assistant professor at University of Michigan School of Social Work and director of the Curtis Center program evaluation group. Sue Ann has her M.S.W., M.B.A. in economics from University of Michigan and for over twenty years has practiced as a program evaluator and training students in evaluation. She specializes in the areas of program design, [inaudible] modeling, evaluation planning, implementation evaluations, outcome measurement, and reporting for continuous quality improvement, and she's published numerous articles and book chapters on techniques for building evaluation capacity within organizations and coalitions. And she's also part of the evaluation team that we're working with on one of our current Kellogg Foundation grants. And our last speaker today is Amanda Edmonds. Many of you likely know Amanda as founder and director since 2003 of Growing Hope, which is dedicated to helping people grow and access healthy food. Amanda is also an alumna of the University of Michigan School of Natural Resources and Environment. She served as a policy leader, both as a [inaudible] appointee to Michigan [Inaudible] Policy Council, as vice chair of [Inaudible] Council, and in November, 2014 was elected mayor [inaudible]. And Amanda travels nationally as a speaker and trainer in community organizing, organized, organizational development, and community food systems, and has also contributed her experience and expertise to the advisory committee for the Shared Measurement project. So with that, I'm going to turn it over to Courtney to talk a little bit more about the Shared Measurement Process.

Courtney Pinard: Hey, great. [Inaudible] thanks for signing up and being interested in this. You can hear me, right, Kathryn.

Kathryn Colasanti: Yes, I can.

Courtney Pinard: OK, great. I just wanted to check. So this slide here will be an overview for those of you who have been along with us for the Shared Measurement project. Just a refresher on what has been done and where we're going with all of this and how program evaluation really fits in with, within the Shared Measures framework. So as Kathryn mentioned, we have conducted interviews as well as collected a survey, and that really fed into our understanding of what's currently being conducted as far as measures that organizations and stakeholders in Michigan are currently doing and where the gaps might be and how we can find a focus area to align for a potential pilot. And so this is where we are right now, looking at building capacity and understanding more about what the needs are across organizations and then implementing a pilot which we'll be talking more about in 20, early 2016. And this is all been possible really through the input and guidance of the advisory committee, which has been helping us with making decisions on, you know, what, what this data really means, where we should be going. [Multiple speakers] and what are the needs of the communicators, and you'll hear more about that in, you know, the coming months, but really I think might be good to mention, our focused area has been on healthy food access, the economic impact, and institutional procurement. But, so that's just so a little reminder. Move this slide ahead. Kathryn, can you move the slide forward one.

Kathryn Colasanti: Yeah [multiple speakers].

Courtney Pinard: I thought I had it there. Thank you. And so through these interviews and surveys, it became apparent that there's really a wide range of data that's already being collected, and some of it might not even be considered data by those collecting or using it, and I think it's just important to highlight this is a strength and potential for Shared Measures and looking at where these overlaps are and how we can align. So looking across all these different types of data collection, groups that have common interest and even common types of measurements but not necessarily the same. So I think that's important, that we can look at those pieces that might overlap and how we can better align. And so we had, you know, a lot of qualitative information that people were already collecting through interviews and focus groups, and this I think is really right now is more of a novel area in understanding where consumers and communities are at. So we have develop kind of this basis with this qualitative information, and now as we move forward, you know, we have a better evidence base and understanding of what's going on. We can get into some more quantitative assessment. Not that we leave behind qualitative, but I think they kind of go hand in hand and also as a progression forward. And some of the things that people were doing that might not have been considered data collection per se by them or others is, you know, simple things with their programming like registration forms or embedded data. So anytime someone's participating in, say, a community garden, they complete a, a registration form, and we have information there. As well, embedded data is things more like tracking of participation. And even things in our food that's produced in, and distributed throughout the community. So things that are, you know, not necessarily a formal survey or, you know, a psycho-social variable, but it's very important to understand what's going on in the community. And then some of the, the most common types of surveys, we're really looking at food access. As I mentioned, that was one of our key areas that we will be focusing on. And so here is an opportunity to look at some of this overlapping types of information and how we can better align and help build the strength of what that data is being collected. And so next slide, Kathryn. They'll be a couple clicks. I think you can just put them all up there. So, you know, there's vary, various reasons why groups are collecting data, and I think they're all important, and they all relate, these and others would relate to Shared Measures. So people are interested in knowing does the program work, and then they can tailor at their practices and approaches better understanding your community and population. What are the barriers to food access, say, and then you can design a program around that. And also the need to have to report out to stakeholders like funders and policy makers and being able to demonstrate success so that you can further continue to support that programming. So those don't have to happen in siloes. They can really feed in and relate to Shared Measures, and you can see along the bottom where we are in terms of progress towards our won Shared Measures idea, and, you know, we've sort of built through identifying these existing measures where the gaps and overlaps and working on capacity and support across the organizations and thinking really strategically on how we can pilot something small that could then be disseminated more widely and then have future iterations that may pulled in other new areas beyond save food access. And thinking along the way how this aligns with what's going on across the state and nationally. So just keeping everyone on, you know, up to date on what are some of the best practices that, and keeping it also extremely useful and meaningful at the local level. So that's all for that piece. I can pass it back to Kathryn. 

Kathryn Colasanti: Alright. Thank you, Courtney, and just one technical detail I forgot to mention when I did the introduction. If you have questions along the way, go ahead and take that into the, the chat box, [inaudible] make available for everyone to see or just for the panelists, and we'll moderate that as we go [inaudible] see those questions for the end. So I'm going to turn it over to Sue Ann Savas for an overview of program evaluation.

Read the full Overview of Program Evaluation Webinar Transcript