Evaluating and Optimizing Your Nonprofit Programs, with Allison Shurilla

[00:00:05.030] – Intro Video
Welcome to The Nonprofit Hero Factory, a weekly live video broadcast and podcast where we’ll be helping nonprofit leaders and innovators create more heroes for their cause and a better world for all of us. Da Ding!

[00:00:21.670] – Boris
Hi, everybody. Welcome to another episode of The Nonprofit Hero Factory. We talk a lot about storytelling. We talk a lot about donor engagement, and we talk a lot about technology. And today’s guest is actually at the intersection of all three of those I think, in that she helps organizations figure out what is working and what is not working within their programs so that they can then apply it to their storytelling, their technology and their programs in general and create better connections with donors, but also deliver more value.

[00:00:52.730] – Allison Shurilla
So I’m going to bring her on in a second, but let me tell you a little bit about Allison Shurilla. She is the founder and lead consultant of AS Community Consulting, where she helps community organizations build a culture of evaluation and integrate evaluation into their organizations so they can learn about their work, to do it better, and have the greatest impact. AS Community Consulting supports community organizations in building evaluation culture through consulting, coaching, and leading trainings and workshops. And that’s kind of what I’m going to have her do today. Her superpower Allison describes is connection, both in terms of connecting people and ideas.

[00:01:26.660] – Boris
And with that, let’s connect with Allison and bring her onto the show. Hi, Allison.

[00:01:31.760] – Allison Shurilla
Hi, Boris. Thank you for having me here.

[00:01:34.470] – Boris
It is absolutely my pleasure. We’ve been talking about having you on for a while now. So I’m excited that you are finally here, and I’m ready to pull as much information as I can out of you for all our nonprofit heroes at home or at work or in their cars, wherever they’re watching or listening to this show. Don’t watch and drive, it’s bad. But before I do, before we dive in, Allison, tell us a little bit more. What is your story? Why do you do what you do today?

[00:02:00.310] – Allison Shurilla
Yeah. So my background is actually in education and youth work. I started out as a nonprofit professional, and I decided that I wanted to do something at not the ground level. I originally thought I wanted to be doing policy work, and I got a Master’s in Public Policy, and I thought I wanted to be doing research. And I ended up finding evaluation while I was in policy school and while I was kind of pursuing learning about research. And evaluation really struck me as the thing that could have a really huge impact by helping people and organizations use the information that they have to make decisions to improve their program, to have the impact that they ultimately want to have. And ever since I learned what evaluation was, I have been running with it and trying to find the best ways to do it and to find the best way that it can be helpful for organizations and really help them do what they are here to do.

[00:03:00.550] – Boris
Awesome. So many of us started out in the nonprofit space on the inside and then realized that we could hopefully have a bigger impact and help more organizations do more. It’s a common transition. And I appreciate that you have taken on because it’s not easy going out and suddenly opening up your own shop, if you will, trying to get your message out there. But you know that it’s important. You know that it’s helping organizations. So I, for one, appreciate what you’re doing, and I’m sure your clients do, too.

[00:03:30.930] – Boris
But let’s talk about what is the problem that you’re solving. And let me start by asking you what’s happening right now? Maybe things have changed since the pandemic began. Maybe not. But what’s happening right now in the nonprofit space, from your point of view?

[00:03:45.970] – Allison Shurilla
I think there’s still a lot of uncertainty and there’s still a lot of questions. Like uncertainty is a word that we still hear every day. People are still talking about whether or not they’re going to do programs online or in person or how they’re going to navigate changing or what’s going on with the populations that they’re working with and what’s going on in their communities. And there’s just a lot of questioning. And evaluation ultimately helps to answer those questions. That’s why I’m here to do what I do. So I’m finding that everywhere from what’s going on with the participants in our program, what’s going on with their lives, how can we help them? How do we optimize our program so that we’re serving them the best way? To what is the impact we’re having anyway? We can’t tell because things don’t look the same way they used to.

[00:04:34.270] – Boris
I definitely can see that. And especially since the move to digital, the great jump into digital that everybody had to take, I feel like a lot of organizations did things based on instinct or reactionary, which was necessary, and I’m not judging any of them for doing it. But they may not then realize what the effects have been or they’re not sure how to look at it. Is that the kind of thing that evaluation is really there to help them with?

[00:05:02.230] – Allison Shurilla
Absolutely. I think that evaluation sometimes gets put in this box of surveys or reporting or just looking at your numbers. But it’s really there to help you answer those questions. And when things change or you know things are going good and you want to figure out what exactly is working, evaluation is here to help you dig into that and really answer those questions.

[00:05:25.390] – Boris
So what’s the problem with the way that, let’s say most or the organizations that don’t incorporate evaluation into their lives, into their work lives? What’s the problem with the way that they’re making decisions today?

[00:05:38.470] – Allison Shurilla
I think that it is lacking a specificity and the type of information they’re getting in real time. So program developers, executive directors, whoever they may be, they’re making their decisions based on a lot of knowledge, a lot of experience. Maybe they went to school for something. Maybe they’ve been living in the community they’re working in, and they have this really in-depth knowledge about what goes on or how to design a program. But they’re not necessarily getting regular feedback about what’s happening at this moment or what’s happening in real time or if something like a pandemic comes along, what you do now, because everything you know, it’s not the way that things have become. And so the problem is that if you don’t have an evaluation system that’s helping you look at things as they change or even just look at things in general, you’re kind of just running on guesses based on maybe previous knowledge, based on academic knowledge or community knowledge, which are all important. But evaluation gives you that extra real-time piece.

[00:06:42.910] – Boris
I find that that happens a lot in organizations. I work with their storytelling all the time, obviously, with their communications and marketing. And I’ll check out their websites. And there’s a lot of inside the bubble speak, right? We’re inside our organizations. We’ve been in them sometimes for 20, 30 years. So we know everything so well. And we assume that all of that gets communicated. Similarly, we know everything so well we assume that we’re doing the right thing. We’re doing the best thing without necessarily having that evaluation or testing our hypotheses or to see whether or not there’s a better way or something else that we can optimize.

[00:07:23.110] – Boris
So then I guess what’s the solution to that? You mentioned surveys, and that’s what a lot of people think when it comes to evaluations. Maybe we should start with what does it mean? What does evaluations mean in the sense that you’re using it? And what does it mean for nonprofits in general?

[00:07:40.810] – Allison Shurilla
Yeah. So kind of along the previous conversation that we were having, I was having a conversation with a group of evaluators recently, and we find that we’ll come into an organization and they’ll ask us for the answer. They want to know what the best way is to develop a program and they want to know what the best practices or they’re using what is known as the best practice. And we as evaluators are in real time creating the best practice.

[00:08:06.090] – Allison Shurilla
So in the organization, we will come in and help you look at talk to people in your community or talk to your staff to harness that knowledge. So all of those things that we just talked about, the best practices, the going to school, the knowledge of the community, evaluation kind of takes that and puts it together and looks at it to answer very specific questions so that you can use that information to make changes or to increase efficiency in your organization; to find out what it is about your program that’s really having the biggest impact; to find out what your community really needs so that you can be serving them to the best of your ability.

[00:08:47.890] – Allison Shurilla
And then it also gives you tools to communicate that, so you can communicate that to your funders, to your donors, to the community itself. If you want to get more people coming to your program, you can use that as evidence of saying, “Hey, come here, look at this great time all these people are having.” Or, “This is how it’s influencing people and impacting people.”

[00:09:07.690] – Boris
So it definitely sounds like a very powerful tool to use in our processes, in our systems. I know you talk about this all the time. Is it something that we should be doing on an as-needed basis, or is it something that organizations should be doing on a continual basis hand-in-hand with whatever program they’re delivering or whatever services they’re working on?

[00:09:31.810] – Allison Shurilla
It should be done along with the program. It should be something that’s continuously done as part of the work that you’re doing. So evaluation, in my opinion, is most powerful when it is part of what you’re doing and not a separate add-on. So a traditional evaluation or a way that is very commonly done is that I, as an evaluator, might come in, do a big research-looking study. I might design some surveys, or I might do some interviews or focus groups with people. Then I would write you a nice report that tells you what I think and what I found out.

[00:10:04.570] – Allison Shurilla
I think it can be much more powerful for an organization to be able to build in processes so that they’re looking at data, they’re looking at the information, they’re talking to people and gathering feedback from people in a way. And then using that to make their decisions as they’re trying to develop program changes or applying for a grant or trying to decide what to do for their big event that they’re doing, right? Evaluation is the thing that can give them these little pieces of information that can help them make those decisions with real-time information that’s happening in the community.

[00:10:41.050] – Boris
So then, you mentioned surveys early, and it sounded like you were saying that’s what people associate with evaluation, but that’s not necessarily the heart of it, which I totally understand. But then are surveys then… How do you do ongoing real-time evaluation unless it’s something like a survey? Are there other tools that you put in?

[00:11:03.370] – Allison Shurilla
I talk about surveys a lot because they’re still very common in evaluation. And that’s the thing that I get asked about a lot. And so it comes up a lot, and I witness it happening. Most organizations have familiarity with it in some way. But they can be busy work. They can be giving you information that’s not that valuable. They could be not giving you the right information that you need. And so I deal a lot with conversations and talking to people. A way that I really like to work with organizations is, say that you have something like a youth program, like my background is in working with youth in education.

[00:11:43.810] – Allison Shurilla
We have a youth program that we meet every Tuesday, and so we have this population of kids, right? That we’re talking to our students or young people, whatever language that you want to use for them. And we give them a survey once a quarter. They give their answers. It feels like a test. It doesn’t necessarily feel like something where they can feel like they can authentically engage with the organization or with the program.

[00:12:07.750] – Allison Shurilla
Another option might be to have a conversation that’s integrated with the program, an activity that you design to gather feedback from the students to find out what they think and how they’re responding to the program and what they think would make it work better, how they can get their friends there, maybe how it’s impacting their grades and their relationship to school or to their families or to community.

[00:12:29.000] – Allison Shurilla
Whatever your goal is or your focus of that program, instead of just handing over a piece of paper or an online survey to students, you can have a real conversation with them. And that can be extrapolated to your staff or to parents or to other community members you’re working with. I also work a lot in public health. And if you’re working with patients, how can you really get some authentic, real information from them? That’s not… The survey can be a little dry and a little removed.

[00:13:07.410] – Boris
I see. So whereas a survey feels more designed to collect data, you’re actually trying to be more interactive conversational and almost extract stories from them, but really let them guide some of the conversation as well, rather than just a one-directional or I guess two-directional, but an exchange. It is an actual kind of interactive conversation about the subject. Is that right?

[00:13:37.060] – Allison Shurilla
Yes. And a little bit more about my background is that the methodology that I use is based in story collecting and qualitative methods is what we call them, as opposed to quantitative methods, which are like statistics. And a survey is actually considered a quantitative method. And in community-based participatory research methods, which is a little bit of a big word. But basically the essence of it is that the people that we’re gathering information from have a lot more to give than just a data point. They have a lot more to give than just a one to five on a survey. I’m sorry I’m hating on surveys so much. I don’t hate surveys. I use them. I think they’re very valuable. Please, nobody come for me for hating on surveys, but they can be overused and they can be improperly used. And I see that happen a lot, which is why it’s such a common example that I give.

[00:14:32.670] – Allison Shurilla
And so the type of evaluation that I do and the way that I work with organizations tries to move them beyond that so that evaluation isn’t just a piece of information, but it’s a process, and it is a relationship, and it is something that harnesses both the expertise and knowledge and talents of the people working in the organization as much as the people that are receiving services from the organization or interacting with the organization.

[00:15:06.510] – Boris
So if I’m a nonprofit professional right now, listening to this episode, I’m thinking, okay, that sounds nice, but it sounds super resource intensive. It’s going to take a lot of my time or my staff time. How do you answer that? Is it worth it, first of all? I’m sure you’re going to say yes, but how do you justify all of that time and expense in terms of staff power, to do this kind of work?

[00:15:37.170] – Allison Shurilla
I mean, certainly it can be resource intensive, and it can turn into a very big, comprehensive thing if that’s what you want to do. But it can also be very simple. One of my passions is to work with organizations to make it simpler and to make it integrated into what they’re doing. I don’t want to create extra work. I don’t want to create busy work. We have enough to do. Nonprofit professionals, we’re doing everything, right? And so how can evaluation be something that is a part of all of that? So with the youth program example, we’re already using the program that we have. We’re not developing anything new. We might take that information that we get and use it in a staff meeting to talk to our staff, or we might integrate a couple of minutes of a staff meeting every time to talk about… To go over maybe a dashboard or to talk about evaluation.

[00:16:30.390] – Allison Shurilla
And the way that I also work is in looking at the information that’s actually going to be the most valuable to help you do what you need to do, to have the impact that you want to have, to have the processes that are efficient. Evaluation can actually create efficiencies by finding the things that are working well and the things that aren’t working as well; and using the things that you’re doing every day and just putting a different lens on it. Like, looking at it a little bit differently so that you’re using it as an evaluative process or an evaluative culture so that you can learn from that way and you don’t necessarily need to do a big extra thing.

[00:17:15.450] – Boris
So in the case of your youth example, your youth group, are you talking about at the end of each… I don’t know. Let’s say they do ongoing meetings. At the end of each meeting, they spend five minutes asking for feedback.

[00:17:29.010] – Allison Shurilla
It could look like that, or it could be a dedicated session that you work with them on it. I’ve also worked with youth programs to do what’s called youth participatory evaluation. So they’re actually involved in the whole process and helping you make those decisions and answer questions. And it’s a program in itself. It can be… So it can be a very generative thing for those students as they learn how to look at data and process data and talk to people and writing skills, and they can learn all kinds of things. That’s a very robust example, but … I think in my work, I find that it works different ways with different programs and whatever is going to work for you, it could be as simple as doing something every once in a while, or it could be a lot more comprehensive. I hope that answers your question.

[00:18:23.030] – Boris
It does, it does. I’m still just trying to figure out exactly how much extra resource it might take up. It sounds like what you’re saying is that’s really up to you. It could be as little as you want or as much as you want. And you can make an entire program out of just your evaluation—or at least an event out of your evaluation—where you might discover what it is that you should be evaluating in the first place because you’re going to get input from your constituents.

[00:18:51.710] – Allison Shurilla
One of my goals as a consultant is to help organizations find the sweet spot about how much they want to invest in this and in what way they want to invest in it, right? So do you want to do a big extra project? Those are really valuable and really important sometimes. But do you want to develop a system within your organization so that it’s so seamless you don’t even know that you’re doing it? It’s hard to answer that question because they’re just limitless options, right? It could be very simple or it could be very, very comprehensive. And I think even when it’s simple, it can be very powerful because you’re just starting to look at information a different way that can help you do your work a little bit differently.

[00:19:39.930] – Boris
Okay. So how do you know that your evaluation program is working? Let’s assume that we’ve signed on for this concept of continual integrated evaluation. How do we know that it is working, that we’re asking the right questions? What are some of the results we might see from the process and the work?

[00:20:07.570] – Allison Shurilla
The results that you’ll see is that you will have more confidence. I think is one of them could be that you are making the right decisions instead of being like you’re guessing. I don’t know. Maybe I don’t want to say that. I don’t want to say that people aren’t confident in the decisions that they make. But what you’ll see is that when you are talking to your community or you are working with your community, you know the way that you’re affecting them, you know the way that you’re impacting them—is really powerful, and it’s something that is your unique way of working with them. As opposed to, like, they couldn’t just get it from anywhere else. Because they know that your organization is the one that’s making the difference. Your organization is the one that’s affecting them that way because you have the data to back that up. You have the stories, you have the numbers, you have whatever it is, those things are going to tell you what exactly is happening.

[00:21:02.110] – Allison Shurilla
Another thing you might see that I witnessed in my work is that you can find out what it is about your work that is having the biggest impact, so that you can be dedicating your time and resources to those things that are really having the impact instead of the things that may not be.

[00:21:18.500] – Allison Shurilla
So you might know that you’re having a great impact on your students or on your community. You might know that their grades are doing really well or they’re having great conversations about health with their peers. But you don’t know which part of your program is doing it, right? You don’t know if it’s because they have the one class or if it’s the entire program or if it’s the frequency of the program or if it’s the guest speakers you have in. So you might be dedicating all these time and resources to developing these programs. Evaluation can help you understand what about them is working, and it can help you poke the holes in the gaps that might be not working so well so that you can change those and not be wasting your time doing something that’s not so effective. And you can take that and extrapolate it to anything in the organization: your staff, your work processes, whatever you might have a question about evaluation is going to help you kind of boil it down into what’s working really well, what’s not working, what’s missing so that you can fill in those holes, all those pieces.

[00:22:26.350] – Boris
That sounds pretty great and I think really important. Again, for organizations that… like most organizations that I’ve worked with, for example, go on instinct, they go on experience and they might start a new program. It may or may not work. Then they’ll try something else, which is totally fine and fair. But really knowing why something is working, what parts of it are working, what parts that are not working will definitely help you optimize and decide what to double down on, what to pull back from, so that you can better serve your community ultimately and so that you can empower them better with the tools and the things that they really crave or are responding to in your particular situation.

[00:23:08.230] – Allison Shurilla
Yeah. I mean, nonprofits, the organizations I work with, ultimately, they’re mission driven, right? They have a purpose. They have something they’re trying to achieve. And evaluation helps you know whether or not you’re achieving that, how you’re achieving it, and what you can do to achieve it better, what you can do to do it better.

[00:23:29.130] – Boris
Or as we say on the show, how to create more heroes for your cause.

[00:23:32.210] – Allison Shurilla
Absolutely.

[00:23:33.430] – Boris
So if an organization is not currently doing evaluations, if it’s not currently in a culture of evaluation, within the organization. Where should they get started? How do they start approaching this or thinking about this concept of evaluation?

[00:23:50.290] – Allison Shurilla
Yeah. So my recommendation is always just to start with a question. And when I say that, I mean a really simple question. What is something… If you’re talking to someone in the hallway, you’re talking to a family member or a friend, you say, “I wish I knew this.” Or, “I wonder…” The kind of thing that you’re like, “I wonder about this because I think knowing the answer to that would have an impact on my work.” It would help me be more confident in my work or make changes or be able to do my work to the best of all.

[00:24:20.100] – Allison Shurilla
You just start with a question and then you look at how you’re answering it. And when I lead workshops on this and when I lead people through this process, I encourage them not to think about evaluation. I’m already there as an evaluator. But I’m like, don’t think about it right now. Think about the really organic ways that you’re answering that question. Right? So we talk to people. I talk to the staff member, I talk to people in my community, I talk to my students or my patients or Joan at the front desk. And she told me, right, like, what’s going on? Those are the things that you can take.

[00:24:57.700] – Allison Shurilla
And then once you think about it that way, you can start to drill it down into something that looks a little bit more strategic. So how can you turn that into a more robust conversation, like a focus group or develop a survey? This is when we decide whether or not we need a survey, right? This is when we say, you know what? It actually would be really helpful if we send out a survey to every single person we talk to to ask this question or whatever it might be. That could give you some really great information. And so it can be very simple. And you can start with one question or one issue or one topic and put those little pieces in so that it doesn’t feel like a lot and it’s integrated into what you’re doing.

[00:25:43.810] – Boris
You just brought to mind the known unknown matrix. You’re familiar with that one? Where you’ve got four quadrants of known knowns. What is it? Actually, I’m going to look at it. Known knowns, known unknowns, unknown knowns, and unknown unknowns.

[00:25:59.180] – Allison Shurilla
Yes.

[00:25:59.790] – Boris
Right? And it sounds like and maybe this is where the whole survey thing comes in is a survey can measure the unknown knowns, right? Or the known unknowns. But it can’t measure the unknown unknowns, whereas an evaluation process is going to help you discover the things you don’t even realize you don’t know. And some things that you might know that you didn’t realize you knew.

[00:26:23.770] – Allison Shurilla
Yeah. That’s actually a really good framing. That’s absolutely correct.

[00:26:28.990] – Boris
Free of charge. You can incorporate that into your next workshop. So then it looks like we start from what you’re saying. We start with first, evaluating or no, just is it brainstorming and writing out what is it that we wish we knew? So what are our known unknowns things that we know we don’t really know yet, and then start going deeper and deeper from there?

[00:26:53.300] – Allison Shurilla
Yeah, I think that would be a good way to put it. And if you want to start real small, I have a tool that you can use that you just take like 10, 15 minutes, like just a few minutes, and just sit down and kind of brainstorm some ideas and kind of work through this in your head what it might look like. And it can start with baby steps like that, and eventually it can become something a little bit deeper.

[00:27:19.630] – Allison Shurilla
So you start with, what do I wish I knew? If I could know anything, what would be the answer? What would be the information that I would have? And then you drill it down into very simple things like, what do I know now? And how do I know that? And then you learn how to make it more specific. And ultimately, at the end of all of this, you will have a process where this is so seamless that you’re just, like asking questions. You’re bringing out ways to answer them. You’re answering them. You’re making your program changes based on what it is, and then your senior community flourish because you are such an amazing, efficient nonprofit that you’re doing all your work the best you can.

[00:28:04.330] – Boris
That’s awesome. And I think that’s a great point to wrap up the conversation. But I do want to ask you, if people are interested in learning more about evaluations in general or maybe an example of evaluations done well. Are there any tools or resources, books even that you recommend people might go check out?

[00:28:21.010] – Allison Shurilla
Yeah. So I have my own resources on my website that talk a little bit about evaluation that do help you walk through the process that I just described. I have a workbook and like a one pager that kind of helps you start to think about it.

[00:28:34.810] – Allison Shurilla
One of the resources that I always point people to is called, I believe the title of it is like, “Why Am I Always Being Researched?” And it’s from an organization called Chicago Beyond based in Chicago. And it’s basically like this tool that talks about research, which evaluation is kind of a type of research, and relates it to the human side of it and the person side of it, instead of just seeing people as ways to get data or ways to get information or to extract information from. It’s a very long kind of comprehensive thing. But if you look at it on a very basic level, it does help you put into perspective what evaluation can look like that isn’t just a data point. And I just love the resource because it’s one of the bases for kind of how I do my work.

[00:29:27.430] – Boris
Fantastic. We’re going to link to that and to the resources you have on your website, because I have checked them out. I like them. They’re a great, very simple framework to just start thinking and brainstorming around these topics and then to hopefully take some actions to implement things. So we’re definitely going to link to all that. Do you have any other calls to action for our audience? How should they connect with you? What should they do once they’ve finished listening to this episode and wanted to follow up with Allison Shurilla?

[00:29:55.450] – Allison Shurilla
Yeah. So you can go to my website, you can send me an email, get in touch with me that way. I do free consultations. And there is also a link in my website that you can go ahead and just schedule that directly with me. And I have a newsletter that you can sign up for and a blog. And so you could sign up for my newsletter, keep in touch with the kind of things I’m working on, what I talk about. If some of these ideas are interesting to you, but you want to hear a little bit more about them, that’s a great way to just kind of follow me. So that you know what I’m doing and what I’m talking about and what I think about some of these things.

[00:30:32.290] – Boris
Awesome. And I do encourage people to go and do that. Check out Allison’s site, get those resources, and then book a call, spend some time picking her brain for free to figure out what it is that you could be getting from evaluations and how you can conduct evaluations to better optimize all your processes and ultimately your impact on the world.

[00:30:55.690] – Boris
Thank you, Allison, so much for joining us today and breaking down what is really a difficult topic to just wrap around. But I think we’ve really gotten to a point where hopefully if people don’t fully understand it, didn’t fully understand it before, that they get a really good idea of it now and all the benefits that it can provide them. And I appreciate you taking the time out of your schedule to share that knowledge with us and how to start implementing a culture of evaluation or thinking about evaluation within our own programs.

[00:31:22.990] – Allison Shurilla
Alright. And thank you for having me. It’s been great to talk to you today.

[00:31:26.540] – Boris
Awesome. Thank you, everybody who is watching, listening or reading the transcript of this episode, be sure to check out our show notes at nphf.show for all of the takeaways from this episode and all of the resources that Allison has shared with us. We’re going to, of course, link to them right there on our site. And if you like this episode, please do share it with your friends. And please leave us a review on iTunes, Spotify, or whatever your favorite platform is, YouTube as well. We do, of course, have the show live there every time too. Thank you everybody. Have a great week. We’ll see you soon.

[00:32:00.250] – Intro Video
Thank you all for watching and listening to the Nonprofit Hero factory. We hope this episode has given you some ideas and strategies for creating more heroes for your cause and a better world for all of us. Please be sure to subscribe to this show on YouTube, Facebook, iTunes Spotify or your favorite podcast chat platform and let us know what you think by leaving a review.