3 tips and tools to ensure evaluation is actionable and responsive to shifting community needs

It’s no surprise that the last few years have created a new sense of urgency for using data to respond to shifting needs and goals for many communities. Like many other organizations, our partner Opera Philadelphia pivoted their services during COVID-19 by adapting their youth programming from in-person to online. But even before the pandemic, our team at ImpactED has partnered with the Community Initiatives Department at Opera Philadelphia to develop and implement an evaluation strategy that centers participant voice and generates actionable insights that inform learning and program improvement.;

At the core of this work, the Opera Philadelphia team has been focused on making their evaluation processes (from data collection to analysis to reporting) meaningful to their stakeholders. This has entailed building an online dashboard that summarizes program outcomes, strengths, and areas for improvement in a way that is accessible and understandable for internal staff to make meaning and identify strengths and opportunities. The team then utilizes the dashboard to pull out key findings, create data visualizations, and generate communication materials for other stakeholders like board members and teaching artists. Routinizing this process of updating and sharing information has helped Opera’s stakeholders see the value of data collection and increased their interest in investing in it.

Below are 3 important tips that will support your organization in generating actionable insights that prioritize learning:

1. Identify the data that will be most meaningful to key stakeholders.  Opera Philadelphia collaboratively built a theory of change to ensure shared understanding of program goals across their key stakeholders. They engaged a range of staff members within the Community Initiatives department as well as staff from Opera Philadelphia’s development team to align their goals with any reporting requirements to funders and ensure it would support internal learning. By establishing this shared understanding early on, Opera Philadelphia was able to tailor their data collection strategies to meet their learning goals with their staff capacity Ask yourself: What data matters most and how will stakeholders be able to use that data to advance equitable outcomes?
2. Ensure you have the capacity to collect quality data.  Data collection is time consuming but can be extremely valuable when the necessary time and resources are dedicated to the process. Opera Philadelphia was deliberate about building time into existing structures, like team meetings, to discuss their key findings and incorporate staff member input into the process. Additionally, they assessed staff time and clarified how data collection and analysis are built into team roles, so that they could balance bandwidth with the scale of their data collection activities. Ask yourself: Is there useful data available on a timely basis and if not, can you develop ways to collect it that are not burdensome to staff?
3. Consider how the data will resonate with a broad range of stakeholders.  Data collection can often result in an overwhelming amount of information, making it challenging to identify and prioritize what data matters most. This makes it critical to identify the audience—or often, multiple audiences—for your evaluation work. Opera Philadelphia addresses this by conducting regular meaning-making sessions where stakeholders can reflect on what findings resonate, discuss emerging questions, and explore how results can inform adjustments to the program. While each stakeholder group has their unique perspective, Opera Philadelphia’s goal is to work towards developing a collective understanding of impact. Involving stakeholders in making meaning from results in this way can help your organization surface the most significant takeaways and determine where to dig deeper. Ask yourself: Will the data be accessible and understandable to your stakeholders, and in particular, the individuals and communities you serve?

Below are 3 tools to support your organization in getting started with this process:

1. Dashboard Development Resource - This blog from Ann K. Emery describes how to identify what kind of dashboard would work best for your audiences and common mistakes to avoid in development. 
2. Stakeholder Mapping Resource - This Mural article discusses the purpose and benefits of stakeholder mapping and provides practical examples and templates for each step along the way.
3. Creative Ways to Solicit Stakeholder Feedback - This Public Profit toolkit offers a variety of quick start guides that provide advice and tools for soliciting feedback from different types of stakeholders.

We want to hear from you! How is your organization thinking about generating actionable insights to respond to your community's needs and goals? Email us at impact-ed@sas.upenn.edu or tag us on social media (@ImpactEDphl) to share your perspectives and learnings.

*This blog post is a part of our 5 Insight Series. Each month we’ll be diving deeper into one insight from our 5 Insights for Accelerating Social Change and sharing additional examples and actionable strategies to support you and your organization. If your organization is wrestling with similar issues and interested in partnering with ImpactED, please reach out to us at impact-ed@sas.upenn.edu.