ARTICLES & ESSAYS

Because we in the social sector have been trained to believe that smart decisions thrive on evidence, it’s easy to assume that the act of gathering evidence will make the difference between a poor decision and a wise one. Yet one of the most striking and consistent findings from research with foundations, policymakers, and nonprofits is that even when evidence is available, it is rarely used. Why is it so hard to apply evaluation and research findings to philanthropic decisions? Why does evidence so often seem to generate more questions than answers? How can social sector leaders better bridge the disconnect between knowledge and action?

Past literature on the use of evidence in philanthropy has noted several barriers to its greater adoption, including a lack of time, technical training, and political motivation among the target users. But the gap is also the result of factors relating to organizational design, misconceptions about the role and value of data, and a failure to prioritize decision-relevant questions. In this feature article from Stanford Social Innovation Review's spring 2020 issue, Julia Coffman, Tanya Beer, and I outline these less frequently acknowledged reasons why evidence so often falls short of its potential, and detail five ideas to help social sector leaders use evidence more effectively.

Like many of you, I saw my world change dramatically in what seemed like the blink of an eye back in mid-March. Within a span of a long weekend, we went from dropping off my older child at preschool as normal to the governor of my state ordering all restaurants and other gathering spaces closed by nightfall. As social sector leaders, we must remember that behind each change of this nature is a policy decision. For every state or municipality that implemented (or didn’t) a stay-at-home order, for every office that shut its doors (or didn’t) before it had to, for every event producer that canceled its programming (or didn’t) out of an “abundance of caution,” there was a person or a group of people who considered and made that choice. For several years now, I’ve been arguing to anyone who will listen that the decisions of social sector leaders sometimes carry enormous consequences. While choosing whether and when to order a lockdown is a particularly dramatic demonstration of that principle, the decisions facing philanthropists, government agencies, and nonprofits in the days and weeks ahead are similarly fraught with both immense import and great uncertainty.

How has your life been shaped by the decisions of government officials, donors, and nonprofit executives? If you're anything like me, probably in too many ways to count. I think that’s why we all got into this work in the first place: to make a difference in people’s lives. But it took me a while to realize that the difference we make and the decisions we make are one and the same. Better decisions foster a legacy we can be proud of.

So how can we get better? Lots of ways! Decision-making is an exciting frontier in social impact precisely because there is so much untapped potential to improve how we do things. For decades now, scholars and practitioners have been pioneering methodologies for analyzing and making decisions that have seen barely any adoption in philanthropy, government, or impact investing. Drawing from that body of work, I’ve developed my own process to help myself and my collaborators make critical decisions with intention and focus. Here, I’m sharing the key elements of that process so that you can benefit from them in your own practice.

Decision-making is life. Over time, our decisions carve an identity for ourselves and our organizations, and it is our decisions, more than anything else, that determine how we are remembered after we’re gone. Despite their importance, though, we barely pay attention to most of the decisions we make. Biology has programmed in us a powerful instinct to make decisions using our intuitions rather than our conscious selves whenever possible. There are good reasons for this; if we had to think about every little decision we made, we’d never get anything done. But complex decisions require us to compare the likelihood and desirability of many possible futures on multiple, disparate, and often conflicting criteria, something our intuitions just aren’t naturally equipped to do.

Thankfully, there is a better way. The secret to resolving complex, risky dilemmas with justified ease and confidence is to model your decisions explicitly. At its best, modeling our decisions can help us make the very human exercise of decision-making not only more likely to lead to the outcomes we want, but more instinctively satisfying as well.

We've all been there. The time when the client seemed to forget the project ever happened as soon as the final check was cut. The time when your report stuffed full of creative recommendations got buried by risk-averse leadership. The time when stakeholders really did seem engaged by the findings, had lots of conversations, and then...nothing changed.

If you suspect these stories are more the rule than the exception, the evidence suggests you're right. And if the trend continues, chances are it's eventually going to catch up to those of us who generate and spread knowledge in the social sector. If we really want our work to be useful, we have to continue supporting decision-makers after the final report is delivered, working hand-in-hand with them to ensure whatever choices they make take into account not only the best information available but also other factors that matter to them, including their values, goals, and perceived obligations. For this reason, knowledge providers who want to see their work have greater impact might find value in partnering with a decision consultant in the form of a "wrap-around" service for knowledge initiatives.

In 2015, the Center for Evaluation Innovation and the Center for Effective Philanthropy surveyed evaluation and program executives at 127 US and Canadian foundations with at least $10 million in annual giving. The result was the report “Benchmarking Foundation Evaluation Practices,” and it contains one of the most amazing facts I’ve encountered in nearly two decades of working in the social sector. In response to a question about the challenges they encounter in their work, more than three-quarters of respondents said they have a hard time commissioning evaluations that yield meaningful insights for their own foundations. To put that in plainer language: most of the folks who commission evaluations of social programs have trouble getting people — including even their own colleagues — to use them.

This is a remarkable finding, but you could be forgiven for wondering how much we can conclude from just one study. Sadly, there (ironically) is plenty of evidence to reinforce the point that people with influence over social policy simply don't read or use the vast majority of the knowledge we produce, no matter how relevant it is. What's really astounding about this is how much time, money, attention we spend on evidence-building activities for apparently so little concrete gain. We are either vastly overvaluing or vastly undervaluing evidence. We need to get it right, because those are real resources that could be spent elsewhere, and the world is falling apart around us while we get lost in our spreadsheets and community engagement methodologies.

Measurement is not a simple act of observation disconnected from any larger plan. Instead, it’s an optimization strategy for reducing uncertainty about decisions we need to make. That’s the central argument of Douglas Hubbard’s How to Measure Anything: Finding the Value of “Intangibles” in Business, which remains one of the most important books on decision-making I’ve read since first encountering it more than seven years ago. This revolutionary reframing argues that measurement can only have value if it can reduce uncertainty about a decision that matters. It points toward an ultra-applied approach to evaluation and research that would represent a radical departure from the way these functions operate at most organizations today.

As a strategy consultant, I spend a lot of time working with professional staff members at foundations. Many of these thoughtful people have developed robust approaches to help them understand the landscape in which they operate, articulate clear goals for their grantmaking, and gather relevant evidence to help them track their progress. All too often, though, the work we can do together is fundamentally constrained by decisions that were made much earlier in the life of the institution by different people and under extremely different circumstances. Foundation staff rarely have decision-making authority over the mission of a foundation, the distribution of resources to each grant-making priority, or who, exactly, the foundation seeks to serve. Decisions like these — decisions about mission, values, and scope of concern — are arguably the most important an organization will ever make, and they are usually made right at the beginning of a philanthropic institution’s life. But how often do these considerations get the kind of attention they really deserve?

To answer that question, we must look beyond the rarefied world of foundation staff members to the even more rarefied world of foundation board members and living donors.

At the heart of any strategy are two questions: what do we want to accomplish? And how are we going to do it?

In many situations, answering these questions might not seem difficult. We may already have a mission statement or set of values that guides all of our actions, addressing the first question. Likewise, we may already have a plan of action in place, a set of activities that seems to match the goals we’ve set out. As intuitive as it is to imagine the beginning and the end of that process, though, all too often the devil is in the details—or more specifically in this case, all of the pesky steps in between. Figuring out what those ought to be takes real work, and is generally not something that can be done in one’s head. And because it takes work, a lot of times we don’t bother to do it.

Fortunately, there is a tool called theory of change that provides a means of figuring out all the steps. A theory of change is a visual depiction of your strategy. You probably already have a notion in your head of what your strategy is, but a theory of change gives you a means of articulating it more clearly to yourself and others who care about your success.

The tagline you see at the bottom of this website is "smarter decisions for a better world." It’s a common assumption that the definition of “smarter” includes something like "based on objective evidence." Yet one of the ironies of the work I do is that there is no non-subjective way to make judgments about what counts as "objective." Some writers and activists go so far as to claim that the concept of “objectivity” itself is racially coded, too often used as a shield behind which historically white institutions hide their culturally determined values and priorities.

Perhaps no writer has taken on this thesis as directly as Patricia Hill Collins, whose 1990 book Black Feminist Thought offers a chapter entitled “Black Feminist Epistemology." She writes, "far from being the apolitical study of truth, epistemology points to the ways in which power relations shape who is believed and why." Collins is no apologist for Western scientific paradigms, and yet I came away from her work with the impression that it is less a bid to discredit objectivity, and more a bid to hold it to account.

It has been almost a century since the great W.E.B. Du Bois–one of the co-founders of the NAACP–offered his stirring call for what, today, we would call “cultural equity.” And yet, read Chang–and below, Campbell, Lowry, Rosen, Erickson, Kourlas, Sonntag, Vega, and Grams–and one thing soon becomes clear: “cultural equity” means different things to different people. Over the course of some ninety years, distinct, and sometimes competing, visions of success have jostled for attention, complicating a complex conversation, and creating tensions that often go unresolved. Diverse goals and desired outcomes–over time and between different groups–have made change a juggling act; meanwhile, efforts to add on fixes to a system that was not built with equity in mind have met with mixed results.

That there are different visions for cultural equity is clear. Where exactly the lines are drawn, however, is somewhat less so. There is an inherent difficulty in examining positions forged through dialogue via documents authored by a few, and any attempt to develop a taxonomy will have its flaws. But in our own conversations, we found it helpful to divide the visions for success we were reading and hearing from advocates into four archetypes: Diversity, Prosperity, Redistribution, and Self-Determination. In the rest of this article, we present the differences between these visions, and consider the implications for a healthy arts ecosystem.