By Barbara J. Poppe, USICH Executive Director
I think of myself as an activist at heart with a head for data. For more than 30 years, I’ve combined activism and data to focus on getting results for those who have no home. By training, I am an epidemiologist – I see the world through a public health lens. Everything can be measured, and data can be harnessed for good – prevention, early intervention, treatment, and to create better public policy.
Why use data?
Baseball, beyond being an athletic competition, is also a game of numbers. Measurements. Statistics. If you want to know Red Sox right-fielder Shane Victorino’s on-base percentage against right-handed pitchers in the night games of the 2013 World Series, that statistic exists.
If only the data was that robust in our fields.
I’m sure many of you recall the book “Moneyball: The Art of Winning an Unfair Game,” by Michael Lewis, or at least the movie starring Brad Pitt. It’s the true story of Oakland A's general manager Billy Beane's successful attempt to assemble a winning baseball team on a lean budget by using unconventional data analysis—called Sabermetrics—to evaluate athletes. In an “unfair game” Billy Beane leveled the playing field against richer teams by using better data and better analysis to determine more accurate indicators of a player’s potential.
Here, in another unfair game—one without big budgets and profits, but with real bottom lines and consequences—we can all learn something about looking at relevant data to identify the factors that are the most critical to progress.
What Data Should I Use?
Just like baseball statisticians, we could collect an extraordinary amount of data. Everything can be measured.
Data is a great tool in our performance management tool box, but only if it is used appropriately. To me, the primary purpose of using data is to drive results. It also has the additional benefit of increasing your credibility with the business community, meeting requirements for your funders, and impressing potential donors.
The three guideposts that have served me well in determining what data to use are:
- Go for the highest purpose
- Fewer indicators are better than too many, and
- Synergism helps
Billy Beane found that in order win on a smaller budget, he would need to think in terms of buying runs and wins, and not in terms of buying fancy baseball players. Beane knew the win was the higher purpose.
When I came into office at USICH, I heard over and over again that VASH was being under-utilized. Housing vouchers had been made available, but Veterans weren’t being served.
VA, HUD, and USICH worked to solve the problem by requiring grantees to move the needle on two key performance measures: the speed of lease-up and the voucher-utilization rate. Local organizations created plans that focused on process improvements.
They also collected data, analyzed it, and reported on results. The expectation was that by making the process easier and with fewer steps to navigate, more Veterans experiencing chronic homelessness would be served. Unfortunately, the opposite seemed to occur. Though the speed of lease-up increased as processes were shortened and the proportion of units leased increased dramatically, the number of chronically homeless Veterans served by the program wasn’t increasing in proportion. Why?
With the focus solely on speed, it was easier to serve Veterans who did not have long histories of homelessness than to take the time necessary to find Veterans living on the streets. The pressure to meet the key metrics drove providers to reach the easiest to serve not the hardest to serve.
So what happened? We changed the key performance metric to be the proportion of chronically homeless Veterans served. What happened next? We starting winning more; we started serving the harder-to-reach Veterans.
Fewer is Better
Paul DePodesta was the analyst behind Billy Beane’s new approach to managing a baseball club. He found a way to reduce everything the data could tell him about the value of an individual ball-player into in just one number, thereby giving the Oakland A’s the ability to find value in players no else could see. The lesson here is fewer indicators is better than too many.
We data geeks want to measure everything. We want to run endless reports and consider all the ways the data can be cut. Unfortunately, this has two major negative consequences. First, it costs money to collect, analyze, and report. Second, it takes time. We know that time is money, but beyond that, it also delays action. Too often the call for more analysis is simply a delaying tactic. If we are trying to drive change, we need to use the data to make decisions—these could be course corrections, acceleration of promising strategies, or trumpeting good results to raise more funding.
Billy Beane and Paul DePodesta were able to reduce player-value to one number, and baseball—with all its rich data and statistics—ultimately has just one measure of success: games won. But there is a big differencebetween what we do and what baseball teams do…we don’t play baseball. Our measure of success is more nuanced and complex than what can be displayed on a score board. So, I’ll say that while too many measures can be overwhelming and counterproductive, relying on a single measure can lead to unintended consequences.
In the mid 90’s, when I joined the Community Shelter Board, we decided to move to using outcomes as part of our grantmaking process. The most important measure of success for a program that is addressing homelessness is to end homelessness for that family or individual – that is, to have a successful housing outcome. What we didn’t want was for the programs to only admit/enroll households who were most likely to be successful— i.e. cream—and we also didn’t want housing placements that weren’t for the long term— i.e. family is evicted because the housing isn’t affordable. So we created interlocking measures that represented the components of success:
- A successful housing outcome was one that not time limited, and the household had control (i.e. a lease).
- The household was quickly re-housed—that is the length of time from admission to exit was short.
- The household did not become homeless again— recidivism was measured by return to any homeless program.
- Additionally, the program was fully utilized.
All of these measures are fairly easily captured by the Community Shelter Board data system and can be measured at the program and system level.
The best performing programs met all the measures, and we had reasonable confidence that these programs were meeting both client and community needs.
If programs weren’t meeting some of the measures, we could dig into the area of concern and discuss how to improve performance in that area without compromising the positive result area. Data is just one tool in the toolbox. You need to use the other tools in your toolbox—collaboration and partnerships, a feel good anecdote about one of your successful program participants, advocacy messages, and the business case for your effort. Data is powerful; use it wisely.
P.S. This blog was crafted from a speech I gave on September 27, 2013. On October 18, 2013 as many of us came back from furlough, I learned that Moneyball for Government went live. Check it out – it’s bipartisan as well as pointing the way for the federal government to avoid a future budget crisis. What’s not to like about that?