In my last post (#4), I put out a call for evidence that the poor reduce their costs (explicit and/or implicit) by going to microfinance providers. This request begs the question: What constitutes good evidence? What are we supposed to be looking for?
Regarding the debate between those who seem to insist we can know nothing about impact without RCTs vs. those who seem to reject the utility (even the validity) of RCTs for informing policy and practice, I conclude that RCTs are essential to anchor our knowledge base in good science, to give us confidence in what we think we know. I also conclude that RCTs can’t be helpful to practitioners and those who support us without supplementation by other methods of evidence-gathering and interpretation.
At the Opportunity Collaboration last October (2011), I had refreshing conversations about mixed methods with Kat Rosqueta of the Center for High Impact Philanthropy at the University of Pennsylvania and Melanie Moore, founder and leader of See Change in San Francisco. Refreshing because they stand outside the relatively narrow worldview of those who focus on microfinance. Kat’s group summarizes the mixed methods approach very well:
To meet our goal of providing smart, practical guidance to donors who care about impact, we synthesize the best available information from three domains: research, informed opinion and field experience. We believe the most promising opportunities exist where the recommendations of these three domains overlap.
I’ve never read a more succinct and sensible statement of evaluation methodology. For the purposes of this Evidence Project, however, I want to reframe the three domains slightly. “Research” is what we learn from RCTs and similar experimental-design field research methods; and “informed opinion” is what comes from non-experimental observation and correlation methods used by program evaluators and even practitioners. “Field experience” means the subjective knowledge or intuition that comes from years of laboring in the vineyards of international development. All three domains contribute toward building our confidence in what we think we know.
Melanie Moore is developing “story science,” which resonates with Freedom from Hunger’s project to develop a rigorous method of collecting in-depth life stories from randomly selected microfinance clients and then following up with the same people about three years later (see our preliminary report, Human Faces of Microfinance Impact). This is not unlike the case study methodology that dominates the business schools, but at the level of individuals (who are more easily selected at random and in statistically powerful sample sizes than are companies or programs). These and other efforts reflect a convergence of experimentation toward more rigorous, credible methods to generate “informed opinion.”
Story science and other objective methods of observation and correlation increase or temper our confidence in what field experience leads us to think we know about the impacts of microfinance and related services.
However, even rigorous, credible methods of observation and correlation do not serve to demonstrate what causes or doesn’t cause the results we see. We can do one excellent study after another that shows a strong correlation between being a microfinance borrower and being successful in starting or growing a microenterprise. But without controlling for “self-selection bias” and “program placement bias” and other ways we allow our minds to be misled, we never show that borrowing itself causes better business. We need a field experimental design that allows valid comparison of those who would borrow but can’t to those who would borrow and can (and do)—a randomized controlled trial, or RCT.
Without an RCT that shows causality, we can still have some level of confidence derived from observations and correlations that mostly point toward borrowing as a cause of better business. With an RCT, even better with two or more RCTs that point in the same direction as our field experience and our informed opinion, our confidence can soar. But not too high! It takes just one new study that contradicts what we think is true to send our confidence plummeting—as it should, if the study is valid, both internally and externally. We have to “know” with humility about the prospect of being wrong.
With this clarification of what kind of evidence is sought, let’s return to Theme Two and look into the proposed causal pathway from a microfinance institution providing financial services to groups of poor women to decreased costs of borrowing and saving to poverty reduction in the households of these women. This pathway will serve to illustrate the method of this Evidence Project and to explore what may seem like a self-evident truth. But is it?
Does microfinance actually reduce the costs of borrowing and saving when compared to informal, traditional opportunities to borrow and save? Please share your evidence, in the form of experience, stories, and studies you consider useful.