What constitutes good evidence?

In my last post (#4), I put out a call for evidence that the poor reduce their costs (explicit and/or implicit) by going to microfinance providers.  This request begs the question:  What constitutes good evidence?  What are we supposed to be looking for?

Regarding the debate between those who seem to insist we can know nothing about impact without RCTs vs. those who seem to reject the utility (even the validity) of RCTs for informing policy and practice, I conclude that RCTs are essential to anchor our knowledge base in good science, to give us confidence in what we think we know.  I also conclude that RCTs can’t be helpful to practitioners and those who support us without supplementation by other methods of evidence-gathering and interpretation.

At the Opportunity Collaboration last October (2011), I had refreshing conversations about mixed methods with Kat Rosqueta of the Center for High Impact Philanthropy at the University of Pennsylvania and Melanie Moore, founder and leader of See Change in San Francisco.  Refreshing because they stand outside the relatively narrow worldview of those who focus on microfinance.  Kat’s group summarizes the mixed methods approach very well:

To meet our goal of providing smart, practical guidance to donors who care about impact, we synthesize the best available information from three domains: research, informed opinion and field experience. We believe the most promising opportunities exist where the recommendations of these three domains overlap.

I’ve never read a more succinct and sensible statement of evaluation methodology.  For the purposes of this Evidence Project, however, I want to reframe the three domains slightly.  “Research” is what we learn from RCTs and similar experimental-design field research methods; and “informed opinion” is what comes from non-experimental observation and correlation methods used by program evaluators and even practitioners.  “Field experience” means the subjective knowledge or intuition that comes from years of laboring in the vineyards of international development.  All three domains contribute toward building our confidence in what we think we know.

Melanie Moore is developing “story science,” which resonates with Freedom from Hunger’s project to develop a rigorous method of collecting in-depth life stories from randomly selected microfinance clients and then following up with the same people about three years later (see our preliminary report, Human Faces of Microfinance Impact).  This is not unlike the case study methodology that dominates the business schools, but at the level of individuals (who are more easily selected at random and in statistically powerful sample sizes than are companies or programs).  These and other efforts reflect a convergence of experimentation toward more rigorous, credible methods to generate “informed opinion.”

Story science and other objective methods of observation and correlation increase or temper our confidence in what field experience leads us to think we know about the impacts of microfinance and related services.

However, even rigorous, credible methods of observation and correlation do not serve to demonstrate what causes or doesn’t cause the results we see.  We can do one excellent study after another that shows a strong correlation between being a microfinance borrower and being successful in starting or growing a microenterprise.  But without controlling for “self-selection bias” and “program placement bias” and other ways we allow our minds to be misled, we never show that borrowing itself causes better business.  We need a field experimental design that allows valid comparison of those who would borrow but can’t to those who would borrow and can (and do)—a randomized controlled trial, or RCT.

Without an RCT that shows causality, we can still have some level of confidence derived from observations and correlations that mostly point toward borrowing as a cause of better business.  With an RCT, even better with two or more RCTs that point in the same direction as our field experience and our informed opinion, our confidence can soar.  But not too high!  It takes just one new study that contradicts what we think is true to send our confidence plummeting—as it should, if the study is valid, both internally and externally.  We have to “know” with humility about the prospect of being wrong.

With this clarification of what kind of evidence is sought, let’s return to Theme Two and look into the proposed causal pathway from a microfinance institution providing financial services to groups of poor women to decreased costs of borrowing and saving to poverty reduction in the households of these women.  This pathway will serve to illustrate the method of this Evidence Project and to explore what may seem like a self-evident truth.  But is it?

Does microfinance actually reduce the costs of borrowing and saving when compared to informal, traditional opportunities to borrow and save?  Please share your evidence, in the form of experience, stories, and studies you consider useful.

 

  • http://www.bottombillionfund.org Tom Coleman

    How do we move from anecdotal measures of client success to measures of what percent of MF clients are “clearly” better off or better yet “leaving extreme poverty” or becoming “measurably less poor”.

    Love the 3 domains of research together with the humility to acknowledge that we may be wrong, even with all three.

    I listened to an interesting forum with the founders of Groupon and other expert practitioners of social media that are making billions together with some academic researchers on the benefits of new technology.

    The practitioners were clear that social media and the internet that makes it possible are already producing enormous social value as evidenced by market participation, the money their companies are making, and the price investors are willing to pay.

    The academic researchers said that there was still not rigorous and definitive research that proves incremental value from the internet.

    Rather than expecting to completely answer and quantify benefits of MF, perhaps we need to keep looking at all the research more as a general guide to our direction and decision making and a partial guard rail to keep us on the path.

    This assumes we have a well defined path or social (or financial) goal–destination that we are pursuing.

    The lack of specificity in microfinance about what is meant by “poverty” remains a major obstacle.

    One way to help address this is to create a more detailed segmentation of poverty, e.g. divide developing country income into 4 quartiles–
    4th quartile Bottom Billion–1.4 billion people living under $1.25/day (now about 1.2 billion based on most recent Chen and Ravaillion estimates at World Bank
    3rd quartile–up to midpoint of developing country income of $2/$2.50 (it was $2 in 2005.
    2nd quartile–over $2 or $2.50 up to $6/day/$2000/year??

    The point is not how accurate or non-holistic these numbers and segmentation may be but rather do they help us identify which segments microfinance and other development are working with and perhaps what the common needs of certain segments are.

    For those who care about reaching and serving the extreme poor (1.2-1.4 billion people, can we usefully think about testable hypothesis of segmenting the extreme poor into
    –a top half with millions?? already being served by microfinance and other approaches that choose to include the extreme poor, albeit with specialized small credit products, savings, insurance and other supports
    –a 2nd quartile from the bottom with many people who are well suited to approaches such as the TUP/ultra poor approaches, and
    –a bottom quartile of the Bottom Billion of people mostly best served by safety net programs and charity approaches because of severe disability?

  • Pingback: News and Events 4-27-12: Finding Good Teachers, Scaling Local Impact, Food Deserts, and More « High Impact Philanthropy