There’s the way things are supposed to work, and then there’s the way stuff actually works.
At my age, you sort of get resigned to the general cussedness of the real world….People mean well, but gee–so if an organization has a theory that didn’t exactly work out, it’s pretty incentivized to put a positive spin on it.
That being a fairly typical reaction to products or programs that didn’t do what their creators had hoped they would do, I was stunned–and excited–to read a Vox article about a nonprofit that just came out and said “Well, I guess we were wrong.”
Last week, a major international development charity did something remarkable: It admitted that one of its programs didn’t seem to work.
No Lean Season is an innovative program that was created to help poor families in rural Bangladesh during the period between planting and harvesting (typically September to November). During that period, there are no jobs and no income, and families go hungry. By some estimates, at least 300 millionof the rural poor may be affected by seasonal poverty.
No Lean Season aimed to solve that by giving small subsidies to workers so they could migrate to urban areas, where there are job opportunities, for the months before the harvest. In small trials, it worked great. A $20 subsidy was enough to convince people to take the leap. They found jobs in the city, sent money home, returned for the harvest season, and made the trip again in subsequent years, even without another subsidy.
So Evidence Action, the nonprofit that funded the pilot programs of No Lean Season, invested big in scaling it up. In 2016, it had run the program in 82 villages; in 2017, it offered it in 699. No Lean Season made GiveWell’s list of top charities.
Evidence Action wanted more data to assess the program’s effectiveness, so it participated in a rigorous randomized controlled trial (RCT) — the gold standardfor effectiveness research for interventions like these — of the program’s benefits at scale.
Last week, the results from the study finally came in — and they were disappointing. In a blog post, Evidence Action wrote: “An RCT-at-scale found that the [No Lean Season] program did not have the desired impact on inducing migration, and consequently did not increase income or consumption.”
Why was this admission such a big deal? As the Vox article notes, it is exceptionally rare for a charity to agree to participate in a research project, to discover that its program as implemented doesn’t work, and then to actually publicize those results in a major announcement to donors.
It would have been easy, on multiple levels, for Evidence Action to do otherwise. It could have ignored or contested the results of the RCT; the research would still be published, but it would attract a lot less attention and publicity. Or it could have dismissed the failure as unrepresentative — there were unusual floods in Bangladesh in 2017, it could argue, which might have caused the program failures. Or it could have put a more positive spin on the results. After all, while the RCT was discouraging, it wasn’t devastating — there was, in fact, a small increase in migration.
Evidence Actiondid the opposite. “Consistent with our organizational values, we are putting ‘evidence first,’ and using the 2017 results to make significant program improvements and pivots,” the group wrote. “We are continuing to rigorously test to see if program improvements have generated the desired impacts, with results emerging in 2019. We have agreed with GiveWell that No Lean Season should not be a top charity in 2018. Until we assess these results, we will not be seeking additional funding for No Lean Season.”
Honesty. Respect for evidence. Respect for one’s donors.
This, of course, is the way things are supposed to work. This is why intellectually honest research is so important–to gather and consider evidence, and use that evidence to shape further efforts. To learn from reality, and to apply what has been learned in order to inform what we do going forward.
Empirical research. Honest evaluation of the results. Learning from our mistakes.
What a concept…..
Too bad this “evidence first” approach is not used by the State Of Indiana when evaluating some of its hair brain educational programs. Think vouchers. Even with student test results in front of their eyes Republican leaders march onward with their scheme to fund religious education at the taxpayer’s expense.
Does anyone foresee an assassination attempt? I don’t know who the perp or the target might be, if any.
This whole scandal reeks of shakespearean drama with much more at stake than a throne.
The situation must be a threat to Pence’s aspirations making him subject to suspicion.
(See Pence shift on his seat atypically nervously with Schumer-Pelosi-Trump-
Temper-Tantrum at the Oval Office.)
Somebody write the names of suspects who are not safely in jail on bits of paper and
toss them in a hat. Have a drawing; someone draw one for me, please.
The religious institutions lobby hard for their preferential treatment. They promise votes in exchange for this special treatment.
Justice Scalia found no quid pro quo in lobbying efforts.
He also found no signs of racism within the South in regards to voter suppression. After the sham election we just witnessed, what possible evidence could Scalia have been looking at?
The principle of owning up to your mistakes is spiritual or moral. It’s called being responsible and it’s the mature response. However, it is rarely used today. As Shelia points out, we mostly see either spin (propaganda) or blame.
Watching Trump on Twitter is a great example of amoral behavior. His inability to own his mistakes and grow from the experience is why he is often referred to as a Baby.
So, with the addition of civics being taught in the classroom, I also recommend classes in morality – making informed choices. It’s not even a religious discussion. Doing the right thing has nothing to do with God.
Our entire culture from top to bottom seeps with a lack of civics and morality – two things needed to live peacefully and responsibly in society.
As others will agree, it’s systemic. When our public servants are allowed to take bribes from the oil industry and then sit on committees and/or vote on matters impacting the environment, then an example is set for all watching. Using evidence is irrelevant.
The lesson we teach is making a profitable choice is preferential over acting morally.
The “Just Say No” drug program failed; it appeared to be a dare to dealers and users and look where we are today with the opioid epidemic.
And Theresa; “No Child Left Behind” has worked its way to the entire public education system leaving behind the children who most need better education, test results at all levels have proven to the current system to be a failure. With Indiana at the top of the list regarding number of voucher students but near the bottom of the list regarding students receiving quality education, this state came up with the ridiculous motto “Honest To Goodness, Indiana”. That can be taken in more ways than the way the state meant it.
Pence’s RFRA was supposed to work for all who were members of any religious denomination but works against those Pence declares unworthy of receiving his evangelical protection.
And the 2016 presidential election was touted to be Democrats vs. Republicans as is the norm; today we know that didn’t work out. Trump did not become a Republican; the Republican party became the Trump party; no way to put a “positive spin” on it. We are at another perilous Trump Friday which may become another “date that will live in infamy” as we are losing the last, and possibly, the most vital “cool head” in his administration. Secretary of Defense James Mattis may have been working behind the scenes to calm the waters Trump has disturbed by insulting the leaders of our allied nations. Deserting their troops may be the final straw. This Trump Friday could push us – the U.S. – over the edge or could be the turning point to our salvation – if the Republicans have the balls to act against this insanity as well as speak against it. That tactic hasn’t worked out so far.
“Empirical research. Honest evaluation of the results. Learning from our mistakes.”
If the aim of a program, such as vouchers, is to control the narrative and use the people you purport to serve, then you could say that nearly all of the programs initiated by right winged or evangelical groups have been wildly successful. We only need empirical data for those few organizations whose aim is the betterment of the people they serve.
It is refreshing to see at least one organization actually use data regarding the results of their work. Most organizations measure themselves by the amount of money they take in each year. If it’s more than last year, we must be good.
Just for the record this is far from the first use of science for this purpose. USAID and the United Nations Population Fund were conducting such evaluations of reproductive health/family planning interventions beginning in the late 1960s. Pilot tests of intervention were fairly routinely conducted. Several models were often tested and compared, and by and large the unpleasant results used in re-engineering the outcome. It ain’t new, but perhaps it is still underutilized.
Theresa, Indiana’s experiments in education “reform” have been a resounding success – from the point of view of the perpetrators. Their goal is to kill public education and under-educate the population. SO many birds with one stone – kill the unions, enrich the cronies, undermine class mobility, and most of all, prevent the existence of an informed citizenry. SO much win for the party of “We got ours, so f— you.” And the peasants, their obedience earning them eternity with their god, roar with approval.
Most of the social programs that don’t work are Republican brain children. They don’t work because they aren’t properly implemented, supported or measured. No Child Left Behind is a perfect example of the failure of Republicans and their capitalistic sponsors (See the ultimate in right wing failure regarding public education in Betsy DeVos.) It was predicated on punishment without any element of progressive or positive vectors toward improving the plight of teachers, buildings and supplies.
In short, the politicization of Republican programs is merely a smoke screen to garner votes, nothing more.
Only people with LOW levels of education could continue to support the IN Republicans
so
I guess “dumb kids” is a good outcome for the Republicans
and MORE CHURCH — YAY for them
The Tyranny of Metrics by Jerry Muller:
“The belief that the best way to motivate people…is by attaching rewards and penalties to their measured performance.” “Metric fixation is the persistence of these beliefs despite their unintended negative consequences when they are put into practice.”
I have often wondered how effective these non-profits are in terms of the effectiveness in implementing their mission statements.
Over it @ 8:51 am good comments. Privatization has at it’s core turning a profit for the stock holders/owners anything that interferes with that goal must be eliminated. Unions are a prime target.
As a Boomer, I remember that sense of awe back during the space race which began with Sputnik. So much science and technology used to expand our knowledge of the earth and space, with peaceful intentions. Astronomy, archeology, combined with geology and genetics have identified the truth of the Big Bang billions of years ago and evolution.
Sadly, in the 60 or so years since Sputnik we still have a sizable number of people who reject Science and Truth in favor of a supernatural origin of the universe and in particular a bind belief in humanity being the culmination of creation.
It’s inspiring to see a charity do an honest evaluation of the outcomes of one of its programs. Hopefully, they will learn from their mistakes and find better ways to help the people of Bangladesh.
I wish that we did the same sort of outcomes research with all government policies and programs. We need to fund what actually works for the greater good. Fortunately, the states that reformed their criminal justice system did this which in turn led to the recent passage of a bill that reforms the federal criminal justice system.
It would also be better if our health care facilities , treatment centers, and pharmaceutical companies did rigorous outcome studies. They need to be more concerned with creating positive outcomes and less concerned with huge profits that make shareholders happy.
John Neal @ 9:36 am
“The belief that the best way to motivate people…is by attaching rewards and penalties to their measured performance.” “Metric fixation is the persistence of these beliefs despite their unintended negative consequences when they are put into practice.”
As someone who labored in a White Collar environment for most of my working career this idea of sales equals profits over rules and regulations has had dire consequences. The Savings and Loan Scandals, dot com bubbles, housing bubbles, etc.
The Fraudsters of Wall Street can always count on the Corporatist elected officials and their lackeys in the enforcement arm of the government to give them that Get of Jail Card, when they find themselves in legal trouble.
Science of course is the gold standard for insuring that we can tell reality from what we wish was true by mearuring it very carefully under very controlled conditions. While that is the most reliable means we have it’s not always the most popular means and one of the examples of that is anthropogenic global warming. We don’t like the reality that’s been discovered. So instead of using our knowledge of reality to mitigate present and upcoming very expensive consequences to tax payers, many choose to ignore the threat and argue that science got the data wrong. Not because there is conflicting data at all, only because the answer is inconvenient.
There may have been times in the past in which we could afford mythology but they are long gone. Yet the more true that statement gets it seems the more true that we we are unable to let go of comfortable myths and adapt instead to hard reality.
Take a deep breath, Sheila, and enjoy your wonderful family this holiday season. Wishing you many good times and happiness in the new year.