“Development” takes time.
It’s not quite like planting a tree — even if it takes as long — because with a tree we mostly know what will grow if the conditions are ok. But with society — complex, inter-connected, dynamic, chaotic and with lots of feedback loops — its actually quite difficult to know what will happen.
We can of course have hopes, perhaps even expectations, based upon “theories of change” or evidence from other times or places. In the end, though, any kind of social change process is a step off into the unknown. We have to wait for the passage of time to reveal what will happen. Even if things go “wrong” and don’t work as hoped, there will always be ripple effects that will be uncertain and unpredictable.
So it was with much curiosity that I found myself reading a recent draft paper on an intriguing education program that was tried in Ghana about a decade ago.
The Teacher-Community Assistant Initiative (“TCAI”) was only the second funding program I ever worked on in development.
It was 2010. A year earlier we had happily secured Board approval at CIFF for a multi-pronged strategy focused on increasing “educational achievement”. The first thing we did was school-based deworming (a whole different story for another day).
With that funding approved, contracted and in the capable hands of my portfolio management colleagues at CIFF, the next priority was to look for ways of leveraging the tremendous insights that were coming out of the remedial education programs being done by Pratham in India, involving what has now come to be known across the world as “teaching at the right level” (“TaRL”).
Michael Kremer (whom I had cheekily cold-called and asked to sense-check the draft education strategy, which he had kindly agreed to do) suggested it might be worth speaking with Innovations in Poverty Action (“IPA”) on something that they were exploring with the Government of Ghana.
And so in 2010, I found myself on a phone call with Annie Duflo and Wendy Abt.
Ghana has always prioritised, even prided itself on, education. The country launched free compulsory universal primary education in 1995, well before the UN Millennium Development Goals of 2000. Education has often represented 30% of the total government budget or 10% of GDP, well above other low-, middle- and high- income countries.
And while access had improved (from 60% in 2000 to 90% eight years later), quality had not tracked this. The influx of large numbers of “first generation learners”, combined with teacher deployment challenges, had placed considerable strains on learning. By 2005, only 19% of third-grade students were proficient in maths (and 16% in English).
The Government, however, was not standing still. Reflective of the societal, and hence political, priority placed on education, Ghana Education Services (“GES”) was actively engaged in a variety of measures to turn this around.
Systematic learning assessments were being implemented, from proctored statistically representative national ones, through low-stakes non-proctored ones in every school, to attempts to set up continuous assessment schemes. Local language instruction was being introduced into earlier primary grades. Lots of teachers were being constantly recruited and trained. And, school governance approaches aimed at increasing accountability to parents were being trialled.
Yet these same systematic assessments showed that the picture was still grim. By 2007, while sixth-grade proficiency levels showed some marginal improvements, third-grade proficiency levels had actually deteriorated even further from 2005.
It was in this context that GES and IPA were exploring what else could be done. And had turned their gaze towards India and Pratham’s achievement gains involving remedial education and TaRL.
From this exploration (and an initial process pilot funded by DFID and Hitz Foundation) was born a fascinating — and at that time unprecedented — nationwide initiative: TCAI. An initiative that also involved the Ministry of Youth, the largest teachers union (Ghana National Association of Teachers, “GNAT”), the curriculum division of GES, an interesting local non-profit called School for Life (which regularly delivered significant achievement gains working with out-of-school children in rural northern Ghana) among many others.
At its heart TCAI involved curriculum-aligned TaRL pedagogy. But TCAI was also about much more than TaRL.
From a pedagogy perspective, the idea was to create local-language curriculum aligned content that would be tailored to help the bottom 50% of primary students in 1st, 2nd and 3rd grades. So far, so TaRL.
But this would then be delivered in ways involving local high-school graduates who would be trained as “teacher assistants” and who would work in an officially sanctioned capacity within schools.
The “teacher assistants” would be non-unionised and be funded through the National Youth Employment Program. Further, the “teacher assistants” would have a governance line into local parent teacher committees, not just into teachers or district education officials.
Additionally, the remedial lessons would be trialled in various ways: after-school vs during the school day; delivered by the assistants vs by the teachers; to the bottom 50% of students vs just dividing the class into half.
All this would be trialled in a statistically representative nationwide pilot involving hundreds of schools, and tens of thousands of primary school children, across Ghana.
And vitally, this would all be done as a government program, through existing machinery. A very lean (four person) technical support unit was set up centrally within GES. There would not be a heavy, ongoing involvement of volunteers or non-profit staff involved in monitoring or administering TCAI. (This was quite important to try and avoid Hawthorne effects in the pilot, and thus get a more realistic sense of whether it worked or not).
The sensitivities in this are self-evident. For example, the idea of non-unionised, non-accredited high school graduates coming into schools in an official capacity on government payroll. The idea of comparing the relative efficacy of remedial lessons delivered by such assistants against unionised, university-educated teachers. The idea of officially sanctioned staff in schools having governance lines to local parents. All this in a national pilot, with important precedent-setting potential in a variety of other ways.
It took a lot of careful, delicate structuring and negotiation to get all the parties on board. GES and IPA did a very good job of bringing the key stakeholders along. And the willingness of GNAT to be part of an approach that would take many other teacher unions outside their comfort zone really deserves to be acknowledged. It was a sign of both enlightened self-interest, as well as a commitment to a national priority over and above parochial interests of defending their members’ interests at everything else’s expense.
After much due diligence (including asking lots of questions on the cost-effectiveness of education interventions, which probably drove J-PAL and Caitlin Tulloch mad looking at lots of other studies to compile their invaluable analyses!) CIFF’s Board agreed to fund the national pilot, with the International Growth Centre and the Hewlett Foundation also joining CIFF in funding a large, multi-arm, randomised control trial that would be conducted by IPA and J-PAL.
So, what happened?
Once the grant was funded, and my portfolio management colleagues took over the deal within CIFF, I moved on to the next topic — doing one of the most enthralling deep dives I have ever done, into early childhood education. And once that was kick-started at CIFF, I left to join Omidyar Network…
…which is basically a long way of saying that, embarrassingly, I sort of lost touch with the details of how TCAI was performing. Until I came across this recent draft paper by Duflo, Kiessel & Lucas on TCAI earlier this month (in this review prepared by Evans & Fei).
And it has been fascinating reading, to see what happened.
I won’t try and summarise the whole paper. You really should read it yourself (it features twice in Evans & Fei’s blog, it’s that rich in learnings). But some of the things that jumped out to me include the following.
The impact was heterogeneous across the country. We expected that to happen given the many differences in rural, peri-urban and urban settings (for example, more alternative jobs possibly changing assistant characteristics or attendance, or the linguistic diversity in the classroom in places with more incoming internal migration — Ghana has 11 official national languages).
There were implementation challenges. Materials were delayed (only 12 % of head teachers reported receiving them by the first term — this had happily increased to 90% by the third term). The program had to be tweaked about six months in and some re-training done (dynamic course correction at the start was expected). It turned out that the assistants didn’t get paid on time (some had to wait as long as eight months). Assistant-absenteeism happened (just as with teachers). Some schools were shut due to unrelated teacher strikes.
But there were still learning gains!
Two years of exposure to TCAI (with all those challenges described above) still raised childrens’ learning levels by 0.08–0.15 standard deviations, depending on which treatment variation you look at. That’s equivalent to gains of 18–34% of a year of schooling.
Girls benefited more from remedial classes than boys (by at least 0.10 standard deviations more). I find this particularly intriguing. TCAI was not designed with any explicit gender focus. The assistants were more likely to be male when compared to the teachers. Yet girls benefited more from remedial classes. Could it be that girls get crowded out in normal classrooms by boys in terms of teacher-attention?
TCAI’s effects persisted — children with only a partial exposure to the program still had up to 0.12 standard deviations of better learning a year after they had left TCAI. (This would be consistent with the importance of foundational learning as setting the base for subsequent progress).
Assistant-led remedial classes had 2x the effect of teacher-delivered targeted instruction. Teachers’ absenteeism did not reduce, but teacher engagement and time-on-task did improve.
There is a whole raft of other insights in the paper (including fascinating glimpses into the students and the assistants, the learning dispersion within classrooms and so on). It is well-written and I highly recommend spending time with it.
So, “would it have been worth it, after all, would it have been worth while”?
Well that’s up to each person to decide.
The program never did make the step from a very large national pilot to a permanent national program — although in fairness, Ghana went through a lot of macroeconomic stress during the last decade. [ERRATUM: this is no longer true — see exciting update below the end!]
Yet the counterpoint to that apparent failure is the size and persistence of the learning gains, as well as the gender benefits.
Evans & Fei’s review of effect sizes across 299 RCTs and quasi-experimental studies describes 0.15 standard deviation gains in learning as “kind of big”. Further, the working paper supporting that review also breaks studies into sample size buckets, since unsurprisingly smaller sample size studies see larger gains.
Set against those comparators, its learning gains of 0.08–0.15 standard deviations put TCAI into the 60th-80th percentile of RCTs. Even the 0.12 standard deviation persistent gain in learning a year after leaving TCAI would put it into the 70th percentile. And don’t forget that the effect on girls seems to have been a fair bit higher still, putting TCAI higher still if just a gender lens is used.
That doesn’t seem too shabby for a program that was also designed to try and minimise Hawthorne effects? But I’m no expert.
The A12n of TaRL pedagogy in a systematic way, that educationalists from Ghana and India compared notes on over a decade ago, has also evolved into ten countries on the continent trying TaRL approaches in various ways.
And even within Ghana, the lessons from TCAI have fed into new work — again being driven by GES — aimed at strengthening managerial support of teachers implementing targeted instruction, and which seems to be yielding promising gains.
Why was I reflecting on TCAI, even before I realised that this paper existed?
Because I’m struck by the similarities between certain key features of that approach and the challenges that education systems face now.
COVID is tending to increase learning gaps, budgets are constrained, and there are large numbers of under-educated and under-employed but motivated youth. Is it time to look at systematic ways of killing as many birds as possible with the fewer stones that we have left in our pockets?
For example, this recent joint piece, by the CEO of the Luminos Fund and a former Liberian minister of education, rightly draws attention to the difference between the health and education sectors in terms of last-mile leveraging of “local, motivated, high potential young people”.
And this recent RCT in Tamil Nadu by Muralidharan, Ganimian and Walters highlights the significant impact achievable in a highly cost-effective way at scale by using 10th grade educated local young women as “helpers”.
So is it time to look more systematically at better ways of reshaping and enhancing current over-strained systems? To learn from what was tried in the past in education, what has been done in other sectors, and what is being tried now in education elsewhere?
Was TCAI worth it — what do you think?
_______________________________________
ERRATUM: after I published this, Prof Adrienne Lucas updated that point with some v-e-r-y exciting news on how a revised program called STARS that builds on the insights from TCAI is going to be implemented by the Government of Ghana to deliver targeted instruction across 10,000 schools reaching 2 million students! See here for links to more information. Now the answer to the last question seems unquestionably clear :)
