A/B testing

Pioneering iterative A/B testing in the social sector: rigorous, rapid, and regular

We are a learning organization: we optimize our programs through ongoing rigorous, rapid, and regular testing termly across all countries and programs. These tests act as our in-house learning muscle and help us maximise impact, cost-effectiveness and scalability.

A/B testing is a simple but powerful way to compare two versions of something — version A and version B — to see which works better. In the tech world, companies might test two website designs to see which drives more sales. In the nonprofit sector, the same approach is increasingly being used as a powerful tool for organizations seeking to maximize their impact while operating under resource constraints.

Unlike traditional evaluations, like RCTs, that ask “does the program work?”, A/B testing focuses on how a proven program can work even better. This type of rigorous learning enables implementers to adopt the best-performing, most cost-effective, and most scalable program model. Using A/B testing as an ongoing part of an organization’s M&E practice allows for continuous, iterative learning and program improvement.

How does it work?

We randomly assign participants to two different versions of a program — “A” or “B” — and measure which version performs better. We focus on what we call “golden indicators”; These indicators aim to achieve the sweet spot of being able to be measured quickly, while also being consequential enough to inform decision-making. For example, in education this might mean test scores; in health, it might be perceptions, or behavioral intent. This balance allows us to act on evidence in real time. Because participants are randomly assigned to groups A or B, we can be confident that differences in results are caused by the program itself, not by other factors. The process is cyclical, meaning we are continuously optimizing.

A/B testing connects evidence to action through the ‘Three Rs’:

Rigorous

Like randomized controlled trials, A/B tests are rigorous because they use randomization, which means we can be sure that results capture causal impacts. Unlike RCTs that have a no-intervention control group, comparing two versions means all participants receive proven programming.

Rapid

Results can be reported in weeks or months to inform real-time decisions, rather than waiting years for evaluation findings. This speed is achieved through streamlined data collection focused on shorter-term indicators, which we refer to as “golden indicators”.

Regular

A/B testing is built into regular organizational M&E systems; multiple related tests are conducted in rapid succession to facilitate cumulative learning over time. Rarely is every A/B test transformative; it is powerful as a cumulative process, with small effects accumulating into large returns over time.

Two types of A/B tests we run

A/B tests aim to address both sides of the cost-effectiveness equation – reducing cost and enhancing effectiveness.

Cost-reducing tests focus on maintaining program impact while reducing implementation costs. These tests typically remove a component from a program. These cost-reducing tests make programs substantially more scalable, either for organisations themselves or for facilitating government adoption.

Effectiveness-enhancing tests aim to increase program impact at minimal additional cost. These tests typically add a component to a program. Effectiveness-enhancing tests often aim to improve efficiency while measurably enhancing impact.

Example questions we have tested include:

Cost-reducing test

Is conducting follow-up virtual teacher coaching via phone call as effective as in-person school visits for at-scale programs?

Yes. And it's cheaper, so we adopted this model for follow up coaching sessions.

Effectiveness-enhancing test

Do additional WhatsApp video materials improve impact at low cost?

No, so we dropped this idea.

Effectiveness-enhancing test

Are caregivers able to effectively co-instruct during remote tutoring sessions?

Yes, when caregivers actively engage, this more than doubles the impact! This breakthrough test revealed one of the most cost-effective program modifications in the education sector.

Our A/B testing history

We have been running A/B tests in our programming since 2017, and have built a sector-leading knowledge base and infrastructure around it. We now run regular A/B testing every term across all our programs. This ensures consistent improvement in program results. Over time, regular A/B testing becomes more than a method: it becomes a core part of our organizational DNA. Each test, whether a breakthrough or a small tweak, strengthens our learning muscle. This embedded approach ensures that evidence is not a one-off exercise but a continuous driver of program excellence.

*A/B testing history

Case study: Optimizing the ConnectEd remote tutoring program over 12 A/B tests

One-on-one tutoring is one of the most effective educational approaches, yet it has remained difficult to scale due to high costs. We used A/B testing to adaptively test and improve our technology-enabled tutoring program, ConnectEd, in order to enhance cost-effectiveness and scalability. Seven of the twelve tests led to efficiency improvements, ranging from 9 to 30 percent. Our findings both reveal the returns to iterative testing in social programs and contribute new evidence on simple, cost-effective strategies to improve learning outcomes.

To the right, an example of four A/B tests we ran as part of this process.

A/B testing support for the social sector

Building on nearly ten years of A/B testing experience, we now support other organizations interested in integrating the approach. If you are an organization interested in using A/B testing to optimize your programs, read more about the support available here.