is one of GiveWell’s top charities. The main factor driving our grantmaking to New Incentives is an analysis of its program’s cost-effectiveness. We summarize our cost-effectiveness analysis in detail in our
separate report. This page provides additional information on New Incentives’ program, our qualitative assessment of New Incentives, and the monitoring and evaluation data it shares. This information feeds into our overall recommendation (alongside cost-effectiveness) and provides additional context so that we can understand and appropriately model the impact of its program.
Published: April 2024; Last updated: September 2024
Summary
What do they do? New Incentives runs a conditional cash transfer program in northern Nigeria. The program aims to increase uptake of routine vaccinations (which is low in this part of Nigeria) through small cash incentives (worth approximately $9.50 in total). In areas where it works, New Incentives also conducts awareness-raising activities to encourage vaccination, and various activities to strengthen the vaccine supply chain. (More)
We recommend New Incentives’ program as a top charity because of its:
- Sole focus on a program that we think is very cost-effective (more).
- Standout transparency and commitment to making its program cost-effective (more).
- Processes for gathering and sharing detailed monitoring data on all aspects of its program (more).
Our main reservations about New Incentives are:
- New Incentives is at an earlier stage of its development compared to GiveWell’s other top charities. While it has grown successfully in recent years, it might be more susceptible to unexpected problems or a decline in the quality of program delivery as it grows further. (More)
- New Incentives’ program (which handles a large volume of cash directly) poses a higher risk of fraud than other programs that GiveWell funds. (More)
- There is some opposition to New Incentives’ program within the Nigerian federal government, which poses a potential threat to the sustainability of the program long-term. (More)
These assessments are based on the following components:
What do you get for your dollar? GiveWell believes New Incentives is one of the most cost-effective programs donors can support. We estimate that it costs roughly between $1,500 and $6,000 (varying by state) to avert a death in areas where New Incentives works. We think New Incentives’ program is cost-effective because it leads to a substantial increase in vaccination rates (9 to 18 percentage points depending on the location) at moderate cost (approximately $18 per child enrolled, and ~$58 to $153 per additional child vaccinated), and vaccines are effective at protecting against child mortality.
We also have a number of uncertainties about our analysis of the cost-effectiveness of New Incentives’ program. These include:
- Whether vaccine efficacy is lower in Nigeria than suggested by evidence from other locations.
- Whether our analysis is adequately accounting for increases in vaccination rates that would have taken place anyway, regardless of the program.
See our separate report on the cost-effectiveness of its program for more details.
What information has New Incentives shared about its program? We ask organizations that we fund to share monitoring data and other detailed information on their programs. We use the data as inputs in our cost-effectiveness analysis, and its quality and reliability also inform our overall assessment of the program. Our analysis of New Incentives’ cost-effectiveness is based on a randomized controlled trial (RCT). New Incentives has collected several monitoring indicators that allow us to compare performance of the program today to performance in the RCT. Overall, our assessment of the data is that New Incentives has maintained a high level of program quality at scale, similar to the level achieved in the RCT, although we have seen a rise in supply problems for some vaccines. We will continue to monitor this issue as New Incentives grows further, since we think programs often face unexpected challenges as they scale. (More)
New Incentives’ monitoring data does not show evidence of increasing fraud since the RCT. But we believe this is inherently a high risk for its program, and we’re still uncertain whether there are types of fraud that wouldn’t be captured by this data. (More)
New Incentives also conducts follow-up surveys to assess vaccination coverage after it has worked in each of the new areas for a year, and then every six months. We have only just begun analyzing these surveys in detail, but expect them to provide useful information about the program’s effect in the future. (More)
What is GiveWell’s qualitative assessment of New Incentives? We make qualitative assessments of our top charities alongside our cost-effectiveness analyses to inform our grantmaking. Overall, our qualitative assessment of New Incentives is very strong, even compared to our other top charities. Factors we take into account for this include (more):
- We think New Incentives stands out for its dedication to identifying, responding to, and being transparent about issues it faces. For example, it has proactively brought challenges it is facing to our attention even when they might negatively affect our view of its program.
- We think that New Incentives shares GiveWell’s focus on evidence-based decision making and maximizing the cost-effectiveness of its program to an unusual level among organizations that GiveWell funds.
1. What do they do?
1.1 In a nutshell
New Incentives runs a conditional cash transfer (CCT) program in northern Nigeria. The program aims to increase uptake of routine childhood vaccinations (which is low in this part of Nigeria) through small cash incentives. Caregivers who bring their children for routine vaccines, which are provided through government clinics free of charge, can receive 1,000 naira at each of the six vaccination visits in the routine immunization schedule. As of May 2024, New Incentives is planning to offer an additional 5,000 naira incentive at the measles 2 vaccination visit for caregivers whose infants completed the entire routine immunization schedule. This would bring the total incentive amount to 11,000 naira (or about $9.50).
In addition to disbursing incentives, New Incentives also:
- Conducts awareness-raising activities to inform caregivers about the program and the importance of vaccinating children. (More)
- Works with the government and other partners to ensure sufficient vaccine supply by identifying and addressing bottlenecks in the vaccine supply chain. (More)
New Incentives started this program in May 2017. The program was evaluated by a randomized controlled trial (RCT) from August 2017 until February 2020, which found that it increased vaccination rates of key childhood vaccines by 14-21 percentage points. As of January 2024, New Incentives’ program is operating in nine states in Nigeria. Between January and November 2023, New Incentives enrolled 1.39 million infants in its program.
In Nigeria, New Incentives is known as the All Babies Are Equal (ABAE) Initiative.
States where New Incentives operates its CCT program, as of January 2024, are highlighted in blue.
1.2 Vaccines incentivized
New Incentives directly incentivizes (i.e., provides cash transfers conditional on children receiving) the following vaccines:
- Bacille Calmette-Guérin vaccine (BCG) against tuberculosis
- Pentavalent vaccine (Penta) against diphtheria, tetanus, pertussis (whooping cough), hepatitis B, and Haemophilus influenzae type b
- Pneumococcal conjugate vaccine (PCV) against pneumococcal disease
- Measles (MCV) vaccine
Some of these vaccines are delivered in multiple doses. Other vaccines (Hepatitis B, OPV and IPV (the vaccines against polio), meningitis A, yellow fever, and rotavirus ) are delivered at the same visits as the directly incentivized vaccines. We consider these vaccines to be "indirectly incentivized" because we’d expect the cash incentives to increase the number of children who receive these vaccines too. See the vaccination schedule below.
| Age of visit | Directly incentivized | Indirectly incentivized |
|---|---|---|
| Birth | BCG | Hep B, OPV0 |
| 6 weeks | Penta1, PCV1 | OPV1, Rotavirus |
| 10 weeks | Penta2, PCV2 | OPV2, Rotavirus |
| 14 weeks | Penta3, PCV3 | OPV3, IPV, Rotavirus |
| 9 months | Measles1 | Yellow Fever, Meningitis A |
| 15 months | Measles2 | n/a |
Caregivers receive a portion of the overall cash incentive at each visit. As of December 2023, New Incentives offers caregivers 1,000 naira (about $0.86) per visit. This incentive schedule has been updated over time to account for inflation and increase caregivers’ motivation to receive all recommended vaccines (details in footnote).
1.3 How does the program work?
The main goals of New Incentives' program are:
- Increasing vaccine demand by (a) distributing cash incentives and (b) conducting awareness-raising activities.
- Ensuring sufficient vaccine supply by supporting the government and other partners to identify and address bottlenecks in the vaccine supply chain.
Distributing cash incentives
Vaccines incentivized by New Incentives are administered by the staff of government clinics that partner with New Incentives (more). Vaccinations and cash disbursements take place on days during which clinics provide routine immunizations ("immunization days"). Some children are also reached through outreach efforts targeted at communities with particularly low vaccination rates. (More)
After a child is vaccinated, New Incentives checks whether the child meets the eligibility criteria to receive a cash incentive (more) and, if so, disburses the incentive to the child's caregiver (more). New Incentives also conducts audits and remote vetting of cash disbursements to prevent fraud and ensure disbursements reach eligible caregivers (more).
Partnership with clinics
New Incentives pursues partnerships with clinics that :
- Offer routine immunization services
- Are located in an area where the clinic is safe to visit during the day
- Are public clinics
Eligibility for enrollment
Children are usually enrolled in New Incentives' program during their first vaccination visit (for BCG, for example, see the schedule above ), with one exception (more below). The eligibility criteria for an infant to enroll in the program are:
- The infant's caregiver lives in a local government area (LGA) where the program is operating. This is checked by looking at the residence (settlement) reported on the Child Health Card (CHC) by the clinic staff and by asking questions of caregivers.
- The infant received a BCG vaccine on that immunization day (and has not previously received a BCG vaccine). Receipt of the BCG vaccine is used as a criterion for enrollment because it is the first vaccine in Nigeria's routine immunization schedule (scheduled to be given shortly after birth). Infants brought in later (up to 12 months of age) will still be administered the BCG vaccine at their first clinic visit and can be enrolled in the program at that point. New Incentives uses a fresh BCG vaccination mark as evidence that a child has been vaccinated with BCG that day, and it uses the scar left by the BCG vaccine as a way of checking that the infant has not previously received BCG.
- The caregiver gives consent to collect their and their infant’s personal data.
An exception applies to these eligibility criteria:
- In the first 6 months of operations at a newly expanded clinic, infants who received the BCG vaccine before New Incentives' expansion can be enrolled in the program if the infant’s name was previously recorded in the Child Immunization Register (CIR), a ledger that tracks which infants have received vaccinations at the clinic.
Our understanding is that this exception, introduced in 2022, is designed to reach additional children who could benefit from the program, but who would otherwise not be eligible (e.g., because they received BCG before New Incentives began operating in a particular area). The proportion of children enrolled under this exception (i.e., those who received BCG before they were enrolled) was 20% in 2022 and stayed high throughout 2023 (18% - 29% depending on the month in 2023). We see this as potentially concerning (more below), but we did not have strong prior expectations for how high this proportion should be. As of the time of writing we have not yet investigated the implications of this in depth, and are considering whether to make an adjustment to our cost-effectiveness analysis to account for this.
Eligibility to receive cash transfers
To be eligible to receive a CCT, an infant must meet the following criteria :
- Be enrolled in New Incentives' program. New Incentives checks that infants are enrolled by asking caregivers to provide their All Babies Are Equal (ABAE) ID, a unique ID provided by New Incentives at the point of enrollment (more below). This ensures New Incentives can identify each caregiver and avoid disbursing incentives to caregivers whose infants are not eligible for the program.
- Have received a directly incentivized vaccination during the same immunization day. New Incentives verifies this by checking that there is a vaccination marked with the current date and a golden dot on the child's Child Health Card (a document provided to caregivers that reports the infant's residence, date of birth, and vaccinations the infant has received on which dates). The golden dot comes from a golden pen provided by New Incentives’ staff. Clinic staff mark each child’s Child Health Card with the pen to indicate that a child has received a vaccine.
- Have had both arms checked for a BCG scar, and have had the injections reported on their Child Health Card confirmed with their caregiver. Our understanding is that New Incentives checks for BCG scars to assess whether infants are being enrolled in the program more than once, and confirms the injections reported on the Child Health Card with caregivers to ensure that the reported vaccinations were actually administered.
- Have had a New Incentives Field Officer directly observe their vaccination.
- Be age eligible (e.g., under 1 year old for new enrollments, older than 9 months if receiving the first dose of the measles vaccine). New Incentives verifies this by comparing the current date with the date of birth reported on the Child Health Card.
Immunization days
Cash incentives are disbursed during immunization days. The process is:
- New Incentives assigns one or more Field Officers (FO) per clinic. On immunization day (or the day before), the FO collects cash from an ATM before traveling to the clinic.
- At the start of the immunization day, clinic staff hold a health talk for caregivers, during which the FO also introduces the New Incentives program.
- Clinic staff provide new children with a Child Health Card (CHC) and add them to the Child Immunization Register. A CHC is a document provided to caregivers that reports the infant's residence, date of birth, vaccinations the infant has received on which dates, and other information.
- Infants are vaccinated according to the routine immunization schedule and their child health card. After administering a vaccine, the clinic staff write down the vaccination date on the CHC and mark it with a golden dot near the record of the vaccination.
- For new infants, the FO then confirms their eligibility for the program according to the criteria discussed above. Caregivers of eligible children are provided with an ABAE Card (a card that includes basic information about the New Incentives program and immunization schedule) and an ABAE ID (applied to both the ABAE Card and the Child Health Card). As part of the enrollment process, the FO takes a picture of the infant's BCG injection mark.
- For all infants, the FO disburses cash to their caregivers after confirming that they are eligible to receive a cash disbursement that day (according to the criteria discussed above). The FO then takes a picture showing the caregiver holding the cash, the child, and the Child Health Card (including the ABAE ID). Later, the pictures are reviewed by remote staff to check for possible fraud (see below).
New Incentives also provides cash incentives during targeted outreach events (vaccination sessions targeted at settlements with particularly low vaccination rates). New Incentives identifies settlements for targeted outreach based on data and feedback from New Incentives staff, clinic staff, and community members. At these events, clinic staff administer vaccines and New Incentives staff enroll new infants and disburse incentives according to the same criteria outlined above. Clinic staff are reimbursed by New Incentives for transportation costs for targeted outreaches. We have not investigated these events in detail, and our impression is that they account for a low proportion of the caregivers reached by the program.
Process compliance and anti-fraud checks
New Incentives’ Field Officers collect evidence over the course of each disbursement day and submit it using a custom New Incentives app. Remote New Incentives staff (called “console agents”) are then assigned to review the data to check for potential fraud, as part of the process for approving staff expenses. The data console agents check includes:
- Disbursement photos: For each disbursement, Field Officers are instructed to take a photo of the caregiver, their child, Child Health Card and ABAE ID, and the cash disbursed (more above). Console agents review these photos to check that incentives appear to have been given to eligible recipients (e.g., caregivers of young infants who have child health cards with an ABAE ID attached) and that caregivers received the correct amount of cash. In order to provide a deterrent against fraud, New Incentives deducts money from Field Officers’ submitted expenses if the money shown in the disbursement photo does not match the amount reported by the Field Officer. Our understanding is that these checks are conducted for all disbursements.
- Location reviews: On every disbursement day, Field Officers are required to submit an "attendance photo" of themselves in the clinic or at the outreach location. They also submit background photos of the clinic every 2 hours over the course of the day. Console agents check this data along with photo metadata (i.e., data showing the time and location the photos were taken). This is to check that disbursements took place at an approved location and that the Field Officer was the person conducting the disbursements.
- Before and After Disbursement Session Reports: Console agents review photos taken of the cash on hand at the beginning and end of each disbursement day, to ensure the sums match what would be expected based on the number of disbursements. They also check a photo of the clinic tally sheet and compare the number of vaccinations recorded to the number of disbursements given, to validate that incentivized vaccinations were recorded.
Other anti-fraud processes used by New Incentives:
- Biometric reviews: New Incentives uses the Amazon Rekognition service to review the photos of caregivers taken during immunization days. The service is employed to assess potential duplicate enrollments and duplicate disbursements. A sample of flagged matches are manually reviewed by select New Incentives staff. Our understanding is that these checks happen after each disbursement day, meaning that the biometric review is not able to detect fraud in real time. Instead, it is used to deter fraud and identify possible fraud after the fact. Note that GiveWell has not independently vetted how this service is used or how effective it is.
- Audits: New Incentives employs staff in the role of auditors who visit each clinic at least every eight weeks to assess compliance with protocols and check for evidence of fraud by (a) observing activities; (b) interviewing caregivers, clinic staff, and New Incentives staff; and (c) conducting an office money spot check (i.e., verifying that the FO is in possession of the amount of cash expected based on New Incentives' records). New Incentives reports that ~98% of these audits are unannounced. Separately, New Incentives Field Managers audit at least 5% of disbursement days. New Incentives reports that ~92% of these visits are unannounced. We do not have a strong understanding of how these visits differ from the main audit procedure.
- Caregiver interviews: During the enrollment and disbursement process, Field Officers ask each caregiver whether they gave out "dashes," or tips, to anyone at the clinic. Auditors and Field Managers also ask caregivers whether they gave tips to Field Officers during their audits. This data is then submitted through a New Incentives app and reviewed to identify possible fraud.
- Vendor spot checks: Auditors conduct periodic calls to randomly selected vendors from which New Incentives staff submitted expense claims. These calls are designed to verify that the expense claims were legitimate.
- Fraud investigations: Cases of suspected fraud are collected through the processes above, as well as a hotline and anonymous staff reporting channels. Those are then assessed by auditors, who review the evidence to establish whether there is reason to believe fraud occurred.
We discuss how we interpret the risk of fraud to New Incentives’ program, and how effectively these procedures account for it, below.
Awareness-raising activities
New Incentives' awareness-raising activities aim to increase the demand for routine vaccinations by increasing awareness of the program and sharing information with caregivers about immunization. These activities fall into two main groups: activities during immunization days and activities outside immunization days.
Activities during immunization days
- During the health talk (before children are vaccinated), a New Incentives Field Officer (FO) introduces the program to caregivers, discusses the benefits and potential adverse effects of immunization, and explains eligibility requirements.
- After cash incentives are disbursed, FOs communicate and/or indicate on caregivers' ABAE Cards the date of the next vaccination.
- New Incentives advertises the program by putting up posters outside the entrance of partner clinics.
Activities outside of immunization days
- Awareness meetings. Meetings in targeted communities aimed at increasing knowledge of the program and addressing concerns about vaccinations. The meetings are typically run by New Incentives staff and clinic staff, who distribute awareness cards (including basic information about the program).
- Community member outreach. New Incentives recruits community members to increase program awareness and to track infants who are behind schedule for receiving vaccinations and encourage their caregivers to complete the immunization schedule. Community members receive a stipend of 1,000 naira. Community members can include town announcers or town criers, village leaders, youth members, traditional birth attendants (TBAs), community health workers (CHWs), and others. For example, "town criers" may be engaged to make announcements informing caregivers about immunization days and outreach sessions.
- Engagement with traditional and religious leaders. New Incentives engages with traditional and religious leaders to understand the sources of vaccine hesitancy and methods to help address these. These efforts can result in community discussions on the benefits of vaccination and alignment with religious beliefs, reasons for providing incentives, as well as announcements in places of worship.
- Ambassadors. Field Officers sometimes ask supportive caregivers to promote the program in their areas of residence or help bring back enrolled infants who might be delayed in getting their next vaccinations. Ambassadors are unpaid volunteers.
- Radio messages. New Incentives introduced paid radio advertisements in November 2023 in the states of Bauchi and Gombe. These advertisements provide information on New Incentives' program and the benefits of vaccination. New Incentives may expand the radio messages to additional states after assessing their impact in Bauchi and Gombe.
During review of this page, New Incentives shared the following estimates of the frequency and reach of awareness activities:
| Activity | Engagements per clinic, future expected frequency | Caregivers reached per engagement (expected) |
|---|---|---|
| Immunization day activities | 1.25 per week | 20 |
| Awareness meetings | 1 every 2 months | 40 |
| Community member engagements | 2-3 per month | - |
| Targeted outreach | 1 per month | 12 |
New Incentives also shared data on how new program participants say they became aware of the program:
| Where participants heard about the program | 2018-2020 | 2021-2023 |
|---|---|---|
| Neighbor | 48% | 58% |
| Friends | 39% | 52% |
| Family | 31% | 43% |
| Health worker at this clinic | 15% | |
| New Incentives staff | 11% | |
| Traditional birth attendant | 11% | 9% |
| Village leader | 11% | 11% |
| Town crier | 7% | 8% |
As of the time of writing (December 2023), we have not independently reviewed this data.
Supporting vaccine supply
New Incentives’ supply-side work aims to ensure consistent vaccine supply at the clinics where it operates. Its work can be divided into responsive and preventive activities.
Responsive
Specialist New Incentives staff (called “supply side officers”) identify incidents of low supply. These include stockouts (where a clinic does not have a vaccine throughout an immunization day); runouts (where a clinic runs out of a vaccine during an immunization day); shortages of Child Health Cards, syringes, or other supplies; and incidents where children were not served.
Supply side officers identify these cases through a mix of methods including
:
- Field Officers’ reports from each immunization day.
- Phone calls to government staff responsible for ensuring vaccine supply (“cold chain officers”) at the state, local and zonal levels and at “apex clinics” (larger clinics that store vaccines for clinics with more limited storage capacity). These calls aim to identify whether vaccines are in low supply and if any vaccines are due to expire in the following months.
- Information from other New Incentives teams (e.g., audit findings).
- Cases where New Incentives Field Officers have identified expired vaccines being used at clinics (more below).
Supply side officers then attempt to resolve supply problems where they’re identified. This can include :
- Conducting follow-ups and reminding cold chain officers to divert vaccine stocks to clinics where they are needed (e.g., by retrieving stocked out vaccines from apex facilities, LGA cold stores, or state cold stores).
- Encouraging cold chain officers to submit vaccine utilization and vaccine requests (the form needed to request vaccines from national supplies) on time, and checking they have the information they need to do so.
- Occasionally providing financial support to relieve bottlenecks (for instance, by paying for transport costs to deliver vaccines).
We think it is possible that New Incentives' program increases the likelihood of vaccine supply shortages in areas where New Incentives does not operate, though we currently judge this risk to be low. We discuss this potential risk further in our report on the cost-effectiveness of New Incentives' program.
Separately, Field Officers conduct vaccine vial monitor (VVM) checks at partner clinics to ensure that vaccines are in a usable condition and have not expired. Field Officers randomly select one vial of each directly incentivized vaccine during immunization days and check it is in a usable condition. Where any vaccines are not usable, Field Officers request that clinic staff check all the vaccines at the clinic. They also record data on these checks in a log so that supply side officers can verify the VVM assessments and escalate issues to respective government agencies when warranted.
Preventive
New Incentives also aims to prevent supply issues from occurring. It uses strategies including:
- Requesting same-day “top-up” disbursements of vaccines from Apex Clinics or LGA Cold Stores, when New Incentives staff have notified their managers or supply side officers that there is a pending stockout at their clinic.
- Notifying local cold chain officers about clinics that use an unusually high volume of vaccines (“high-consumption facilities”), and encouraging them to allocate more vaccines to these clinics.
- Reviewing stock availability at state and zonal levels against estimated consumption and prompting requests for vaccines ahead of widespread vaccine stockouts.
- Engaging with vaccine supply stakeholders to share findings and coordinate to close supply gaps where possible, including through regular monthly round table meetings with state and zonal cold chain officers.
New Incentives collects extensive data on supply-related issues - see the monitoring section below for details on how we interpret this data.
Relationships with government
New Incentives works in government clinics and requires the support of agencies within each Nigerian state responsible for those clinics to operate. When it expands to a new state, New Incentives negotiates and signs a Memorandum of Understanding (MoU) with the state authorities, setting out each party’s responsibilities. New Incentives reports that it also has support from and has previously signed MoUs with several national health and social protection agencies.
Overall, our impression is that New Incentives’ relationships with authorities at the state and local level in states where it works are strong, and it conducts a number of activities to strengthen those relationships (e.g., Joint State Meetings (roundtables) with health officials in the states where it works). However, we haven’t investigated this aspect of its work in detail. Our understanding of these relationships is also based largely on discussions with New Incentives itself. In 2021, we spoke to two government officials and a UNICEF representative to understand their perspectives on New Incentives' work, but we have not prioritized further conversations with New Incentives’ partners since then.
New Incentives also faces some opposition to its program among some members of the Nigerian federal government. Critics have raised concerns about the program, particularly that there could be a backlash if New Incentives discontinues its program in a given area, leading to falling vaccination rates. We discuss how these concerns affect our view of the program in detail in our separate report on the cost-effectiveness of New Incentives’ program. Overall, our best guess is that this consideration does not significantly offset New Incentives’ program benefits in areas where GiveWell supports the program. However, there is limited high-quality evidence on this question, and we may be over or underestimating the effect of discontinuing the program. We see it as an important priority to engage more with critics of the program in the future.
Our understanding is that the decision-making authority to support New Incentives’ program rests primarily with the states where it works, and therefore that the opposition in other parts of the government is not a formal obstacle to New Incentives delivering its program. However, we see this opposition as a concern for other reasons:
- We think that it’s possible that this could impede New Incentives’ ability to deliver the program in other ways.
- In general, we are wary about the risk of undermining government authorities in locations where we fund programs. This is for a mix of reasons, including governments possibly having access to better local information than we do, and being potentially more responsive to the wishes of their citizens.
We do not currently account for the second of these concerns. We currently account for the first concern with a -5% adjustment in our cost-effectiveness analysis, discussed here in our separate report on the cost-effectiveness of New Incentives’ program. This adjustment is small because New Incentives has been running its program since 2017 and we have not yet seen examples of opposition from the authorities impeding its ability to deliver the program. We plan to continue monitoring this issue to understand whether it appropriately captures the risk to New Incentives’ program.
1.4 New Incentives as an organization
Overview
New Incentives was founded in 2011. It initially began piloting a program using conditional cash transfers (CCTs) for the prevention of mother-to-child transmission of HIV in Nigeria, and pivoted towards its current program (focused on routine childhood vaccinations) in 2016. As of 2023, it is solely focused on delivering its childhood vaccination program in northern Nigeria.
Staff structure
As of the end of 2023, New Incentives employs almost 3,000 staff, with most staff based in the parts of northern Nigeria where the program operates. Some of the most common New Incentives staff include:
- Field Officers, who are responsible for disbursing CCTs, running awareness activities, and collecting information about vaccine supply during immunization days.
- Field Managers, who are responsible for supervising Field Officers and identifying and addressing problems with enrollment, retention, and vaccine supply at the clinic level.
- Console agents, who conduct the anti-fraud checks discussed above.
Spending breakdown
Around one third of New Incentives’ costs are the cash incentives themselves, with the rest split between staff compensation, supply costs, and other activities. See below for a breakdown between November 2017 (around when the program began) and December 2023 (the most recent data available).
| Expense category | Total ($m) | % |
|---|---|---|
| Conditional cash transfers | $10.78 | 23% |
| Staff compensation | $8.90 | 19% |
| Transport and supply costs for in-clinic and awareness activities | $8.73 | 19% |
| Contractors and Consultants | $6.12 | 13% |
| Stakeholder Relations (meetings and vaccine transport) | $0.70 | 2% |
| Other | $6.32 | 14% |
| Cash transfer commitments | $4.66 | 10% |
| Total | $46.21 | 100% |
2. Monitoring and information sharing
2.1 Overview
GiveWell asks our top charities to share detailed monitoring information on their programs. The aim of reviewing this information is to assess whether the program is being conducted to a high quality and whether it is reaching recipients as intended. We use the outputs from these reviews as part of our cost-effectiveness analyses to make funding decisions.
Because GiveWell’s analysis of New Incentives’ program is based on a 2017-20 RCT, we have focused most of our attention on analyzing data that could help us understand how the program’s impact at scale might differ from the RCT. We have reviewed (or plan to review) three main sources of information:
- Program performance indicators (retention through the vaccination schedule, repeat enrollments, supply-side issues, and reduction in the value of the incentive because of inflation).
- Data on vaccination rates from coverage surveys conducted by New Incentives in areas where it works. We can compare these to the expected impact of the program based on the RCT.
- Fraud monitoring data that we can compare to the RCT.
Our findings:
- Program performance indicators: These have largely remained stable since the RCT, although we have seen a worsening in the rate of supply-side problems at New Incentives-supported clinics for some vaccines. Overall, we interpret this as evidence that New Incentives’ delivery of the program has remained high-quality at scale. (More)
- Data on vaccination rates: As of December 2023, we have only just begun analyzing these surveys. We plan to incorporate them in our analysis of the program in the future. (More)
- Fraud monitoring data: We have not seen evidence of fraud risk increasing since the RCT. But we remain uncertain about this because we’re not sure whether the monitoring data we’ve seen would capture all kinds of possible fraud. (More)
Our main open questions and uncertainties are:
- We’re concerned that fraud is inherently a large risk in a program like New Incentives’, and it’s possible there are types of fraud that the monitoring indicators we have reviewed to date do not account for. (More)
- Although we’ve updated our analysis of New Incentives’ monitoring data for successive grant decisions, our last in-depth review of the methodology used to collect the data was in 2020 (shortly after the RCT). This is a less intensive level of review than for GiveWell’s other top charities, and it’s possible we have missed things or that aspects of our analysis are out of date.
- We do not have a strong understanding of how some of the monitoring indicators we use are collected and aggregated. It’s possible that this means that some aspects of our interpretation are incorrect.
2.2 Assessing the program’s impact over time
New Incentives maintains a database of over 100 monitoring indicators that it tracks to assess program performance. A summary of the main indicators we’ve reviewed is in this spreadsheet. In this report, we focus specifically on the indicators that we think are most important for understanding how the impact of the program might have changed since the RCT.
Overall, we think that the monitoring information we’ve seen from New Incentives is relatively high-quality and comprehensive. Our understanding is that many monitoring indicators are based on disbursement data gathered by New Incentives Field Officers during all disbursements, reducing our concerns about comprehensiveness and sampling bias. In the last systematic review we conducted of its monitoring methods (in 2020, available here), we assessed the risk of bias for most of the data collection methods it used as medium or low.
Retention through the vaccination schedule
Why is this important? Retention rates refer to the proportion of children enrolled in New Incentives’ program who receive subsequent vaccine doses at any point in time. If retention rates decrease over time, this might indicate that New Incentives' program is becoming less effective at incentivizing immunizations.
How we interpret the evidence
Our bottom line: Retention rates have been stable since the RCT, suggesting New Incentives’ program is still motivating for caregivers.
In more detail:
- New Incentives collects and shares data on retention rates for each of the 3 doses of the Penta vaccine, and both doses of the measles vaccine. Our understanding is that this data is based on entries submitted by Field Officers to a New Incentives app during the disbursement process. Field Officers record the vaccines that a child received during their visit, according to their Child Health Card. Our understanding is that this data should be recorded for every child.
- Overall, retention rates have stayed high, at levels consistent with those observed in the RCT. For example, retention for the first dose of the Penta vaccine was 96% in 2023 (compared to 93% in the RCT), and 87% (in 2023) vs 91% (RCT) for the first dose of the measles vaccine.
- The retention rate for the second dose of the measles vaccine has been considerably lower than other vaccines since it was introduced in 2021 (75% overall in 2023). We see this as unsurprising, since this dose comes last in the vaccination schedule (at 15 months). In January 2023, New Incentives decided to increase the incentive for the measles 2 visit from 500 to 1,000 naira in an attempt to increase vaccination rates. We have not yet analyzed how effective this change has been.
- A limitation in this evidence is that it cannot distinguish between the impact of New Incentives’ program and other factors on retention rates.
Repeat enrollments
Why is this important? Caregivers have an incentive to enroll their infants in the program multiple times in order to receive additional incentives. It is possible that some infants are enrolled in the program more than once, either at the same or different clinics. We think that infants are unlikely to benefit from receiving the same vaccine more often than it is scheduled to be received, so if we count children who are enrolled more than once as new children, then we will overestimate the impact of the program.
How we interpret the evidence
Our bottom line: We estimate that ~10% of all New Incentives enrollments are repeat enrollments, based on data New Incentives collects on BCG scarring. This has been increasing since the end of the RCT.
In more detail:
- New Incentives aims to prevent the same child being enrolled multiple times by limiting enrollment to the visit in which the child receives the BCG vaccine (with an exception described above), and (b) checking new enrollees for BCG scars, which indicate that the child received the BCG vaccine previously. This system is imperfect because most but not all children develop BCG scars and because scars take a couple of weeks to form, leaving a window for re-enrollment.
- Field Officers also look at children’s arms when they return for subsequent vaccinations and record the number of BCG scars on each arm using a New Incentives app. By comparing the proportion of children with scars to the proportion we’d expect to have scars (from independent data sources), we can estimate the proportion of children who have received BCG vaccination more than once. We use this as a proxy for children being enrolled in the program multiple times, since we’d guess that the main reason children receive more than one BCG vaccination in areas where New Incentives operates is to get additional cash incentives.
- Our best guess, based on a quick literature review, is that 90% of children who are vaccinated against BCG develop a scar that will be detected and correctly identified when checked for (details in footnote). By contrast, 100% of children checked by Field Officers at return vaccination visits in 2023 had 1 or more BCG scars. Based on this data, we estimate that roughly 10% of total enrollments in the program in 2023 were repeat enrollments (calculation in footnote). Scar formation is a biological process which we’d guess has substantial variation, and so our estimates of re-enrollment based on scar formation are moderately uncertain.
- While we see this proportion as moderately high, overall we are less concerned about repeat enrollments than other possible fraud risks. That is because (i) we think New Incentives’ processes for deterring repeat enrollments (such as the BCG scar check) are reasonable and (ii) we are already accounting for repeat enrolled children in our analysis of New Incentives’ costs, meaning we’re less concerned that this factor could lead us to overestimate the impact of the program.
Supply-side issues
Why is this important? If supply-side problems mean that children are not vaccinated when they arrive at New Incentives-supported clinics, the program might become less effective at boosting vaccination rates over time (e.g., if caregivers learn to expect that vaccines will not be available).
How we interpret the evidence
Our bottom line: The rate of stockouts has significantly risen since the RCT. This largely reflects supply problems with indirectly incentivized vaccines, especially the rotavirus vaccine, which was introduced in the routine immunization schedule in 2022.
Because indirectly incentivized vaccines are only a small portion (~20%) of the program benefits we model, we haven’t yet included an adjustment to account for this. We plan to keep monitoring this issue.
In more detail:
- New Incentives collects and shares a number of different supply-side monitoring indicators (we list 12 indicators in our latest summary of New Incentives’ monitoring data). We have paid most attention to data on the number and rate of vaccine stockouts during disbursement days. This data is recorded by Field Officers based on their observations, discussions with clinic health staff, and the clinic’s immunization tally sheet, and submitted through a New Incentives app through a daily survey called “Clinic Daily”.
- We would expect this data to be reasonably reliable. We would guess that clinic health staff might be incentivized to underreport stockouts, but that this probably does not apply to the same extent to New Incentives Field Officers (since they are not directly responsible for vaccine stock in the clinics where they work). However, we have not investigated this in detail and it's possible that the data might be biased in ways we haven’t considered. Supply-side data is available for a very high proportion of disbursement days (101% since the end of the RCT), increasing our confidence that this data provides a reasonably comprehensive picture of the supply dynamics in New Incentives-supported clinics.
- Overall, the data we have suggests that there has been an increase in supply-side issues over time. The rate of stockouts of any vaccine rose from ~8 per 100 disbursement days during the RCT to ~45 per 100 disbursement days in 2023.
- We see this as somewhat concerning. But as of December 2023, we have not included an adjustment in our cost-effectiveness analysis to account for the risk that this increase could weaken the impact of the program. This is largely because the increase in stockouts is concentrated heavily in indirectly incentivized vaccines, particularly rotavirus (which was only introduced in the Nigerian immunization schedule in August 2022). The stockout rate for directly incentivized vaccines rose more modestly, from ~7 per 100 disbursement days during the RCT to ~14 per 100 disbursement days in 2023. A high proportion (~80% as of December 2023) of the benefits of New Incentives’ program in GiveWell’s cost-effectiveness model come from directly incentivized vaccines, reducing our concern that this could significantly weaken the impact of the program.
- We plan to continue to monitor this issue over time and will consider an adjustment in our analysis to account for increasing supply-side issues, especially if we do not see a decrease in rates of rotavirus stockout over time (we would expect to see this decrease as the length of time since it was introduced in the immunization schedule increases).
Reduction in the value of the incentive because of inflation
Why is this important? Inflation might cause the real value of New Incentives' cash transfers to decrease, which may weaken their effectiveness as an incentive.
How we interpret the evidence
Our bottom line: We think the real value of the incentives has fallen somewhat since the RCT. We’re less concerned about this than other potential risks because retention rates have stayed high, suggesting caregivers are still motivated by the program.
In more detail:
- We have monitored this issue by (i) analyzing the real value of the cash incentives over time relative to their value in the RCT (based on data from the Consumer price index for Nigeria) and (ii) analyzing data shared by New Incentives on caregivers' transportation costs for health clinic visits. Our understanding is that the latter metric is based on Field Officers asking caregivers to report the one-way cost of getting to the clinic (not the roundtrip cost), and is asked for every disbursement.
- Our analysis suggests there was a substantial fall in the real value of the incentive from the RCT through 2022 (81% of its 2019 value in 2022). Since that time New Incentives has increased the nominal value of the incentives by 50% (from 4,000 naira in the RCT to 6,000 naira as of August 2023, more above). We would expect this to have mitigated the erosion of the incentives’ value to some extent, although we have only analyzed the real value of the incentive up to the end of 2022. New Incentives has informed us that as of November 2023 it believes the current incentive is less valuable than it was during the RCT in spite of this change, because of rapid inflation in Nigeria over this period. As of May 2024, New Incentives is planning to offer an additional 5,000 naira incentive at the measles 2 vaccination visit for caregivers whose infants completed the entire routine immunization schedule. This would bring the total incentive amount to 11,000 naira, which we believe would be roughly equivalent to the incentive value during the RCT.
- There has also been a substantial increase in the average cost reported by caregivers for transportation to the clinic (79 naira in March - December 2020 to 149 naira in 2023), and a slight decrease in the proportion of disbursements reporting 0 transportation cost (75% to 68% in the same period). The change in transportation costs is roughly in line with inflation over the same period, and so doesn’t affect our overall view of the real value of the incentive.
- To account for any residual risk that inflation is reducing caregivers’ willingness to come to vaccine appointments, we use a rough -5% adjustment in our cost-effectiveness analysis. Our concern about this is also reduced because New Incentives has maintained high retention rates over time (more above), suggesting its program is still motivating to caregivers. This analysis could be wrong because of lags in our analysis of retention rates (which only goes up to the end of 2023), or if a fall in the real value of the incentive meant that caregivers were less likely to enroll their children in the first place. This would not be picked up by the retention rate.
2.3 Data on vaccination rates
Why is this important? Our analysis of New Incentives’ program is based on extrapolating data from the 2017-2020 RCT to new locations, and accounting for changes to program delivery based on program monitoring data. We do not (as of December 2023) use direct data on vaccination rates in areas where New Incentives works to assess the impact of the program. Cross checking against this kind of data would be another tool to assess the direct impact of the program over time, and we think it’s likely we’ll start including this data in our analysis in the near future.
How we interpret the evidence
Our bottom line: New Incentives has begun to conduct vaccine coverage surveys in areas where it works. We plan to incorporate these into our analysis of the program in the future, but haven’t done so yet.
In more detail:
- In September 2021, New Incentives began conducting household surveys to assess baseline vaccination coverage before starting to work in new areas. New Incentives then conducts follow up surveys to reassess vaccination coverage after it has worked in each of the new areas for a year, and then every six months.
- We see the follow up surveys as having significant strengths and weaknesses. Their main strength is that they directly measure vaccination rates in areas where New Incentives works, providing additional evidence about the program’s impact on vaccine coverage outside experimental conditions. Their main weakness is that factors apart from New Incentives' program may influence the vaccination rates observed, and we do not observe changes that might have occurred in the absence of New Incentives' program. This means they are less useful for understanding the causal impact of New Incentives’ program. More information on how we expect to use this data can be found in our pre-analysis plan here.
- As of September 2023, we have only reviewed 12-month follow up survey results for four "cohorts" (groups of neighboring local government areas that are grouped together for surveying purposes), representing ~13% of New Incentives' scale at the time. We do not expect the results at 12 months to be very informative for the reasons described here, so we do not currently put any weight on these results in our analysis of the program.
- However, we expect to continue to receive additional results and continue to analyze the data over time. Ultimately, we hope to use these results as part of our analysis of New Incentives’ program in line with our pre-analysis plans.
2.4 Fraud monitoring data
Why is this important?
We think fraud is likely to be a major risk in any program that handles large volumes of cash. We think there are multiple ways in which fraud could take place and multiple actors who could potentially commit fraud (e.g., caregivers, New Incentives staff, and clinic staff).
GiveWell believes that some types of fraud are more worrying than others, depending on their implications for our analysis of New Incentives’ program. Two ends of this continuum are:
- Fraud that results in higher costs, without reducing the number of additional vaccinations: While this is a concern, we think that our analysis would account for this type of fraud (since the higher costs would be incorporated into our estimates of New Incentives’ costs per child enrolled). One example in this category is expense fraud (i.e., New Incentives staff could submit falsified or inflated expenses).
- Fraud that reduces the number of additional vaccinations, or suggests that our estimates of additional vaccination are inflated: An example in this category is a New Incentives staff member recording fake disbursements and keeping the cash for themselves. We see this type of fraud as particularly worrying because it would mean our analysis models the benefit of some vaccinations that didn’t in fact occur.
The most difficult fraud to detect might be fraud that involves collusion between multiple actors (such as clinic staff, New Incentives Field Officers, and other New Incentives staff). Fraud that includes collusion between multiple parties may be less likely, but could have a higher impact.
How we interpret the evidence
Our bottom line: The monitoring data we’ve seen does not show any major increase in fraud since the RCT. We remain uncertain about this because we think fraud is inherently a high risk for New Incentives’ program, and we don’t know if the monitoring data we’ve seen would capture all kinds of potential fraud.
In more detail:
Our summary of New Incentives’ monitoring includes 10 fraud prevention indicators (listed here).
These include the proportion of disbursements accompanied by a picture of the caregiver holding the cash,
the proportion of caregivers who report giving dashes (tips) to clinic or New Incentives staff when asked by Field Officers during the disbursement process,
and the proportion of audits reporting fraud by clinic or New Incentives staff.
Overall, the monitoring data we have reviewed suggests that fraud has remained at the (relatively low) levels observed in the original RCT. For example:
- The proportion of audits reporting fraud by clinic staff was approximately 4% in the RCT and 4% in 2023. The proportion of audits reporting fraud by New Incentives staff was lower, at approximately 1% in the RCT and 0.3% in 2023.
- The proportion of caregivers reporting that they gave tips was 0.1% in the RCT and 0.3% in 2023.
- Throughout the program, 100% of disbursements logged on New Incentives’ platform have been accompanied with a photo of the caregiver holding the incentive.
In 2020, we conducted a review of possible fraud scenarios and determined at the time that the risk of fraud not accounted for by New Incentives’ existing procedures was low. We account for the possibility that fraud is occurring that isn’t otherwise accounted for in our cost-effectiveness analysis with a rough adjustment for “false monitoring results” (-2% as of April 2024). This accounts for the risk that the data we have reviewed is biased in some way, although it does not distinguish between fraud and other concerns about the data collection method. This adjustment aligns with our overall impression (based on many conversations with New Incentives over time) that it has strong procedures for identifying and responding to fraud.
However, we still have a number of open questions and uncertainties about this issue. We’re concerned that fraud is inherently a large risk in a program like New Incentives’, and it’s possible there are types of fraud that the monitoring indicators we have reviewed to date do not account for. For example:
- Field Officers might record fake disbursements to friends and family members and pocket the cash. We don’t know if there are ways this could avoid detection, and think it’s possible that this could be missed by New Incentives’ remote console agents so long as these disbursements took place at a clinic and all the required elements of the caregiver photo were submitted appropriately.
- Different New Incentives staff members or teams could collude to defraud the program (e.g., auditors colluding with Field Officers). We’re uncertain how effective New Incentives’ current procedures would be at catching this type of fraud.
- We would guess that the data on dashes (tips) is underestimated. The data is collected based on caregiver reports to Field Officers during the disbursement process, and we would expect caregivers to feel some pressure to report inaccurately (e.g., since caregivers may not think that this report would be confidential and may get back to clinic staff). Field Officers are also unlikely to report tips that they receive themselves. We do not currently see this as a significant threat to the program because we’d expect this kind of fraud to be taking place at a relatively low level, and for any worsening trends to show up in our monitoring of retention rates (which have stayed relatively consistent over time).
We’re also concerned that our analysis may cover risks that we and New Incentives have anticipated. One continuing concern is that there may be additional risks that we have not anticipated.
We’re planning to investigate this question in more detail in the future. We expect to update our 2020 review of the program’s fraud risks and think through scenarios that might be missed by New Incentives’ current procedures. We may also update our adjustment for “false monitoring results” in light of this analysis.
3. Qualitative assessment
In theory, our funding recommendations are maximizing for one thing: total improvement in well-being per dollar spent. This is what our cost-effectiveness estimates intend to capture.
In practice, there are costs and benefits that we do not observe and are not estimated in our models. We make qualitative assessments to account for these unmodeled costs and benefits. We use these assessments alongside our cost-effectiveness estimates to inform our funding recommendations.
As one tool for thinking through and communicating about impressions we have that aren't captured in our cost-effectiveness estimates, we assess each organization on eight dimensions on a four-point scale (“Stands out”; “Relatively strong”; “Average”, “Relatively weak”). We believe our top charities are exceptional relative to the majority of organizations and so these assessments are intended to capture differences among GiveWell top charities, rather than absolute rankings among all charitable organizations.
Overall, our qualitative assessment of New Incentives is very strong, even compared to our other top charities. We believe it stands out for its focus on maximizing cost-effectiveness and its dedication to identifying, responding to, and being transparent about issues it faces. Our latest assessment of New Incentives (for 2023) is in the table below.
| Dimension | What does this capture? | Assessment |
|---|---|---|
| Responses to our questions | When we ask the organization a question, do its answers generally either indicate that it has thought through the question before or show us why getting an answer is not important to understanding its work? | Stands out |
| Prioritization discussions | Do the organization's explanations about how it allocates funding among different locations and program participants seem to be aimed at maximizing its impact per dollar? | Stands out |
| Self-improvement and attitude toward mistakes | Does the organization proactively share information with us and publicly about mistakes it has made? | Stands out |
| Role in field | Is the organization producing research aimed at informing policymakers or other implementers? Does it participate in global conversations about its field of work? | Average |
| Responsiveness | Does the organization send us information by mutually agreed-upon deadlines? Is it responsive to our emails? | Stands out |
| Giving us feedback | Does the organization catch our mistakes and let us know, thus improving our research? Does the organization make useful suggestions for how we could improve our research process and cost-effectiveness models? | Relatively strong |
| Quality of information shared | Have the documents that the organization has shared with us contained significant errors? Has the organization told us things that were inaccurate? Has the information provided been easy to interpret and use? Have the organization's projections of when it would achieve its goals generally been accurate? | Relatively strong |
| Incorporating feedback from participants and last mile providers | How does the program collect feedback from program participants and from program implementers, i.e. those directly delivering the program? How does the program incorporate feedback to improve service delivery? | Relatively strong |
Some of the main factors informing our assessment are:
Self-improvement and attitude towards mistakes (stands out)
- We value working with organizations that proactively bring negative updates to our attention. We think that New Incentives is highly transparent in informing us about challenges it is facing, even when it knows these may affect our evaluation of the program.
-
For example, in 2023, New Incentives raised an unprompted concern about an increasing proportion of children older than 6 months at enrollment (this is permitted under the program’s eligibility criteria, but is intended to be an exception). New Incentives had reviewed the underlying data and found that some of these children were meaningfully older (e.g., 1-2 years) and that their date of birth was misreported. If a high portion of children were getting vaccinated later than recommended, we believe they would be unlikely to be receiving the full benefits of vaccination, because mortality for several of the diseases is particularly high during the first 6-12 months of life. As a result, we would potentially consider down-adjusting our cost-effectiveness estimates. In this case, New Incentives immediately started working to understand the drivers of this trend and implementing protocols to address it.
- In September 2023, New Incentives informed us that it had found an error in its monitoring data on duplicate enrollments. New Incentives runs caregiver photos through a face-matching service to identify potential duplicates, and reports on the rate of photos flagged as duplicates at the same clinic, at clinics within a 10km vicinity, and at clinics within a 20km vicinity. New Incentives discovered the rates of duplication within 10km or 20km were underreported (due to a coding error). Although underreported rates of duplicate enrollment would imply the program was reaching fewer children, New Incentives brought this error to our attention and kept us updated as it began rerunning the data for the affected time period.
Prioritization discussions (stands out)
We think that New Incentives shares GiveWell’s focus on evidence-based decision making and maximizing the cost-effectiveness of its program.
- An important question when we have considered funding expansion grants for New Incentives is whether the program will be cost-effective in new locations. In discussions about this, we have found New Incentives to be a very thoughtful partner, and we have always been able to come to full agreement efficiently on where New Incentives expands to. This has sometimes involved us demonstrating to New Incentives that the funding could be used more cost-effectively elsewhere (either within New Incentives program or, through making a smaller grant, through another program). At other times it involved New Incentives convincing us that we were missing something important in our assessment.
- Our impression is also that New Incentives is focused on decreasing the cost of the program per child enrolled to an unusual degree among GiveWell top charities. This is reflected in a consistent decrease in our estimates of New Incentives’ cost per child enrolled since the RCT, from ~$39 in 2020 to ~$17 in 2023. We think that a substantial part of this decline is due to external factors, e.g., the fall in the value of the Naira relative to the dollar but we still think New Incentives’ focus on cost-effectiveness has played a role in keeping costs low.
Incorporating feedback from participants and last mile providers (relatively strong)
In general, we would expect that organizations will deliver more effective programs if they have mechanisms for gathering and acting on feedback from program participants.
New Incentives has a large staff footprint in the areas where it works in northern Nigeria, providing frequent opportunities for touchpoints between New Incentives staff and program participants. It also has systems for gathering feedback from program participants and stakeholders on its program (e.g., during immunization sessions and awareness-raising meetings in communities), and acting on the results.
For example:
- Before making the decision to change the incentive structure for its program in 2023, New Incentives gathered feedback from caregivers and other partners (e.g., state health officials). The positive feedback it received on this change informed its decision to go ahead.
- New Incentives’ enrollment criteria include an exception, whereby children who received the BCG vaccine before New Incentives began operating in a given area can still be enrolled in the program in some circumstances (more above). New Incentives reports that this decision was based on feedback from both caregivers and clinic staff that it would be an unfair penalty on caregivers whose children were vaccinated before New Incentives expanded to that location.
We would guess that these processes allow it to gather better data on its program to improve delivery, although we have not yet seen detailed information on the methodology used to collect this feedback.
Other factors
New Incentives has rapidly scaled this program since it began in 2017. The number of children enrolled has risen from ~90,000 children in 2019 to ~1.39m between January and November 2023 and its annual spending has risen from ~$2.2m (March - December 2020) to ~$23.4m (Jan - Dec 2023). We were concerned that program quality could be negatively impacted by rapid scaling, or that unexpected problems could arise. While this continues to be a concern as New Incentives expands further, the evidence we have seen so far suggests that New Incentives is still delivering the program at a similar level of quality as in the randomized controlled trial (with the exception of an increase in the rate of vaccine stockouts, see our
monitoring reviewfor further details). This increases our confidence that the program is still being delivered well, and is still motivating caregivers to receive additional vaccines.
As it has scaled up, New Incentives has also adapted its program over time. For example, our understanding is that it has introduced an increasing number of anti-fraud checks (e.g., a face recognition process and a requirement that staff submit attendance photos to verify that they are present at the clinic as intended). While we have not individually assessed each of these changes for its impact on the program, they contribute to our overall impression that New Incentives is dedicated to trying to tackle problems as they emerge and maximize the impact of the program.
4. What do you get for your dollar?
GiveWell recommends interventions and organizations that we believe are cost-effective in the sense of saving or improving lives as much as possible for as little money as possible. We summarize the full reasoning behind our cost-effectiveness analysis for New Incentives in our separate report. In summary, as of September 2024, we think:
- It costs roughly between $1,500 and $6,000 to avert a death through New Incentives’ program.
This equates to being 10 - 42x as cost-effective as direct cash transfers (GiveWell’s benchmark for comparing different programs). This is because:
- Based on an RCT of New Incentives’ program, we believe the program leads to a large increase in the number of children receiving routine vaccinations (9 to 18 percentage points depending on the location).
- There is strong evidence that vaccination leads to a substantial reduction in child mortality (~50% reduction in vaccine-preventable disease).
- We believe that vaccination probably leads to other benefits like reduced mortality at older age groups and increases in income later in life.
See our separate report for more details.
