Evaluating whether policies are cost-effective

Ask: NUS Economists
By Ivan Png, Published The Straits Times, 9 Mar 2016

Q How can the Government evaluate the cost-effectiveness of economic and social programmes?

A How much does the Gifted Education Programme (GEP) contribute to a child's education? How does bail affect the likelihood that defendants in criminal cases abscond? How does treatment for prostate cancer affect the long-term health of patients?

For effective design and implementation of government policy, it is essential to evaluate the policy. However, evaluating policies can be challenging. For instance, as any parent knows, evaluating the GEP is not simply a matter of comparing the achievements of children who attended the GEP with those of children who did not.

The Ministry of Education selects its students from the cream of Primary 3 pupils. Hence, GEP students should perform better than other students, and so, directly comparing GEP with other students would not be meaningful. (Full disclosure: I did not benefit from the GEP.)

A well-established method of programme evaluation is the randomised controlled trial (RCT). Recruit a large number of subjects with similar characteristics, and randomly assign some of them to treatment, while doing nothing to the others. At the end of the trial, compare the outcomes for the treatment group with the outcomes for the control group.

But how many parents would allow their children to participate in an RCT in which some children would be randomly admitted to the GEP while others would be kept in the non-GEP stream? Would the public and the Ministry of Law agree to randomly allow some defendants to post bail and others to be detained until trial? Would it be reasonable to deny treatment to 60-year-old men with symptoms of prostate cancer in the interests of medical research?

Evidently, in many policy situations, it is unethical, unacceptable or impractical to carry out an RCT. Even so, it may still be possible to evaluate the policy.

Over 55 years ago, educational psychologists Donald Thistlethwaite and Donald Campbell introduced the technique of regression discontinuity analysis as an alternative to RCTs in programme evaluation.

It's easiest to understand the concept of regression discontinuity in the context of the GEP. The Ministry of Education selects students for the GEP on the basis of tests in English Language, Mathematics, and General Ability.

The essence of the regression discontinuity analysis is to compare the borderline children - those with scores just above the cut-off with those with scores just below the cut-off. The children who score just above the cut-off are the "treatment group", while those who score just below the cut-off are the "control group".

These borderline children are almost identical on the objective selection criteria. Hence, it is reasonable to attribute any difference in their subsequent achievements to the GEP. If the students who just qualify for the GEP go on to contribute more to public service or engage more actively in entrepreneurship than those who just failed to qualify, then it is reasonable to attribute the difference to the GEP.

Although clearly non-experimental, regression discontinuity is almost as good as the RCT as a method for causal inference and evaluation of policy.

Similarly, in the context of bail, suppose that judges apply some scoring system to decide whether defendants should be allowed to post bail. (If judges do not apply an explicit scoring system, perhaps they should.) Then, to determine the effect of bail on absconding, researchers should compare the rate of court appearance of defendants whose scores just exceeded the cut-off with those just below the cut-off.

And, in the context of prostate cancer, suppose that urologists apply a scoring system to decide whether patients with symptoms of prostate cancer should be treated. Then, to determine the effect of treatment, researchers should compare the long-term health of patients whose scores just exceeded the cut-off with those just below the cut-off.

As our Government increases spending - from an average of 13.7 per cent of GDP in fiscal years 2010-14 to 17 per cent in the current fiscal year - it is increasingly important to systematically evaluate government spending. Where RCTs are not ethical, acceptable or practical, regression discontinuity analysis may be an appropriate method.

I have illustrated how to apply regression discontinuity analysis to evaluate the GEP, bail system, and treatment of prostate cancer. It is easy to see how the Government could apply the same method to evaluate the effectiveness of business subsidies on innovation, tax incentives on investment, housing subsidies on home ownership, and even national service on subsequent career success, to list just a few of the possible applications.

The writer is Distinguished Professor at NUS Business School and the Department of Economics at the National University of Singapore.