Improving Administrative Data at Scale: Experimental Evidence on Digital Testing in Indian Schools (2024)

Search for other works by this author on:

Article history

Abstract

There is a widespread ‘learning crisis’ in developing countries where, despite substantial increases in school enrolment and average years of schooling, student learning remains very low (Pritchett, 2013; Glewwe and Muralidharan, 2016). Pivoting education systems to prioritise learning, and not just access, has therefore emerged as one of the most important development challenges that faces low- and middle-income countries today (World Bank, 2018).

One important element in this challenge is to collect reliable official measures of student achievement at scale. These are necessary to underpin a common policy understanding of the level and trends in achievement, for prioritising support to those schools and students most in need, and for instituting reforms that target information or accountability. Unsurprisingly, instituting official data systems focused on student testing has featured prominently in the policy agendas of national governments and international aid agencies.¹ Yet, especially in settings of weak governance where such reforms might be most necessary, large-scale learning assessments are often undermined by wide-spread student copying and misreporting by teachers. This calls the usefulness of such assessments into question and illustrates how unreliability of official data on student learning acts directly as a constraint to effective policy making.²

In this paper, I evaluate the potential of digital testing to address the challenge of data integrity at scale. Digital testing on computers or tablets may improve data integrity in several ways: it is possible to make student copying harder by providing different questions to students and designing the user interface appropriately, automatic grading removes the discretion of individual teachers to inflate grades and detailed item-level information, which is not typically available with manual grading, allows for a more detailed scrutiny of cheating. However, digital tests also carry risks: the unfamiliarity of computerised tests in this setting, combined with logistical constraints and the lack of appropriate infrastructure, may lead the assessments to be uninformative or infeasible to implement. Despite increasing policy interest, however, there is little direct evidence on reliability of digital testing for measuring student learning in low- and middle-income countries (LMICs).

I study this question using a large-scale field experiment covering grade 4 students in over 2,400 schools in one district in Andhra Pradesh state. The experiment randomly allocated schools to two treatment arms. In the benchmark case, covering 768 schools, school staff administered paper-based tests to students in three subjects; these assessments used multiple test booklets and were graded centrally to replicate the ‘best-case’ scenario for official tests in India. In the intervention arm, covering 1,694 schools, students took the same tests, but on tablets brought to the school by a cluster resource person. These tests were designed to be diagnostic and carried no formal stakes for students or teachers. Both paper and tablet assessments were conducted by government officials, as they would be in any census-based assessment, and not external staff. Thus, this study represents an evaluation at scale likely to identify a policy-relevant treatment effect (see Muralidharan and Niehaus, 2017; Vivalt, 2020).

I measure distortion in two ways. First, an independent research team conducted an externally proctored paper-based retest to serve as an external benchmark in 117 schools. Absolute differences between the two assessments in the percentage of correct responses, to the same test items by the same students, provide a direct measure of cheating. Second, I use the item-level data on students’ responses in the official assessments to implement the procedure used by Angrist etal. (2017) (ABV hereafter) to flag classrooms with potential manipulation in Italy.³ This allows me to compare the magnitude of cheating with the international literature on detecting cheating, but also, importantly, to provide direct validation of such procedures for use in a very different context. Since statistical procedures for detecting cheating are much more scalable than retest-based audits, establishing their validity is an important step for their use in such settings.

There are three main results. First, there is evidence of substantial distortion in business-as-usual tests in this setting. Using the ABV procedure, 38%–43% of classrooms in the paper-based testing arm are flagged for cheating, a prevalence rate three times higher than documented by Battistin etal. (2017) in southern Italy. In the retest sample, students score 16–20 percentage points higher, on average, in the teacher-administered paper-based tests than in the independently proctored retest. Second, distortion is substantially reduced in tablet-based testing. In contrast to paper-based tests, only 2%–5% of tablet-based test classrooms are flagged as potentially cheating by the ABV procedure. The difference between the official assessment and the retest is much smaller in the tablet-test arm, and not statistically distinguishable from zero in most cases. Third, I find substantial correspondence between direct audit-based measures of distortion and the ABV procedure, which indicates that procedures to detect cheating could be scaled up effectively with item-level data, even where independent audits are not possible or too expensive.

This paper contributes to three areas. First, it complements a large literature that evaluates interventions to improve student learning in developing countries (see Glewwe and Muralidharan, 2016). Many of the most promising reforms in this literature take reliable data on test scores as a pre-requisite. A prime example is performance-based pay in education: while this has been shown to have positive effects in large-scale experimental studies in India, East Africa and China (Muralidharan and Sundararaman, 2011; Loyalka etal., 2019; Mbiti etal., 2019; Leaver etal., 2021), in each of these experiments, the test data triggering payments were collected either directly by research teams or their NGO partners. Similar concerns arise with, for example, report-card interventions using high-quality researcher-administered assessments where experimental studies report large gains (Andrabi etal., 2017; Afridi etal., 2020). Ensuring data integrity can often be a binding constraint to scaling these reforms; my results suggest that digital testing could reduce this risk sharply, even in settings of weak governance.

Second, it adds to a body of work that evaluates technology-led solutions for system-wide public sector reform in developing countries. Recent papers have, for instance, looked at technology-aided solutions in procurement (Lewis-Faupel etal., 2016), social security payments (Muralidharan etal., 2016; Banerjee etal., 2020) and voting (Fujiwara, 2015). In these applications, technology plays three crucial roles: (a) it improves the timeliness, standardisation and detail with which information becomes available, allowing for timely verification, (b) it reduces the risk of manipulation by circumventing corrupt actors and (c) it allows for scale-up with fidelity. Tablet-based assessments play the same role here: they circumvent potential grade manipulation and make copying harder by randomising questions, they make detailed item-level data available for analysing suspicious response patterns and they may have the potential for universal scale-up.

Third, this paper also extends, in both methods and scope, an extensive literature that has examined manipulation of test scores by teachers or cheating by students.⁴ Specifically, I provide the first direct audit measures of distortion, in a new setting, and present a large experimental evaluation of potential reform. I also directly validate the efficacy of indirect procedures to flag suspicious response patterns to detect cheating as opposed to audits. Results in this paper on the use of technology are similar to Borcan etal. (2017), who documented a sharp reduction in cheating with the use of CCTV cameras in Romania (in a much higher-stake setting). Substantively, my results highlight that the banality of cheating in some settings forces a shift in focus from trying to detect (relatively rare) cheaters, such as in Jacob and Levitt (2003), to reducing manipulability in the system as a whole. In Section4, I further discuss the implications of these results for policy and for the potential use of such data in research.

1. Study Design

1.1. Background and Setting

This study is based in the state of Andhra Pradesh, which had a population of 52.8 million in 2022. Education outcomes in this state are better than the all-India average, but still poor in absolute terms: in 2018, 59.7% of grade V students, across private and government schools, could read a grade II level text and 39.3% could divide (ASER, 2019).

The Government of Andhra Pradesh wanted to test and implement a report card intervention informing parents about learning levels of their children, and average achievement in local schools.⁵ At scale, this requires the underlying information collected by government systems to be reliable. Thus, the government authorised a large-scale pilot evaluation to assess if tablet-based assessments could, at scale, measure learning levels more accurately than traditional paper-based tests. I helped design the evaluation in collaboration with the Central Square Foundation. The experiment was conducted in February 2019 in the Prakasam district (see Online Appendix Figure A1). This district is close to state-level averages across several educational indicators (ASER, 2019).

1.2. Experiment Design

1.2.1. Intervention

The evaluation focused on children in primary schools, the principal focus of policies targeting foundational learning. All schools with at least five students in grade 4 were assigned to administer either paper-based or tablet-based tests. All grade 4 students present on the day were tested in mathematics, Telugu (language) and English.⁶ Both treatment arms used centrally designed test papers that were identical in test questions and ordering across treatment arms.⁷ All questions in the test booklets were multiple choice items intended to capture a wide range of variation. Three test booklets (‘sets’) were created in each subject. Each booklet contained thirty questions, of which twenty-four were common across all sets (and underlie all analyses here).

In schools assigned to paper-based testing, booklets were sent with clear instructions for schools to administer the tests and for the answer scripts to be returned to the Mandal Education Office (the administrative unit above schools) for grading. This reference mode of testing, with multiple sets and external grading, resembles the best-case scenario for large-scale paper-based assessments in India.

In the digital testing arm, the tests were staggered across schools. Tablets were taken to the school by a cluster resource person (CRP). The CRP is a frontline official who acts as a link in program delivery between schools and the (higher) education bureaucracy. Each student was given an individual tablet to work on. The test booklet for each subject was decided by the software directly. In both treatment arms, students could use their own scrap paper for calculations if they wanted. All schools were notified of testing at the same time.⁸

In comparison to paper-based testing, tablet-based tests as administered here may reduce manipulation through several channels. First, they make student copying harder since students only see one question at a time and it is much harder to copy from the booklets of nearby students. Second, for the same reason, it is harder for teachers to help all students, or to see whether they have answered correctly (and provide the right answers). Teachers also cannot retrospectively erase and correct answers after the test has ended. Third, the mode of administration, since it required the CRP to take the tablets to the school and collect them after the test, ensured that external observers were actually present at the time of testing to invigilate.⁹ My goal here is not to distinguish between these potential channels, but rather to evaluate whether tablet-based tests, as delivered at scale by government officials, reduce distortion. This composite policy effect includes both the direct effects of the technology in inhibiting cheating and the indirect effect from complementing existing monitoring capacity in the system. Program implementation by government staff reduces the concerns of external validity that accompany high-fidelity implementation by motivated NGOs or research teams (Bold etal., 2018; Vivalt, 2020).

1.2.2. Data

We have data from three sources. First, we have administrative data on enrolment, staffing, infrastructure and monitoring of all recognised schools in the state, obtained from the Unified District Information System on Education (U-DISE). These data were used to construct the sampling frame for the randomised experiment and will be used for heterogeneity analyses.

Second, central to the analysis in this paper, we have item-level data from the official tests administered to 38,857 students in 2,443 schools. Of these, 12,741 students were tested in the paper-based arm and 26,116 in the tablet-based arm.

The third source of data is a retest-based audit that serves as an external benchmark to detect cheating. We randomly sampled 120 schools, spread equally across the paper- and tablet-based testing arms, to retest students using a traditional paper-based assessment, but with external proctoring and supervision by the research team. Schools were only informed about 1–2days in advance about the independent test. The anonymity of results was guaranteed. This retest was conducted within 2–3 weeks of the original assessment.¹⁰

1.2.3. Randomisation and experiment validity

Randomisation was carried out at the level of an academic cluster, which typically covers multiple villages/urban wards, to keep the mode of testing unchanged within a single educational market. Out of 284 clusters, 196 were assigned for tablet-based testing and the remaining 88 to paper-based testing.¹¹ Thus, the experiment is an ‘evaluation at scale’ in all three respects highlighted by Muralidharan and Niehaus (2017), i.e., representative of large populations, studying implementation across a large number of treated units, and studying implementation at a large unit (to estimate effects that are net of spillovers). The randomisation was stratified by sub-districts (mandals) and mandal fixed effects are included in all regressions.

Our design had one pre-determined deviation from the initially assigned status. In rare cases, academic clusters do not nest villages/urban wards. If schools in the same village/ward were assigned to different treatment arms, all schools in the village were reassigned to be in the same arm; 191 schools (out of 2,462) were reassigned. In the final sample, 768 schools were assigned to paper-based testing and 1,694 to tablet-based testing. Observed characteristics are balanced between the testing arms (Table1) by initial assignment (columns (1)–(4)), final assignment (columns (5)–(8)) and in the retesting sample (columns (9)–(12)).

2. Results

2.1. Comparisons between Paper- and Tablet-Based Tests

Students who were tested on paper score much higher—about 28 percentage points higher in mathematics and English, and 21 percentage points higher in Telugu—than students who were tested on tablets (Figure1(a), panel A of Table2). Relative to the distribution of test scores in tablet-based tests, these differences equal about 1 SD in each subject.¹² The resulting distribution of achievement in paper-based tests is very negatively skewed, with substantial ceiling effects, whereas tablet-based tests provide a more well-distributed measure of achievement. Figure1(b) shows for each test item that the proportion of students answering each item correctly is substantially higher in paper-based tests. Thus, conclusions about student achievement may differ drastically if administering the assessment in one mode versus the other.

2.2. Direct Audit Measures of Cheating

The primary outcome of the experiment is the magnitude of cheating, which is not directly observed through the official tests alone.

I first use the audit as an external benchmark and measure the discrepancy between the two assessments in the percentage of correct responses. Figure2 compares results in the two treatment arms with the retest. For all items, paper-based official tests significantly over-report student achievement, whereas tablet tests correspond much more closely with the retest (Figure2(a)). Aggregating scores at the student level, differences between the audit- and tablet-based tests are centred close to zero (indicating little deviation on average), but, for paper-based tests, scores are clearly higher in the official tests than the retest (Figure2(b)). Paper tests exaggerate performance by 16–20 percentage points in each subject (panel B of Table2). Point estimates suggest a negative effect of 2–4 percentage points in the tablet-based assessment, plausibly reflecting lower student familiarity with digital tests (only statistically significant in Telugu).

This is a conservative measure of distortion because answering the same question may be easier the second time around, thus leading average student performance to be higher in the retest (‘learning effects’).¹³ We did not conduct the external tests before the official assessment for logistical reasons and to ensure the confidentiality of the questions being used. We were also concerned that, having recently answered these questions in an external test, students may be deterred from cheating in the official tests knowing that external validation measures exist.

2.3. Detecting Cheating Based on Internal Validation

While the audit method provides a direct measure of distortion, it is costly in time and effort, cannot be estimated for all schools and is infeasible to administer in all settings. Thus, I investigate a second measure that only requires access to item-level data from the official tests. I use the method used of Angrist etal. (2017), which is adopted from official practice to flag classrooms with suspected cheating in the INVALSI exams in Italy. The algorithm proceeds in three steps. First, item-level data are used to generate four summary statistics at the classroom level: (a) the mean percentage correct, (b) the variance of percentage correct, (c) the proportion of non-missing answers and (d) an index of hom*ogeneity of answer options in the classroom. Second, these data are reduced to two principal components. Finally, schools are classified into clusters using a hard k-mean clustering approach, flagging the cluster with high mean, low variance, few missing responses and hom*ogeneous responses to individual test questions. I pool the item-level data for both modes of assessment and run this algorithm separately by subject. Details of the procedure and characteristics of resulting clusters are presented in Online Appendix B.

The ABV procedure is particularly suitable in our application. First, the INVALSI assessments are similar in scope and implementation to regular paper-based assessments here: adopting the same procedure to flag cheating allows me to benchmark levels of distortion in this setting to those documented elsewhere. Also, the ABV procedure does not suffer from concerns of potential learning between assessments. Second, and as importantly, this procedure does not posit a common item response model across students in the paper- and tablet-based assessments. This is in contrast to commonly used indices including the |$\omega$|-index, K-index and GBT index (see Romero etal., 2015; Martinelli etal., 2018). This assumption is likely to be inappropriate in our setting: students with the same underlying ability may score differently on paper and tablet tests due to a difference in the underlying item response function and it is also likely that some cheating may be rationalised through a higher estimated level of achievement.¹⁴ Finally, the ABV procedure is computationally simpler and more transparent for non-specialist audiences than alternative IRT-based indices; in addition to already being used by policy agencies elsewhere, these features make it a promising candidate for adoption in this setting.

Panel C of Table2 compares suspected cheating in digital and paper-based tests, as administered at scale. Between 38% and 43% of classrooms in the paper assessment are flagged as suspicious using this approach. This is much higher than the figuresreported by Battistin etal. (2017) in southern Italy, where the proportion of suspected manipulators was assessed to be 11%–16%. In contrast, only 2%–5% of the classrooms are similarly flagged in the tablet assessment.

I use the audit data to validate the ABV procedure against the direct retest-based measure of cheating. Figure3 shows substantial agreement, on average, between the two methods in flagging cheating: in classrooms not flagged by the ABV procedure, there is little discrepancy between the audit and the official test; in flagged classrooms, students score 17–21 percentage points lower in the audit than their original score. The agreement between alternative procedures using different data suggests that the ABV method is reliable for analyses presented here and potentially for scaled-up implementation. This direct validation is important because a major concern with statistical procedures to detect suspicious patterns is how to assess the risk of type-I and type-II errors in the absence of further rounds of testing. Results are often sensitive to the choice of index.¹⁵ Furthermore, without an external benchmark, especially in high-prevalence settings, such indices may only differentiate between excessive and moderate cheating (rather than measure absolute levels).

2.4. Correlates of Cheating

I study heterogeneity in the difference in percentage correct between the tablet- and paper-based assessments by three observed covariates, each of which may affect cheating. The first dimension is whether the school is private. Since private schools are subject to market pressure, they may face stronger pressure to ensure better test results. The second dimension is the number of students enrolled in grade 4, which may affect cheating through multiple channels: teachers’ effort costs of proctoring may be higher for larger classrooms; alternatively, if cheating is costless for students, the probability that at least one student in a class knows the right answer (and can share) rises in larger classes. The third covariate is the reported number of visits by officials (which could be taken as an indicator of official accountability; Muralidharan etal., 2017).

The results are presented in Online Appendix TableA2. In all three subjects, the difference in performance in the paper and tablet tests is greater in private schools than in government schools. The magnitude of this difference is meaningful: whereas government school students perform between 18–24 percentage points worse on tablets than on paper, this difference is larger by 3–8 percentage points in private schools. It also appears that cheating is greater in larger classrooms, although the magnitude of this relationship is much more modest: an increase in the number of students by ten only increases cheating by |$\sim$|0.8 percentage points on average. Finally, there is little evidence that the number of inspections is correlated with lower cheating.

2.5. Intervention Costs and Further Considerations

The results presented above suggest that tablet-based assessments nearly eliminated cheating. Decisions on broader policy adoption, however, face at least two important further considerations.

The first relates to intervention costs. The main additional costs in tablet-based testing come from the hardware: personnel costs are unchanged since teachers and cluster resource staff would also be required to dedicate time to testing in paper-based testing. This experiment used |$\sim$|3,500 tablets to test grade 4 students in the district over ten working days. The marginal cost of testing additional students in the same school is modest. Extending this testing period, with staggered testing across schools, I estimate that 5,000 tablets would suffice for testing around 200,000 students over a ten-week period. Low-specification tablets currently cost about USD 60–100. With a three-year depreciation period, this translates into a per-test cost of about 50–80 cents at scale. This expenditure is feasible even within current education budgets in LMICs. Importantly, these are not incremental costs, but rather displace substantial effort and expenditure related to manual grading, data entry and record-keeping of paper-based tests, and printing. Thus, tablet-based tests may be feasible and cost effective at scale. More granular data also make it easier to detect suspicious responses. Adaptive testing may also make assessments more informative.¹⁶

The second, and important, caveat relates to long-run effects: as a one-off trial, the experiment here does not reflect any long-term adaptation by agents. Such responses have been shown to undermine previous reforms. For instance, Banerjee etal. (2008) and Dhaliwal and Hanna (2017) documented how technology-aided public sector interventions to reduce health worker absenteeism were undermined shortly after introduction, with the implicit agreement of higher levels of bureaucracy. Such complete reversal is not universal though: Muralidharan etal. (2016) presented an example of a reform that persisted and, in a different state, Banerjee etal. (2020) documented a temporary reversal of the corruption-reducing intervention they studied followed by a nationwide scale-up. The long-term efficacy of tablet-based tests, as with any intervention, can only be assessed once it has been in operation for a sustained period. This concern is, of course, not only specific to large-scale policy experiments.¹⁷ Thus, following any such reforms, data integrity will need to be evaluated repeatedly in representative samples.

3. Discussion

3.1. Implications for Policy

The pervasiveness of cheating I find suggests that policies that raise the costs of cheating across the board may be more promising here than those focused on detecting and punishing individual cheaters (which are appropriate with lower prevalence, e.g., Jacob and Levitt, 2003). Digital testing might be an especially promising candidate to scale (and to continue to monitor effectiveness of in sub-samples over time).

These results are of immediate policy relevance: in India, the new National Education Policy mandates a census of student achievement in selected grades alongside a proficiency-based goal of ‘universal foundational literacy and numeracy’ (Government of India, 2020). Similar large-scale assessments and proficiency targets are also recommended as the cornerstone of education reforms to raise student achievement in LMICs by the World Bank and other multilateral aid agencies. However, if assessments do not reflect true achievement, they are unlikely to fulfil the expectation of governments and multilateral agencies to catalyse action on the ‘learning crisis’.¹⁸ Overall, these results suggest that simultaneous enabling reforms to ensure data integrity will need to underpin any attempts to use test score data as a basis for large interventions in similar settings.

3.2. Implications for Research

The widespread adoption of large-scale assessments in many developing countries holds the prospect of transforming education research in these settings, as it has done in Europe, the United States and Latin America (see Figlio etal., 2016). It could substantially cut the costs of education research, enable many more retrospective analyses of policy changes using quasi-experimental methods and improve sample coverage and precision substantially.

Unfortunately, at least in India and similar settings, my results are also sobering for using data for research. Cheating-induced distortion is more problematic than classical measurement error: not only is the level of achievement distorted, this distortion differs systematically by school type and, plausibly, may vary over time and by the characteristics of individual students.¹⁹ It is unlikely that all correlates of manipulation can be adequately captured in large-scale data and some of these correlates may be likely to interact with whatever policy or attribute that the researcher is trying to study. Thus, in settings with weak governance, using these assessments as the mainstay of education research may be inadvisable even with low stakes.²⁰ Improving the integrity of these data systems is thus also important on purely scientific grounds.

3.3. Limitations

This study, demonstrating that digital testing is implementable by LMIC government systems at low cost and can substantially improve data integrity, is best considered as a ‘proof of concept’. There are, however, a few important caveats.

The first of these relates to external validity. My results are likely to generalise to schools in other Indian states where the magnitude of cheating seems to be as high (Kingdon and Muzammil, 2009; Singh, 2020), and the organisation of the education system is also similar. Although India is an important setting by itself—with 264.4 million children enrolled in K-12 education, the world’s largest education system—this study cannot speak to generalisability elsewhere. Estimates of the prevalence of cheating are not, to my knowledge, available for other low- and lower-middle-income settings in South Asia or sub-Saharan Africa that have been the principal focus of policy concern about a global learning crisis.²¹ If the prevalence of cheating is much lower then digital testing may not be an appropriate investment for scarce educational funds.

Second, although I demonstrate that cheating is widespread even in low-stake exams, I cannot answer why this is the case. It is possible that teachers perceive the tests to be high stakes even without explicit incentives (as suggested by Bertoni etal., 2013 for the INVALSI exams in Italy and Singh, 2020 for official tests in a different Indian state). It is also possible that, given the salience of high-stake exams in the Indian education system, many students perceive all tests to be high stakes (and cheat for higher scores, when given the opportunity). Unlike, e.g., Alan etal. (2020), who showed substantial cheating even with no performance incentives in Turkish schools, we did not collect extensive student-level information to study the correlates of cheating.

Third, this experiment was designed only to measure (and address) aggregate levels of cheating. It cannot, however, decompose the extent to which cheating is initiated by teachers or by students. This would be of independent interest to examine, but, unlike tablet-based tests here, would require interventions that only affect one of these sources (e.g., random seating of students, as in Lin and Levitt, 2020) or a precise way of measuring the two sources of cheating separately.

4. Conclusions

I have documented two results in this paper: first, cheating inflates official achievement data and, second, substantial reductions in distortion may be possible at scale within LMIC state capacity and budgets. Given the willingness of governments to adopt such assessments, interventions to improve their integrity may have high rewards both for policy and research.

While measuring student achievement is an important application, the challenge presented by distortion in administrative data on public services is much broader than education alone. Similarly widespread misreporting is possible in many sectors, including health, welfare, disaster relief, agriculture or social security.²² In each of these, the lack of reliable information can constrain the scope of both policy action and public accountability. In this respect, administrative data, both in its scope and reliability, form a key part of the basic infrastructure of state capacity for the implementation of core functions. The core principles of improving administrative data presented in this paper—sample-based audits to provide an external benchmark and technology-aided methods that limit the opportunity to manipulate and enable precise (but cheap) detection—may also generalise across sectors.

The overarching message of this paper, therefore, is to suggest caution for strategies that emphasise ‘data-driven’ approaches to policy reform without considering the provenance of the data. The creation of robust data systems with standardised measurements and individual-level data has historically been a major (and often contentious) challenge in developing administrative capacity across the world (Scott, 1998). Remedying such distortion at scale remains a substantial task for administrative systems with weak governance.

Additional Supporting Information may be found in the online version of this article:

Online Appendix

Replication Package

1

For example, the first policy response to the problem of abysmally poor learning outcomes in LMICs (‘the learning crisis’) suggested in the World Development Report in 2018 was to assess learning ‘using well-designed student assessments to gauge the health of education systems [...] and using the resulting learning measures to spotlight hidden exclusions, make choices and evaluate progress’ (World Bank, 2018). Proficiency-based measures are now central to national and global policy goals, including the UN Sustainable Development Goals (Goal 4).

2

Bansal and Roy (2019) provided direct examples of discrepancies impeding decision-making: ‘As a consequence of conflicting measurements and lack of quality data, the objective of index-based measurement systems—to prioritise and identify weaknesses for improvement—is largely a lost cause. Today, if any bandwidth is spent in states, it is on wondering “what really is the truth”. Data-based learning assessments and rankings that should have been a clarion call to action for states have degraded into a source of frustration and cynicism, as well as the target of ridicule.’ In a complementary paper, I show that levels of achievement in official tests in Madhya Pradesh, a large Indian state, are severely exaggerated, especially for low-performing students (Singh, 2020).

3

This procedure, described in Section2.3, flags suspicious response patterns in classrooms with a high-mean, low-variance, high-within-class similarity in answer responses and a low proportion of missing responses.

4

See, for instance, Jacob and Levitt (2003), Angrist etal. (2017) and Dee etal. (2019), who documented teacher-induced distortion, and Martinelli etal. (2018), who documented copying by students. Of these, only Martinelli etal. (2018) used data from a developing country (Mexico) in the atypical setting of a high-stake incentive program in eighty-eight Mexican schools.

5

This intervention was inspired by encouraging results reported in Andrabi etal. (2017) and Afridi etal. (2020) in other South Asian contexts from similar interventions.

6

Logistically, it was only feasible to test one grade per school. By grade 4, students can be expected to answer written tests (at younger ages, tests would have needed to include oral stimuli to ensure comprehension).

7

In principle, the tablet-based tests could have allowed for more within-class variation in items administered by drawing on a broader item bank and, also, combining items more flexibly than the three sets administered in the paper-based arm. We chose not to do this to keep the test content in the two arms as comparable as possible.

8

The official letter also made clear that the testing was district wide and that it was being pushed by the state government with the intention of generating community report cards. As such, we expect that the tests had high salience for all schools, even though the test carried no formal incentives. The test was not introduced as a pilot.

9

The presence of external observers is frequently mandated even for regular paper-based tests in India. However, these observers commonly do not show up or do so perfunctorily (Singh, 2020).

10

The audit could only be completed in 117 schools. Given a tight retesting window, and ambiguity in identifiers for some schools, we could not track back three schools.

11

The unequal sample split across testing regimes reflected initial plans of the government to follow this study with a subsequent experiment on the effect of community-based report cards (as in Andrabi etal., 2017). Our prior, based on data from a different state (Singh, 2020), was that paper-based assessments would have substantial manipulation (whereas the efficacy of tablet-based tests was unknown). Since we did not want to disseminate report cards in settings where misreporting was severe, it was prudent to increase the proportion of tablet-based tests to two-thirds of the clusters. We randomised at the cluster level to keep testing modes constant within communities (since report cards would have compared schools within villages/wards). Unfortunately, for logistical and administrative reasons, this subsequent report card experiment was never implemented.

12

Percent correct scores in the tablet tests have a |$\sigma$| of 24.11 in math, 22.09 in English and 23.44 in Telugu.

13

This concern may be particularly important in the paper-based assessment arm, where students answer in the same format across the original test and the retest. Furthermore, teachers may have used the initial assessment for revision in the paper-based assessment schools. This is not a concern in the tablet testing arm since no physical question papers were left in schools. Evidence suggests that this is, in fact, the case since students who were initially tested on paper performed somewhat better in the retest (Online AppendixFigureA2, Table A1). Since the retest was administered on paper to all students, I cannot separate the effect of test-mode familiarity from revision.

14

Direct evidence of this is provided in Online AppendixFigureA3 that presents the distributions of estimated ability, as estimated in pooled data by a (common) 3-PL Item Response Theory model, in the two treatment arms. The paper-based test has a substantially higher estimated achievement even though, by virtue of randomisation, one would expect the estimated distribution of ability to be similar across the two groups. The assumption of a common item response model is also unnecessary here since, unlike, e.g., Martinelli etal. (2018), our focus is not on identifying individual students who cheat, but rather on seeing if, on aggregate, one testing regime is subject to less distortion.

15

See, for instance, the discussion of the trade-offs between different types of cheating indices in Martinelli etal. (2018), who found that cheating is higher with student incentives, but that it varied depending on whether they used the |$\omega$|-index (their preferred measure) or the K-index (which shows very little sign of cheating).

16

This is particularly important since students are often very far behind grade-appropriate levels and thus tests tied closely to official syllabi alone may miss even substantial learning gains (Muralidharan etal., 2019).

17

See, for instance, Jayaraman etal. (2016) for an example of how adaptation by agents reverses short-term conclusions about the productivity effects of payment schedules for workers.

18

Note that the ‘learning crisis’ is fundamentally about absolute learning levels, which have been the primary focus of this paper. However, substantial manipulation of levels may still preserve ordinal ranks of students within school or between schools (as documented by Singh, 2020 in a different state).

19

See, e.g., Alan etal. (2020), who used rich survey data to show that cheating in Turkish primary schools varies by socioeconomic status and IQ and that the responsiveness of cheating to incentives varies by altruism.

20

They may however still be useful for specific uses such as in evaluating pre-treatment differences between schools or students in a policy evaluation. See Muralidharan and Singh (2020) for an example.

21

Although, see Berkhout etal. (2020), who presented suggestive evidence of reduced cheating from computer-based tests in Indonesia, and Alan etal. (2020), who reported a prevalence rate of 34% for cheating in a sample of elementary school children in Istanbul.

22

For instance, similar concerns have been raised in India about official data on basic sanitation. Administrative data declaring villages as ‘open-defecation free’ are reported to be substantially overstated, despite in principle having a detailed process for verification by multiple administrative authorities (Agarwal, 2019).

Notes

The data and codes for this paper are available on the Journal repository. They were checked for their ability to reproduce the results presented in the paper. The replication package for this paper is available at the following address: https://doi.org/10.5281/zenodo.10727363.

I am grateful to Erich Battistin, Martina Bjorkman Nyqvist, Konrad Burchardi, Luis Crouch, Lee Crawfurd, Jonathan de Quidt, Jishnu Das, Tore Ellingsen, Clement Imbert, Karthik Muralidharan, Geeta Kingdon, Gaurav Khanna, Derek Neal, Lant Pritchett, Mauricio Romero and several seminar participants for insightful comments. I am also grateful to officials at the Government of Andhra Pradesh—in particular, Ms. Sandhya Rani and Mr. Santhosh Singh—as well as staff at the Central Square Foundation, especially Rahul Ahluwalia, Saloni Gupta, Neil Maheshwari and Devika Kapadia, for their collaboration. Ramamurthy Sripada, Nawar Al Ebadi and Edoardo Bollati provided outstanding field management and research assistance.This project was supported by the Research in Improving Systems of Education (RISE) program funded by UK aid. Ethical approval for fieldwork was obtained from the Institutional Review Board of JPAL South Asia at the Institute for Financial Management and Research.

References

© The Author(s) 2024. Published by Oxford University Press on behalf of Royal Economic Society.

Issue Section:

Short paper

Email alerts

Citing articles via

More from Oxford Academic