Indicators as a Technology of Governance
Sally Engle Merry
The research described in the project was supported by a grant from the National Science Foundation Law and Social Sciences Program and Science, Technology, and Society Program.
Abstract
Achieving compliance with human rights laws is a persisting challenge for the human rights system. This problem is embedded in the theoretical question of how international law functions in practice given its limited enforcement powers. My previous work on the way women's human rights are appropriated and used by women's NGOs in several Asia/Pacific countries shows that these ideas are vernacularized, or reinterpreted in local terms that are often broader and less specific than the conventions where they are stated. In this talk, I examine how indicators based on systems of ranking or numerical scores are entering into the field of human rights compliance and how this approach to measuring compliance is at odds with the vernacularization process.
Until recently, sociolegal scholars had made little effort to examine the nature of international law and global governance. Yet, this is a field ripe for a sophisticated sociolegal analysis that examines laws, institutions, social actors and their networks, and systems of discourse and meaning. Some recent theories, such as Harold Koh's ideas of transnational legal process or Benedict Kingsbury and Richard Stewart's ideas of global administrative law, assume the existence and force of such institutional, interactive, and meaning-based forms of knowledge. These theories are compatible with a more extensive sociological analysis of their dynamics beyond what has been done. There are significant benefits to a sociolegal approach to global governance and law.
Scholars of domestic law frequently complain that international law is not really law since it has no reliable system of enforcement. Indeed, a common critique of human rights is that there is no mechanism for requiring compliance with the terms of human rights treaty conventions. But human rights do act in the world, although not in any direct and straightforward way. They do not compel compliance with clear consequences for failing to comply. On the other hand, as law and society scholars have long demonstrated, neither does domestic law act this way. Not all violations are punished or corrected; not all laws are enforced equally, and not all defendants are treated the same. Law in action differs from law on the books. Similarly, how human rights act, and with what effects under contemporary conditions of governance, cannot be expected to be straightforward. Instead, these are critical questions that warrant further exploration.
In order to understand human rights in practice, we need to see them as part of an emerging mode of governance that we might call evidence-based governance. This form of government depends on producing knowledge concerning performance with reference to shared standards. Performance is then assessed by means of this information. For example, compliance with human rights standards about the right to health is measured by maternal mortality and life expectancy. In the sphere of global governance generally, data is typically collected about state behavior in order to assess government efforts. Evidence of compliance with the right to health, for example, includes the extent of provision of emergency obstetrical care.
Statistical techniques such as indicators, which package together several kinds of information into relatively simple numerical scores or systems of ranking, represent a sophisticated tool used for evidence-based governance. Examples of such indicators are the Human Development Index, which assesses income, health, and education together. Another is the Corruption Perception Index of Transparency International.
Evidence-based governance is based on assessing performance against negotiated standards. Assessment depends on mechanisms of measurement, data collection, and audits of the information to produce evidence about compliance with these standards. Such evidence does not lead to the imposition of sanctions, but correction and advice about how to improve performance. The treaty body monitoring the women's convention, CEDAW, for example, on discovering that Finland did not have any protective orders for domestic violence cases, urged Finland to adopt such as mechanism. It did not impose sanctions but provided recommendations for correcting behavior. This is typical of the way the committees charged with monitoring compliance with the human rights conventions operate. This process of data collection, analysis, and presentation is influential because of damage to national reputations within global communities and loss of benefits such as foreign assistance and foreign investment. It is a mode of governance that depends on expertise in the production of measures, data and analysis, and assessments of performance with relation to standards. It is now widespread in such fields as development and human rights compliance.
These governance mechanisms, which include treaty body monitoring of compliance with human rights treaty obligations, increasingly rely on statistical information. While the use of statistics in country reports is widespread in social, economic, and cultural rights, there are some efforts to adopt statistical measurements for civil and political rights as well. It is of course tricky to collect information on the frequency of torture, for example. Nevertheless, human rights compliance is increasingly defined by sets of measurements and statistics.
Yet, I argue that we need to be skeptical about the apparent objectivity of such quantified knowledge. Despite the appeal of quantified information as scientific and above politics, in practice numerical data and indicators are thoroughly political. They are social constructs that emerge through processes of consensus building and contestation within a regime of unequal power relations. They are the product of expert knowledge, political struggles between richer and poorer countries, and forms of data collection that must be paid for. These forms of data collection differ in their accuracy and cost. Once established and recognized, however, statistical techniques such as indicators become technologies of knowledge that endure and have effects beyond those of the original creators. They grow out of power relationships based on disciplinary expertise and experience and the relative strength of nations, donors, and organizations. Once launched, they themselves exercise power.
I will use a case study of an effort to create knowledge about violence against women globally to examine these processes of knowledge production. This global initiative relies on developing standardized techniques for measuring violence against women and for producing mechanisms for collecting data, centralizing the information, and generating comparative information. It requires shared understandings, comparable categories of measurement, and some consensus on what violence against women means. How these processes take place depend on institutional, cultural, and social structures. I want to argue that any system of measurement contains implicit assumptions about the nature of the violation and the approach that will improve it. These assumptions come from the organizations and the academic disciplines that help to create them. Which approach prevails depends on the power of those advocating it.
I conclude with a discussion about the relationship between quantitative measurement of compliance with human rights terms and the way human rights are understood at the grassroots, through the process of vernacularization. As we will see, this is a major divide. Measurement systems rarely capture the way human rights work in the vernacular. They are often adopted in social movements with meanings quite different from that of the legal system. May be useful as a justification for women standing up for themselves, for example, or critiquing the way courts treat battered women in child custody cases, as ex in New York City.
Rather than measuring such mobilization of human rights, however, measurement systems typically focus on government efforts, especially the passage of laws and the implementation of programs. They also attempt to measure enjoyment of rights, but gathering data on enjoyment of rights is difficult and expensive. Thus, these systems of measurement and accountability focus on government efforts, but miss the primary areas in which human rights ideas act. Thus they fail to measure the way human rights work in practice.
Phenomenon I am describing is a dimension of "new governance" -
Use of information in governing obviously not new- rise of modern nation states saw efforts to count and measure populations through censuses, other forms of data collection, - such knowledge fundamental to colonial empires as well –
But, we are now moving toward a new form of governance, of regulation, where problems are complex and there are multiple actors- eg., in EU, in US where federal structure, in international situations- called new governance or experimentalism. This is a shift from command and control systems which tend to be hierarchical, with rigid rulemaking and centralized decision-making to a system.
New governance is conceived as a pragmatic, incremental approach to problems of regulation that focuses on problem solving, participation of stakeholders, production of data, and systems of monitoring . Goal is to develop a form of governance that is more flexible, effective, and inclusive than command and control. New public management a dimension of this form of governance –
Key features are broadly framed goals, stakeholder participation, flexibility, reversibility, monitoring and peer review, transparency, a data-based approach, learning-oriented, multi-level –(de Burca 2010: 235).
Substantial reliance on information, such as indicators- whose credibility depends on being above politics. Processes of governance themselves are thought of as based on technical expertise, not politics
There are concerns about democratic deficit, role of committees and experts where decisions are made in terms of technocratic concerns, not open to public debate and discussion (Wisc Law Review 2010). Eg, in EU, process of developing soft law bypasses debates in Parliament - mechanism of OMC – Open Method of Coordination
How does evidence-based governance work in the field of human rights?
Series of steps:
1. Establish global standards- done by states, usually . Example- Millennium Declaration, Human Rights Treaties
2. Develop measurements, categories, labels. Eg., MDG targets and indicators. Usually done by experts and people with experience
3. Collect data or locate existing data. Use proxies if necessary. Gather data together and analyze. Audit data collection systems. Create indicators, ranking systems. This is done by national statistical offices, international organizations, private contractors, civil society, academics. Expertise important.
4. An independent body or monitor uses data to assess performance against standards. Develop recommendations and critiques based on performance, focusing on correction and discipline, not imposition of sanctions. Use ranking and comparison to shame and exert pressure. Sometimes use information to allocate development aid, foreign investment. This is done by treaty bodies, development agencies, Millennium Challenge Corporation, other decision-makers.
5. Renegotiate standards and data collection processes. Recursivity (Halliday and Carruthers)
Measuring Human Rights: The case of violence against women (VAW)
I will illustrate this general process through a case study of a phenomenon that has moved from being a subject of political mobilization and contestation to a site of technical knowledge – problem that needs to be measured and counted. Yet, measuring VAW is not simply a technical problem. I will show that there are very different approaches depending on the political leanings of the experts, the institutional support, and the disciplinary framing of the project.
VAW has been a significant area of concern and activism in last 10 years at the international level - national activism in the 1980s translated into global movement at Beijing in 1995, included in Platform for Action, and UN mechanisms such as special rapporteurs on Violence Against Women created in 1994.
2003, UN General Assembly asked the Secretary General to develop an in-depth study of all forms of violence against women. Report was presented to GA in 2006. Request for annual reports on efforts, made to GA
One of five major concerns in the report is the collection of data – need for better knowledge for "policy and strategy development". Some policy makers and activists have called for "a comprehensive set of international indicators on violence against women." (p. 56, A/61/122/Add.1).
2008 – UNITE TO END VIOLENCE AGAINST WOMEN. Secretary General's program on VAW, demand for statistics as one of five key points. Has created a VAW database where countries answer questionnaires, report performance, available online.
Thus, in order to develop a new initiative, tackle a new problem, it is necessary to gather data. Fundamental to global governance, reform efforts, assuring accountability, pressuring states, etc.
Two approaches to data gathering:
Surveys- either dedicated surveys or as modules attached to large scale surveys such as health, demographic data, or crime victimization
And administrative data – police calls, court records, visits to hospitals.
Administrative data is perhaps easier, since already collected by governments – no extra expense. But there are limitations in administrative data for VAW – people who don't call, don't use legal system, hospitals don't notice – but can also be developed and improved. EG., in US, crime victimization surveys, added category of domestic violence in 1993 – in response to social movement pressure. Before that, couldn't be counted. I tried to count number of restraining orders in US- each local jurisdiction defined its restraining orders differently, - protective orders, 209A in MA, etc. – counted differently, no centralized collection of information. To do that- need, of course, to standardize work, court practices, laws, categories of counting.
So, if turn to surveys-
How to measure? What to count? How to carry this out in practice? Costs, need to replicate. Self standing vs. attached poses problems for VAW –
In order to understand the political dimensions of these apparently objective measurements, need to do genealogy of their creation. It turns out that they have distinct and different paths of development, institutional sponsors, and underlying ideologies of ways to change the phenomenon.
To do a genealogy of measurements – need to look at participants, organizations, templates that get adopted and circulated, communities of expertise that are involved. In looking at genealogies of indicators- see critical role of various forms of expertise, moments of political contestation and public debate, also important role of private discussions, reliance on previous research and data collection and category definition – as Latour argues for science more generally, is the creation of a set of networks of support over time, others who buy in, use, etc
Past experience is very important in guiding future developments –countries that have measured VAW become model for others. EG., Secretary General's 2006 report notes that there have been dedicated national surveys in Canada, Australia, Finland, France, Germany New Zealand, Sweden, and the US. (2006: 57). Used as the basis for WHO multi-country study on DV, came out in 2005.
But not generally part of official statistical data collection in most countries. .
In other words, new efforts build on past experience of surveys. My observations of EGMs indicate the same thing. Where do the questions asked, the templates for these surveys come from? Experience in wealthier countries that have already carried out such surveys - becomes model for developing next set of surveys - does this matter? May list acts of VAW, for example, but not include acid throwing – Not consider refusal to return parts of dowry at divorce
Development of quantified knowledge = such as survey data or indicators- is a technical process, depends on experts and those with experience. But it is also political.
In the case of the Sec General's report, an expert group meeting was convened in 2007 to develop survey approaches. Those with experience played important role. Relative technical expertise authorizes some countries to speak rather than others- Canada, not Botswana, is authoritative, for example. Development of data for measuring VAW moved into and out of explicitly political domains. EG., recommendations of 2007 EGM on ways to measure VAW were presented to UNSC for approval. Made up of national representatives. Commission decided to drop FGC because too "culturally sensitive" and child marriage because variable national marriage ages. China wanted to include psychological violence, but it was not picked up by others. Recommendations then sent to a subcommittee to work further, called Friends of the Chair, made up of govt members, UN agency reps, and academic and NGO experts.
Even technical domains are very political- who is present, who speaks with experience, what are templates and models for the developing indicator or measurement system? To show this, I compare four different, parallel efforts, to gather information on VAW to satisfy this global demand. Will examine genealogies of these four approaches to VAW measurement. All are based on work of international organizations, mostly the UN.
Each comes from its own distinct trajectory, with differing institutional supports – and has a distinct set of ideas about what the problem is and how it can be improved, its own intellectual and disciplinary roots. Most come from a long trajectory of research and theory development, often stretching over thirty years or more – creation of models, experimentation, expert work on data collection, consensus building –
Comparing them reveals these underlying, often implicit ideas. Shows that each has a particular theory of social change, an idea about how to diminish violence against women, and what states should do to improve the situation. Moreover, the particular questions asked or modes of data collection reflect these implicit theories. Thus, they will produce different kinds of data and analysis. Differ in attention to local situations, ethical concerns, overarching political orientations.
Examples:
Gender equality approach- builds on Vienna 1993, women's rights as human rights, Beijing conference 1995, Declaration on VAW 1995, sp rapporteurs on VAW, Secretary General asks for in depth report of GA, 2003, asks Sp Rapp to help- finished 2006. Projects tend to ask about attitudes toward, social tolerance; forms of emotional abuse as well as physical and sexual, controlling behaviors. Use of surveys, ideal is a dedicated survey, concern about an attached module in terms of ethics/safety. Now several regional institutions, lead by UNECE and ECLAC, are working to develop a survey module to be attached to an existing survey. This includes several dimensions of VAW not included elsewhere- eg., controlling behaviors and emotional abuse and fear and change in everyday activities, in addition to physical and sexual acts of violence. Although the UNSC did not approve these measures, an expert group I attended in 2009, which had a feminist focus, decided to measure it anyway.
Focus on collecting data on prevalence- no of women of given age who have had experience of violence of various kinds in last year, and in lifetime. Ie, used to describe population of women victims.
Use of WHO model, other national surveys, sp apporteurs report on indicators.
Theory- diminishing VAW depends on increasing gender equality. VAW has emotional as well as physical and sexual dimensions.
Experts who work in this area – tend to be sociologists and anthropologists, familiar with statistical analysis
This is dominant approach in Sec Gen report and in EGM of 2007 and 2009
Human Rights – closely related, idea that VAW violates women's fundamental freedoms and human rights – position of Sec Gens's report. Violations include wide range of behavior, as in OHCHR model, in a variety of contexts. Eg., home, work, community, conflict and emergency situations. Way of specifying rights for progressive realization, focus on state action and accountability- who is to do something? More than on outcome data – characteristic of human rights approach
At 2007 EGM, OHCHR proposed a framework developed for other human rights- food, health, development, early child development - since 1980s
Framework focuses on state responsibility, on determining who is accountable for improving problem- basic human rights approach –
1. structural indicators – measure commitments–laws and basic institutional mechanisms
2. process indicators - measure efforts–measures, including public programs, services, and specific interventions that are implemented to give effect to commitments – eg, no of persons arrested, adjudicated, convicted, or serving sentence for VAW per 100,000 pop
((this set of indicators is about law in action – effects of implementing laws, eg., programs, or what those programs have produced. Assumption is that arrests and convictions will reduce VAW ))
3. outcome indicators – measure results- attainment and results that reflect status of realization of the elimination of VAW –( eg., Reported cases of women victims of trafficking, sexual exploitation, or forced labor, or reported cases of VAW incidents on women refugees, asylum seekers, returnees or internally displaced (From OHCHR framework for VAW)
Here, OHCHR report to EGM 2007 used template promoted by OHCHR here and for other human rights- promoted by OHCHR as a way to harmonize and streamline treaty body process.
OHCHR has developed a broad conceptual map of a range of forms of VAW, not a simple quantified indicator. Instead, map includes lots of specific indicators, such as proportion of women feeling "unsafe" (eg., walking alone in area after dark).
And same indicator as gender violence approach, "Proportion of women who have experienced physical or sexual violence by current or former partner during the last year [life time]" (ie, prevalence data).
So embedded theory is that if states adopt these policies and programs, will improve VAW.
Theoretical framework- human rights law, core leaders of this approach are human rights lawyers and development economists. Focus on state accountability.
Criminal justice – also presented at 2007 EGM – use of criminal victimization survey approach- idea is to add questions about VAW to existing International Crime Victim Survey. Background in UNODC, see behavior as a crime- but not like a crime with a clear definition in the law- Report from UNODC to EGM questions what is VAW –
says, VAW is not "self-defined" as a criminal act. – includes other things such as sexual harassment, trafficking in women, marital rape- not crimes everywhere. Moreover, some forms of violence condoned or perpetrated by state, such as forced sterilization of violence of police or prison guards - so neither administrative crime stats nor general crime surveys are easily able to capture entire range of VAW - suggest making definitions of violence against women compatible with other crime surveys such as CTS - UN Surveys on Crime Trends and the Operation of Criminal Justice- authorized in 1984, carried out since then, now 12th wave taking place –questionnaires sent by UNODC to member states
UNODC report recommended using same categories so can build on this work
– so use, eg.,
Homicide
Rape
Major assault
Assault
Sexual assault
Harassment
Female genital mutilation/cutting
Trafficking in persons
Get data on incidence, not prevalence- ie, how many times an act of violence happens- regardless of whether many events happening to one person or a single incident to many – so, measures no. of crime events, comparable to other crime stats, not experience of victims. See here emphasis on being compatible with other crime measures rather than focusing on experience of women as victims of violence.
Recommend attaching survey to existing crime victimization surveys and approaches - also considers use of administrative data such as calls to police, interest in difference between reported and actual, called dark figure or hidden crime. Focus on finding gaps or flaws in criminal justice system through survey. In report, say that they need to choose indicators that reflect criminal justice system response and relate to developing work in crime trend assessment and monitoring
Suggest counting: 1. criminalization of violent act, 2. victim police reporting and secondary victimization, 3. availability or use of victim support services, 4. policy or practice relating to prevention of violence –
Model is UNODC/UNICEF juvenile justice indicators-
Typically people with criminology background and training
Theory of social change – depends on improved criminal justice system
Clearly, different orientation, different kinds of information considered valuable, different idea of improvement for problem
National Statistical Offices approach- support and facilitate work done by national statistical offices, find ways to make a consistent and coherent set of guidelines, try to develop a common definition across countries, get national buy-in so will carry it out –
Through surveys, administrative data- work on ways that states will buy into the process, develop universal techniques and strategies, replicate data collection over time. Not particularly concerned with attitudes nor a feminist politics – more about supporting capacity development. Model is housing and demographic statistics surveys, where guidelines produce comparability. Tends to involve people with statistical expertise. Need state support – states need to buy-in to the desirability of counting something. May have political debates about what to measure- as in case of debates about what to measure for VAW in UNSC. – eg, physical violence, sexual violence, child marriage, FGC? This was open to public, political debate.
Propose measures that national statistical offices will take on- improving administrative data, perhaps attached module to existing state survey – goal is to make it comparable and something states do on a regular basis - monitor over time. So, somewhat more open to political debate, but when disagreement on terms or resistance to being surveyed and counted, process stalls. At this point, advocates tend to move into more private or civil society approaches -
So- each of these approaches has institutional support, a distinct body of experts and expertise, and templates for how to count and how to analyze. They value different issues, choose to ask somewhat different questions, are motivated by different theories about how to diminish VAW. These variations will produce different results, depending on own assumptions.
Which will become dominant? Depends in part on institutional strength, support of various countries, organizations, prominent academics and universities -
But they are not all measuring the same thing. Instead, each is developing a system of measurement that is founded on a distinct theory of social change and a set of models for improvement. And each is based on past experience, unequally distributed among rich nations and poor. Experts tend to be those with training and experience – international cosmopolitans, but those with experience tend to come from countries with the resources to do this.
Conclusions:
What is relation between these measurements and grassroots ideas of human rights? To the way they are used in practice by human rights groups?
Vernacularization- process by which global ideas of human rights are refashioned in local contetxts. May be reinterpreted, redefined, even created where not in documents. Can be used to argue either side- eg., in abortion cases, right to life and right to choose. If want to understand how human rights work in practice, need to foreground this process of translation and localization. Ex of nari adalat: women's courts in Baroda, nari adalat, built on panchayat model but women only, to deal with divorce and violence issues. Members focused on encouraging women to stand up for themselves, not let domestic violence seem justified, insist on return of some of dowry- said they were doing human rights, but not tied to CEDAW or documents but to sense of women standing up for themselves.
There is a mismatch between the way violence against women as a human rights violation is understood in the vernacular and the kinds of measurements that take place. These vernacular understandings don't make it into the creation of measurements, ideas of experts, necessarily, although they clearly differ. May be developed in some countries but not those where previous research has taken place. So, what appears to be an objective, scientific process of data collection and analysis has important political consequences but works largely outside the sphere of political debate and contestation. As such, it represents a key dimension of new global governance.
References
Dawson, Mark. 2010. "Transforming Into What? New Governance in the EU and the 'Managerial sensibility' in Modern Law." Wisconsin Law Review (2): 390-440.
De Burca, Grainne. 2010. "New Governance and Experimentalism: An Introduction." Wisconsin Law Review (2):227-238.
Expert Group Meeting on indicators to measure violence against women, Geneva 8-10 2007, co-organized by UNDAW, UNECE and UNSD in collaboration with ECA/ECLAC/ESCAP/ESCWA. "Criteria for Identifying Indicators on VAW." Supporting Paper 3, 5 October 2007. Submitted by OHCHR.
Ford, Christie. 2010. "New Governance in the Teeth of Human Frailty: Lessons from Financial Regulation." Wisconsin Law Review 2010 (2): 441-487.
Pad.ma requires JavaScript.
