Who Controls the Machine, III: Accountability in the Results-based Revolution

John R. Mathiason*

Syracuse University

Summary

Following up on two previous analyses of the efforts of Member States to solve the problems of command and control in the United Nations and its organizations, the article focuses on the latest effort, the institution of results-based budgeting.  The approach as applied to assessed budgets is critical for the maintenance of support of major contributors to the United Nations and other international organizations.  After reviewing the problems inherent in previous efforts at using programme planning to control the organization and make it accountable, the article focuses on the use of monitoring and evaluation as means of holding organizations accountable for producing results and on the conditions necessary for this to be effective.  It argues that refocusing the work of the Committee for Programme and Coordination and the Office of Internal Oversight Services as its independent secretariat could make the results-based revolution work and Member States finally to control the machine.

Introduction

Governments created this machine—which over the years has grown into what    is probably the most complex organization in the world.... This ‘Machine’ now has a marked identity of its own and its power is so great that the question must be asked ‘Who controls this ‘Machine?’ So far, the evidence suggests that governments do not, and also that the machine is incapable of    intelligently controlling itself.

Sir Robert Jackson. A Study of the Capacity of the United    Nations Development System, Geneva, 1969

I first used the epigraph from Sir Robert Jackson in 1987 when my first essay on programmatic control in the United Nations was published (Mathiason, 1987), at the time of a major reform effort that focussed on programme planning and budgeting.  Almost twenty years before, Jackson had diagnosed that command and control was an important, unresolved issue for the international system. I used the epigraph again in 1997, in the context of a Secretary-General Kofi Annan’s first major reform effort that also focused on programme planning and budgeting as a matter of command and control (Mathiason, 1997).  Now, five years later, in the midst of the Secretary-General’s second effort at reform, it is time to ask the question once more but this time in terms of whether the results-based approach that has been adopted can really hold international organizations accountable for results.

One of the stereotypes that has emerged in the wake of the 2003 Iraq War is that, in the face of unilateralism by the United States, the United Nations and the rest of the United Nations System is in full decline.  While this may be true of Iraq, there is no evidence that it has spread to other areas and, in fact, there is growing evidence that the expansion of the international public sector has continued and perhaps expanded as more and more services are being provided.  These include defining international norms and standards over a wide variety of issues that were formerly the exclusive concern of States, delivering international public investment, resolving economic as well as political disputes, dealing with complex political issues of non-proliferation and disarmament, and providing direct services for those whose States lack the will or capacity.

As more national public resources have to be channelled into the management, however, national legislatures that have to appropriate funds for international organizations increasingly have to be convinced that the expected results are being obtained.  This is particularly true of programmes funded through the core, assessed parts of organizations’ budgets.  Contributions to development organizations like UNDP, UNFPA, UNICEF or WFP, which are completely funded from voluntary contributions, or extra-budgetary contributions to the United Nations proper or the Specialized Agencies can be reduced or eliminated if the legislatures are convinced that the programmes are ineffective or out of sync with national values. [1]  As a result, development organizations have always placed a premium on showing results. In contrast, assessments, once made, have to be paid as a condition of membership.  It is because of this that major contributors have, for many years, sought to install some form of accountability for results, now reflected in the results-based budgeting approach.

Most international organizations have now adopted a results-based approach to programming, and now face the consequent problem of ensuring that programmes are accountable for the results that they have promised, and that this is incorporated into the budgeting process.  Proving results may be much more difficult than promising them, and this will be a major challenge for the reformers, both within the governments and the Secretariats.

Origins of the results-based programme budgeting approach

International organizations have increasingly adopted results-based programming, budgeting or management.  This is to distinguish the process from earlier efforts at budgeting by programme that in turn replaced line-item budgeting.  It reflects an attempt to apply to international organizations the kinds of tools that characterized the “re-inventing government” movement. (Osborne and Gaebler, 1992).

There is a tendency for international organizations to be encouraged to adopt public administration innovations from the national level, just as national governments abandon them.  The extremely Weberian programme budgeting that was adopted during the reforms of the mid-1970’s drew heavily on the experience of national governments doing rigorous programme planning.  A French member of the Joint Inspection Unit, Maurice Bertrand who became the guru of United Nations programme planning reform, did much of the intellectual work at this point.  His 1974 study of Medium-term Planning in the United Nations System (Joint Inspection Unit, 1974) provided the basis for the eventual adoption of the Regulations and Rules for Programme Planning, Budgeting, Monitoring and Evaluation that are still formally in place. Bertrand posed the problem to be addressed by programme budgeting as “The need to introduce a certain amount of order and clarity into the extraordinary diversity of the activities – too often marginal – of the international organizations and the desire to condense these activities, through a concentration of effort, into a number of programmes who effectiveness is beyond question…” (Joint Inspection Unit, 1974, p. 1).

The initial effort at addressing the problem was to create a complex formal structure for programming and budgeting.  It involved a six-year fixed horizon medium-term plan that would set out objectives and would be translated into output through biennial programme budgets.  There would be regular monitoring of each programme budget within the next year and periodically programmes would be subject to in-depth evaluations.  This approach was formally embodied in the Regulations and Rules for Programme Planning, Budgeting, Monitoring and Evaluation that were intended to be the programming equivalent of the Financial Regulations and Rules.

The implementation of this phase was, in a formal sense, a success.  The programme planning process became routine.  The internal submission process included an effort to match resource requirements with planned output.  A special office, the Office for Programme Planning and Coordination [1] was established in one department to review the programme aspects of the budget in the main sectors dealing with economic and social development, while budgetary aspects of all departments and programme aspects of the political, information and administrative departments were reviewed by the Budget Division.  Programme managers identified those programme elements that were of highest priority as well as those of lowest priority.

In a real sense, the system did not work.  Programming tended to be based on the formulation of politically acceptable, but practically unattainable objectives.   More often than not they were vague and immeasurable.  The assignment of resources to activities was largely arbitrary and inconsistent across divisions within the same departments.  Programme managers defended programmes against cuts by assigning “lowest priority” designations to activities that were politically sensitive and thus unlikely to be cut.  The resulting proposals were voluminous (totalling, in the average year, well over 400 pages) and arcane.  Because of the difficulties of producing them on time, they were issued for review by the Committee for Programme and Coordination piecemeal.  The CPC, in its review, looked at each section in isolation rather than comparatively or in terms of effectiveness.  The budgeting process itself, as frequently happens in national governments, worked only when their was growth, since programmatic priorities could be used to allocate the growth (in the form of new or additional posts).

By 1986, the United Nations was facing a financial crisis because the United States was withholding a large part of its contribution because it argued that budgetary growth was unchecked.  The resulting reform, by a group of 18 intergovernmental experts sought to address the issue.  It did so by recommending an across-the-board reduction in staff (15 percent) and the investing of the responsibility of approving the total budget level with the Committee for Programme and Coordination.  The Group of 18 recommendations reflected the implicit understanding that the programming methods used by the United Nations did not provide a basis for assessing the relative worth of programmes. [2]

Subsequently, it was decided that a six-year horizon was too long to be meaningful, and the medium-term plan period for the United Nations was reduced to four years.  The programme budget was judged to be too voluminous and to make the text more manageable, the number of programmes was reduced by converting all former major programmes to programmes and all former programmes to subprogrammes.  While this did not change the essential structure of the programme budget, it removed much of the detail from the presentation.  Internally, the review process was as detailed as before.

Because this reform essentially made no changes in the programme planning system, it did not work either.  During this period, the United Nations ran on zero-growth budgets (at least formally) but the system did not provide any programmatic basis for reallocating resources from lower priority programmes to higher.  As was noted long ago by Aaron Wildavsky (Wildavsky, 1979), all budgets are incremental, each building on the previous one.  This meant, in the United Nations, that under zero-growth, any increase in one programme would have to come from resources released from another.  But all programmes in the United Nations have their supporters so any criterion based on the “importance” of the objectives being pursued was doomed to fail from the start.  The CPC review of programme budgets during this period essentially involved vetting the citation of mandates and, in a few cases, changing the hierarchical structure of programmes without affecting their content. [3]

The first reform proposals made by Secretary-General Kofi Annan (United Nations, 1997a) in July 1997 and adopted by the General Assembly in November of that year included as one of its core elements the recommendation

That the General Assembly review the existing arrangements governing the planning, programming and budgeting process in order to enhance their role in providing strategic direction, establishing better performance yardsticks and reporting and focusing more on accountability for performance than on input accounting - a shift to results-based budgeting.

In explaining the meaning of the concept of results-based budgeting to the General Assembly, the Secretary-General’s report noted that the main innovation was to shift the focus of budgeting away from control of inputs to accountability for results.  As an addendum to the proposals (United Nations, 1997b, paras. 9-10),

Results-based budgeting is premised on enhanced responsibility and accountability of programme managers for their activities and consequential delivery of results.  It is essentially about giving managers control over inputs so that they can best manage the production of outputs required to support the desired outcomes. … It makes the programme manager responsible and accountable for his or her outputs and consequent outcomes, while making the task more interesting and challenging and allowing for innovation.

As summarized by an evaluation by the United States General Accounting Office (GAO, 2000), the introduction of results-oriented programming, budgeting, monitoring, and evaluation systems that would:

• Focus and clarify the Secretariat’s priorities and objectives by focusing General Assembly mandates on a limited set of priorities and clarifying what it expects the Secretariat to accomplish.
• Adopt results-based budgeting and sunset provisions, with the objectives of (a) formulating program budgets on the basis of objectives linked to the medium-term plan and against which actual performance would be measured and (b) reviewing programs and justifying them before renewal.
• Develop a system to monitor and evaluate the impact of U.N. programs in achieving specified objectives.

Of all the Secretary-General’s reform proposals, the one that caused the greatest controversy and the longest period to implement was the recommendation on results-based budgeting.  It was very difficult to convince the Member States to agree to a change in budgetary policy.  As the GAO report pointed out,

According to U.N. and U.S. officials, some member states believe that results-based budgeting is a tactic to cut the U.N. budget. For example, in 1998, the Group of 77, a group of over 130 developing countries, took the position that results-based budgeting was a radical departure from accepted practices. They stated there should be no predetermined ceilings on budgets and that any attempt to use results-based budgeting would be resisted.

The same GAO report noted that a 1999 report by the Joint Inspection Unit (United Nations, 1999) had found that most of the other organizations of the United Nations system had adopted results-based budgeting with no direct effect on reductions in budgets.  The JIU did point out that there was no consistency in definitions of key terms between organizations.  The GAO report also pointed out that there was no mechanism to monitor or evaluate whether results had been achieved.

Eventually, most of the organizations of the United Nations system have adopted some version of results-based budgeting, with differences in terms of degree and success.  In the United Nations, as in the Specialized Agencies, this did not involve any change in the intergovernmental review structure.  The United Nations itself began to try to apply the approach to its 2002-2003 programme budget.  The first effort was received with mixed reviews by the CPC who, among other things, stated (United Nations, 2001):

36.         The Committee noted the new presentation of the proposed programme budget and recommended that the General Assembly request the Secretary-General to encourage programme managers to continue their efforts to improve that presentation in accordance with General Assembly resolution 55/231 and other relevant resolutions.

37.         The Committee noted also that there were some shortcomings in the narrative sections of the proposed programme budget, and recommended that the General Assembly request the Secretary-General to ensure that programme managers comply with the provisions of the medium-term plan in the preparation of the proposed programme budget, in particular with regard to expected accomplishments and indicators/measurements of achievement, and also taking into account the mandates specific to the biennium.

The focus on the medium-term plan reflected the understanding that the medium-term plan constituted the legislative basis of the budgets and that it had been ignored in many cases when the programme budgets had been formulated.  In contrast, the Secretariat considered the Medium-term plan, which was prepared two years in advance of the first programme budget that it would cover, to be an inadequate, excessively general instrument.  This was one reason why, in the mid-1990’s, the United Nations medium-term plan had been shortened from six years to four.

In his second round of reform proposals, following his re-election in 2002, the Secretary-General proposed further changes in the programming system.  Specifically, he recommended that (United Nations, 2002),

The present United Nations planning and budgeting system is complex and labour-intensive. It involves three separate committees, voluminous documentation and hundreds of meetings. Changes proposed in section V include a medium-term plan covering only two years (rather than the four as at present), which would be combined with the budget outline submitted one year before the actual budget is tabled. The budget document itself would be less detailed and more strategic, and would give the Secretary- General some flexibility to move resources according to needs. The report also recommends that intergovernmental review of plans and budgets should henceforth be conducted exclusively in the Fifth Committee of the General Assembly, rather than being shared as at present between that body and the Committee for Programme and Coordination (which results in a great deal of unnecessary duplication). [Emphasis in the original.]

This approach is in contrast to other organizations of the System, which have maintained their planning horizon as well as their programme review structure.  To put the issues of results-based programming in context, this article will contrast the approach being taken by the United Nations proper with that being taken by the International Atomic Energy Agency, which was one of the first organizations to implement results-based programming as part of its management reform. [4]   The true test of the results-based approach will be whether it succeeds in holding programme managers accountable for their results.

The problem of accountability in international organizations

While the problem of accountability for results is common to all organizations, private or public, it is particularly difficult for international organizations.  This is largely because international organizations usually deliver their services indirectly, in contrast to national administrations.  And, like most national governments, international organizations have no “bottom line” like private sector organizations.  There are no clear monetary indicators of results, like profit or loss, or changes in revenue flows.  Instead, international organizations seek to influence or support beneficiary states in their programmes.  Here, international organizations also differ from national administrations.  For the latter, services are delivered that can be measures, in terms of number of customers served, number miles of new roads, or extent to which diseases are eradicated.

The indirect nature of results has sometimes meant that international managers confuse their intended results with those of Member States.  This can be because the objectives are the same, although the link between what the international organization does and the objective is through the national government.  This problem is the source of one of the criticisms of the first results-based budget in the United Nations, when the CPC said (United Nations, 2001, para. 40):

40. The Committee stressed the necessity of rendering the expected accomplishments and indicators of achievement to measure achievements in the implementation of the programmes of the Organization, and not that of individual Member States…

The dilemma for programme managers is that if their effort to induce national counterparts to take their recommendations into account, or use the technology they provide, can they take credit for the national result?  And if there is no positive national result, can they be held accountable for failing?  If the answer to either question is “no”, then accountability cannot be determined.

For those who advocate the results-based approach, the solution to the dilemma lies both in the way that the expected results are expressed and in how they are evaluated.

Origins of the results-based approach: the log-frame

One of the main origins of the results-based approach in the United Nations is the logical framework, or log-frame, approach. The approach was first formally adopted as a planning tool for overseas development activities by USAID in the early 1970s. Its origins can be traced back to private sector management theory, such as the 'management by objectives' approach that initially became popular in the 1960s. (Australian Aid 2000).  It gradually was adopted by development organizations of the United Nations and was applied conceptually to the regular budgets of the United Nations and other organizations when results-based programming was instituted.  The log-frame essentially is a planning tool that starts with objectives, determines what results are needed to achieve the results, then determines what output is necessary to obtain the results, defines the activities necessary to produce the output and finally, establishes the inputs that are needed to carry out the activities. 

Table 1.  Comparison of definitions of Logical Framework Concepts
Concept	IAEA (Results-Based Programming)	United Nations	OECD/DAC
Goal/Overall Objective/ Development Objective	[Long-term objective] A situation that should exist at the end of a specific programme/project period as a result of the organization’s activities that contribute to meeting the identified needs of the Member State.	An overall desired achievement, involving a process of change and aimed at meeting certain needs of identified end-users/clients within a given period of time.	Intended impact contributing to physical, financial, institutional, social, environmental, or other benefits to a society, community, or group of people via one or more development interventions.
Outcome	A specific identifiable change brought about by the organization’s activities within a specific period.	Desired outcomes involving benefits to end-users/clients, expressed as a quantitative standard, value or rate. Results are the direct consequence or effect of the generation of outputs, leading to the fulfilment of a certain objective. [Expected result]	The likely or achieved short-term and medium-term effects of an intervention’s outputs.
Output	The products or services to be delivered within the context of a programme/project.	Final product or service delivered by a (sub)programme to end-users/clients	The products, capital goods and services which result from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of outcomes.
Activity	Processes that transform inputs into outputs.	Action taken to transform resources (inputs) into outputs	Actions taken or work performed through which inputs, such as funds, technical assistance and other types of resources are mobilized to produce specific outputs.
Input	Organizational, financial, human and material resources required to carry out the programme element activities and to produce the outputs within a given time-frame.	Personnel and other resources necessary for producing outputs and achieving results	The financial, human, and material resources used for the development intervention.
[Source]	Showing the Results: Self-Evaluation Guide for Member-State Counterparts		OCDE 2002 GLOSSARY OF KEY TERMS IN EVALUATION AND RESULTS BASED MANAGEMENT

As can be seen from Table 1, all approaches based on the logical framework have these elements.  However, there are differences in the specifics of how each step is defined.  One of the difficulties in implementation of a results-based approach has been the lack of commonly agreed terms.  The lack of common terminology was one of the criticisms levelled by the Joint Inspection Unit in 1999.  In defining the results for which programme managers are to be held accountable, the United Nations has used the term “expected accomplishments”.  This is a concept that is somewhat more vague than that used in other organizations and differs from the approach originally proposed by the United Nations’ Programme Planning and Budget Division (United Nations, 1998).  In that approach,

Expected results are the desired outcomes of a programme, involving benefits to end-users …  Results are the direct and often tangible effect or consequence of the delivery of outputs, leading to the fulfilment of a certain objective.

The concept of “expected accomplishments” used in the programme budget for 2004-2005 is somewhat more: “Expected accomplishments are more specific and reflect benefits or changes that result from pursuing the objective.” (United Nations, 2003)

The IAEA uses the concept of outcomes in a more explicit sense, which it defines as “A specific identifiable change brought about by the organization’s activities within a specific period”.

If these concepts are applied correctly, it is assumed, Member States will be provided with specific, measurable promises by the international organizations of what will happen as a direct result of their investment of resources.  With this information in hand, the Member States will be able to judge whether the investment is worth making and have a basis for redeploying resources from ineffective to effective programmes, or for rewarding effective programmes and fixing ineffective programmes that are politically-important.

The problem of accountability in results-based programming

The major political issue in results-based programming has been a perception that it will be used to reduce budgets by eliminating programmes that are important, particularly, for developing countries.  While this perception is partly due to the fact that the major contributors to the budgets of the United Nations and the Specialized Agencies were the main advocates of the approach, there is clearly a lingering fear that any results-based approach may find that the programmes that are politically favoured may not be effective and thus make them subject to reduction on that basis.

For this to happen, three problems have to be overcome:

1. Programming documents must be able to indicate in clear and measurable terms what specific results should be obtained once the budgets have been approved.
2. There has to be a means to show whether the intentions were realized in the time period covered by the programme budget.
3. Secretariats and Member States must have the capacity to make a judgment about what to do about the information on attainment – or non-attainment – of results.

Organizations have primarily addressed the first step.  The United Nations and its sister organizations have made a concerted effort to include information on their expected outcomes in programme budget documents.  Starting with the documents for the biennium 2002-2003, most have included sections in each programme or subprogramme that state the intended results of the work.  This first effort has been only partially successful.  For example, in reviewing the United Nations proposals for the 2002-2003 biennium, the CPC (United Nations, 2001, paras. 36-37)

…noted the new presentation of the proposed programme budget and recommended that the General Assembly request the Secretary- General to encourage programme managers to continue their efforts to improve that presentation … [and] that there were some shortcomings in the narrative sections of the proposed programme budget…

There has clearly been some progress, since in reviewing the 2004-2005 Programme Budget, the CPC was kinder in its appreciation, but still not satisfied (United Nations, 2003b, paras. 67-68).  The Committee:

… welcomed the efforts by the Secretary-General to improve the results-based budgeting format taking into account particularly the recommendations made by the Committee at its forty-first session… [and] recommended that the General Assembly request the Secretary-General to encourage programme managers to continue their efforts to improve the presentation of the proposed programme budget in the light of the views expressed in the Committee.

A similar situation is occurring in the other organizations using results-based budgeting.  There is some satisfaction that budget presentations do give reviewing bodies a better basis to judge whether the proposed activities are likely to produce the expected results.

Since the adoption of the Regulations and Rules relating to programme planning, monitoring of programme performance has taken place once every biennium in the year following completion of the budget period.  For most of the twenty years in which reports on programme performance have been considered, the reporting was based on counting outputs.  The method was simple: the number of outputs programmed for each subprogramme would be compared with those actually produced.  A separate tally was made for outputs that had been designated highest priority (and therefore should have been produced).  Outputs that were not programmed but were produced based on subsequent mandates or management discretion were added to the mix.

The system had several inherent flaws.  Many programme managers would place more intended outputs than they expected to produce into their budget proposals as justification for additional posts.  When the new posts were not forthcoming – and this was the usual case during the many years of zero growth budgets, there was no mechanism to remove the outputs from the budget document.  As a result the approved budget contained many outputs that managers had no intention of producing.  And as a further consequence, statistics on programme performance would be artificially low.  Other programme managers, understanding the monitoring system, would only programme those outputs that were absolutely necessary, knowing that over the biennium they would be asked for additional reports, for example.  As a result their performance statistics would be high.  Because of these differences, the programme performance statistics were not particularly accurate.  More importantly, there were no consequences for programme managers of having either a high or a low programme performance statistic. [5]

The timing of the reviews made it difficult to apply any findings about performance to future programmes. For example, the programme budget for 2004-2005 was formulated starting in 2002, reviewed in mid 2003 and approved at the end of the year.  The performance appraisal for 2002-2003 will not be reviewed until mid 2004, so the earliest its results can be applied will be for the 2006-2007 programme budget.  Application to future budgets is constrained because programme managers can argue, “yes, performance was not adequate in the past, but we have fixed it since.”

The greatest flaw in the system is that it only counted output but provided no information about what changes were provoked by the output.  As a result, it was impossible to determine whether results had been achieved.

In reviewing the Secretary-General’s reform proposals for results-based budgeting in 2000, the United States General Accounting Office concluded (United States, 2000. p. 72):

A necessary component for successfully implementing a performance oriented management system is a systematic and reliable means for monitoring and evaluating program results and impact. Although the Secretariat is more candidly and critically evaluating its programs, it lacks a results-oriented monitoring and evaluation system and does not have a centralized organizational strategy for developing one.

In the three intervening years, the problem has not yet been solved at the United Nations, but other organizations have made considerable progress.  One of these has been the International Atomic Energy Agency.  The 2002-2003 Programme and Budget was adopted with a results-based framework: outcomes and performance measures were carefully developed and, while not completely satisfactory, made it possible for the Member States, through the Programme and Budget Committee to review performance on an interim basis in 2003.  The Agency decided to make an interim report by exception: indicating those programmes that were ahead of their intended results and those that were behind.  This allowed the information to be used in reviewing proposals for the next biennium.

The Agency also embarked on a strategy of developing an in-house capacity for self-evaluation, in which programme and project managers would appraise the extent to which they were measurably attaining their results.  These are being applied to the preparation of the first full programme performance assessment based on the 2002-2003 biennium.  Similarly, the United Nations has undertaken in-house training in the formulation of programme performance indicators and is finally beginning to develop a trainable methodology on self-evaluation.

Elements of Accountability in results-based programming

As noted above, achieving accountability in results-based programming requires two different elements.  First, results must be credibly measured.  Second, reviewing bodies must be able to form judgments about performance based on these measurements and apply them to future programming, rewarding good performance and punishing bad.  Neither of these has been achieved to date.

The problem of determining results

Determination of results in international organizations is complicated by the fact that most results are indirect.  Unlike national governments, which deliver services directly to their publics, international organizations typically seek to influence how national, subnational and non-governmental actors deliver services.  This means that results of actions by international organizations have to be reflected in actions taken by national or other authorities.  Unless the intended results are carefully formulated, they can be confused with national results.  This is a problem that was noted by the CPC in reviewing the proposed programme budget for 2002-2003 (United Nations, 2001, para. 40):

40. The Committee stressed the necessity of rendering the expected accomplishments and indicators of achievement to measure achievements in the implementation of the programmes of the Organization, and not that of individual Member States, giving a clear indication of the progress expected during the biennium, taking into account the international character of the United Nations, the purposes of its Charter and its legislative mandates, as well as the fact that the objectives of the Organization might not be realized in one medium-term plan only. The Committee also stressed the necessity of better integrating outputs into the programmatic framework of objectives and expected accomplishments, bearing in mind the direct relationship between input and outputs. The Committee also underlined that expected accomplishments and indicators of achievement should be capable of reflecting what is achievable during the course of the forthcoming biennium. (United Nations, 2001)

Formulating outcomes that show the direct relationship with output has proven to be a difficult task for most programme managers.  First, the outcomes are, almost by definition, not completely under the managers’ control.  Whether they occur or not is very much conditioned by external factors.  Thus, there is, for a programme manager, a considerable risk in promising changes.  Second, because, in terms of very large problems, what the international organizations can feasibly change is limited, there is a danger that the work will be seen as trivial and therefore vulnerable to elimination at the programming side.  The tendency in this case has often been to exaggerate the results that are expected.

The experience of the IAEA, however, has shown that, if carefully drafted, outcomes can be both realistic and non-trivial.  The Agency has also found a simpler way to define performance indicators: the approach consists of taking the intended outcome and asking the question “did it happen”?  The same has not happened in the United Nations where there has frequently been a disjunction between the “expected accomplishment” and the performance measures to be applied.

Assuming that the performance measures are reasonably drawn (and both the IAEA and the United Nations use the concept of SMART indicators (Specific, Measurable, Achievable, Relevant and Timely), the next issue is whether they can be credibly measured.

Will performance be credibly measured

One of the dangers of evaluation has always been that the costs of conducting the study will far outweigh the benefits from them.  This is particularly true when the evaluation is expected to cover the entire programme of an organization, and is to be undertaken every two years.  Performance measurement has to sail between the Scylla of too much data and cost to justify and the Charybdis of inadequate measurement.

For some programmes, performance measurement is relatively easy, because the intended outcomes are simple.  For example, one of the programmes of the IAEA develops protocols for radiation safety that it seeks to have adopted by national authorities.  The desired outcome is that protocols are adopted after the Agency sends them out and the indicator is whether they have been adopted.  The link between the Agency output (the protocols) and the national change (the protocols are applied) is clear.

Most outcomes are not that simple.  For example, many programmes in the United Nations System are intended to influence national action through international conventions.  Once these have been adopted, the next stage is to have them ratified by a maximum number of States.  One of the activities of the relevant programme is to encourage ratification of conventions.  The number of States that ratify a given convention during a biennium is often given as a performance indicator.  However, there is no guarantee that, based on receiving output from the United Nations programme, a State will ratify.  Thus, if a State does not ratify, that does not necessarily mean that the programme has not had any results.  It may be that a significant part of the national constituency might have become sensitized to the importance of ratification, leading eventually to ratification.  And, if a State does ratify, that is not necessarily because of the United Nations activity.  It might have been a result of external factors.   For these reasons, number of ratifications is not a good performance indicator.  The alternative is to express the outcome as the extent to which national authorities (or institutions or organizations) are aware the desirability of ratification.  This cannot be measured simply by counting number of ratifications.

Once performance measures are chosen that can be described in measurable terms, the next step is to determine whether the performance has occurred and report it.  There are two questions that must be answered:

• Was a valid, reliable and relevant methodology used to acquire and analyze the information about performance?
• How much information needs to be reported?

The first question will largely be answered by the organization’s reputation for sound monitoring and evaluation.  This can be conveyed by the extent to which the organization has provided training to its managers in evaluation, and the kinds of systematic support that is provided.  For example, the IAEA has developed a system of training in self-evaluation that has been made available to managers throughout the Agency (and is being made available, on a pilot basis, to Member State counterparts).  The United Nations Secretariat, through its Office of Internal Oversight Services, has begun a pilot effort to develop evaluation training that will serve to help programme managers provide credible information about results obtained.  The methodologies should permit reasonably simple, low cost performance appraisals, except in those programmes where either for political reasons or because of changed priorities, a more substantial review is justified. [6]

The second question is more difficult.  The average international organization has a large number of programmes on which it should report.  The programme performance reports based on outputs were generally unwieldy, even though they tended to present statistical summaries.  Explaining results is likely to be even more bulky.  And yet, if reviewing bodies cannot obtain detailed information they will not be able to make usable judgments.  The IAEA’s answer is to prepare a formal performance report based on exceptions (reporting in detail only on particularly effective programmes or those that have had problems), while maintaining a database that can be used to respond to specific questions about other programmes.

Will the measurements be effectively reviewed

In all results based systems, the ultimate method of accountability is an intergovernmental body capable of determining whether performance has been adequate, making judgments about effectiveness and applying conclusions to future programming.

In the United Nations that function has been assigned since 1976 to the CPC.  In other organizations, like the IAEA, it is a programme and budget committee.

The Secretary-General’s second phase reform proposals call for the monitoring and evaluation function of the CPC to be strengthened.  In some respects, this would be a form of compensation for removing much of the programme budgeting responsibility from the Committee.  In a results-based system, the monitoring functions is in many ways more important that the programme review function since it is through this review that the organizations and their managers will be held accountable to their Member States.  Since the CPC did not perform its programme planning review function very well the question can fairly be asked, is there any reason to believe that it could perform the monitoring function effectively?

If the present system is maintained, the answer would probably be “no”.  Its review of the programme performance reports over twenty years has not had many consequences either for individual organizations or the organization as a whole.  It has two documents on the basis of which its review is made.  The first is in-depth evaluations of specific programmes, which are independent of the programming cycle and tend to focus on managerial issues.  There is a follow-up report after two years.  The second is the biennial programme performance report that presents a static picture of performance.  Only on one occasion did it include data showing trends over several biennia.  The report prepared makes very few recommendations, and the CPC review tends to focus on process issues rather than drawing substantive conclusions.

Programme performance based on verifying whether expected results have been obtained will have to be presented and reviewed in a much different manner.  The report must be credible in the sense that it is clear that it reflects an honest appraisal of results rather than papering over lack of results by clever drafting and appeals to political themes designed to obscure poor performance.  The review by the CPC or any other intergovernmental body will have to draw strong conclusions that will provide an incentive to the Secretary-General to take steps to improve weak programmes or eliminate them, or to allocate more resources to effective programmes.  The conclusions should be drawn based on a clear analysis not only of the static picture during the biennium but also of longer-term trends.

This can be facilitated by the fact that the monitoring and evaluation reports are the responsibility of the Office of Internal Oversight Services.  One of the innovations of the 1990’s was the creation in the United Nations, and in an increasing number of United Nations system organizations, of independent inspection offices.  This has its basis in the role of inspectors-general in national governments, as well as of such institutions as the Operations Evaluation Department of the World Bank. In the United Nations, the Office of Internal Oversight Services (OIOS) under an Under-Secretary –General was created to be an independent body within the Secretariat. 

Unlike the area of programme planning, where support to the CPC is provided by the Programme Planning and Budget Division (PPBD), which has no incentive to provide critical analysis of its proposals since presumably the analysis was undertaken before the proposals were completed, if deployed with an independent philosophy, the OIOS could provide the “independent secretariat” that my earlier articles called for.  However, in making its report on programme performance, OIOS has acted as though it were a subcontractor to PPBD and has submitted its performance reports to that office for clearance.  If that practice is maintained, the OIOS will not be independent.

To function as that independent secretariat, its reports should not be subject to clearance and would have to provide the CPC with much more critical analysis of performance than hitherto.  The OIOS should undertake a major effort to assist programme managers to apply minimum quality standards to their own evaluation of results.  And, rather than accept these, the OIOS should add its own, impartial critical analysis, based on at least an entire plan period, that would show which programmes were ineffective and those that were.

In order to solve the dilemma of how to apply appraisal of results to programme planning and budgeting, given that the programme performance report will be considered too late to affect the review of the next biennium’s proposals, two innovations can be suggested.

First, following the model adopted by the IAEA, an interim performance report could be prepared after the first year of the biennium, that would present information on programmes or subprogrammes with particular effectiveness issues – positive or negative.  The information in this report could be used in reviewing the proposals of those programmes so that the CPC could make recommendations on corrective actions.

Second, given that the medium-term plan or its revisions are considered in the off-budget year and the plan is expected to be the programming framework, the OIOS could prepare a commentary on proposals based on its review of programme performance over several biennia.  This would enable the CPC to draw conclusions about the validity and utility of plan proposals based on the demonstrated capacity of the organization to carry them out.

What must be done? Expected Outcomes

In a results-based system, expected outcomes should be clearly expressed so that whether they have been obtained can be measured and verified.  It would only be fair that this should be applied to both the OIOS and the CPC in the United Nations (and by implication to their analogues in the other organizations of the United Nations System).

For the OIOS, the expected outcome is that it presents a strong analytical report to the CPC that demonstrably uses this to make critical conclusions and recommendations about United Nations programmes.

For the CPC, the expected outcome is that its critical conclusions and recommendations demonstrably change the Secretary-Generals proposals to emphasize effective programmes and improve or eliminate those that are ineffective.

If these outcomes are obtained, a major step forward will be made in answering the question, “Who Controls the Machine?”  And as a result, the machine will be more effective in achieving its objectives.

References

Australian Aid (2000), AusGuidelines, 1. Logical Framework Approach, http://www.ausaid.gov.au/ausguide/ausguidelines/1-1-2.cfm

Bertrand, Doris (2002), The Results Approach in the United Nations: Implementing the United Nations Millennium Declaration, (JIU/REP/02/2), Geneva: Joint Inspection Unit

Bertrand, Maurice (1974), Report on Medium-term planning in the United Nations System (JIU/REP/74/1), Geneva: Joint Inspection Unit.

Mathiason, John R. (1987), "Who Controls the Machine?  The Programme Planning Process in the Reform Effort,” Public Administration and Development, Vol. 7 (No. 2).

Mathiason, John R. (1997), “Who Controls the Machine, Revisited:  Command and control in the United Nations reform effort,” Public Administration and Development, Vol. 17 (No. 4).

Osborne, David and Ted Gaebler (1992), Reinventing Government: How the Entrepreneurial Spirit is Transforming the Public Sector, Reading, MA: Addison-Wesley Publishing.

United Nations (1997a), Renewing the United Nations:  A Programme for Reform: Report of the Secretary-General, (A/51/950).

United Nations (1997b), Results-Based Budgeting, Annex to Renewing the United Nations:  A Programme for Reform:  Report of the Secretary-General, (A/51/950Add.6), 12 November 1997.

United Nations (1998), Guide to Results-Based Budgeting, Version 1.01, Division for Programme Planning and Budgeting, 23 October 1998 (http://accsubs.unsystem.org/ccaqfb-intranet/RBB-RBM/UNguide.htm)

United Nations (1999), Joint Inspection Unit, RESULTS-BASED BUDGETING: THE EXPERIENCE OF UNITED NATIONS SYSTEM ORGANIZATIONS, (JIU/99/3), Geneva, Joint Inspection Unit.

United Nations (2001), Report of the Committee for Programme and Coordination on its Forty-First Session (A/56/16), 10 July 2001.

United Nations (2002), Strengthening of the United Nations: an agenda for further change, (A/57/350), 9 September 2002.

United Nations (2003a), Proposed programme budget for the biennium 2004-2005, Foreword and introduction (A/58/6 (Introduction)), 15 April 2003

United Nations (2003b), Report of the Committee for Programme and Coordination on its Forty-Third Session (A/58/16), 16 July 2003.

United States, General Accounting Office (2000). United Nations Reform Initiatives Have Strengthened Operations, but Overall Objectives Have Not Yet Been Achieved, (GAO/NSIAD-00-150), May 2000.

Wildavsky, Aaron (1979), The Politics of the Budgetary Process, Boston: Little, Brown.