CONVERSION OF VISION INTO PROGRAMME OBJECTIVES AND OUTCOMES

Visions need to be converted into terms that express programmatically what is expected to be achieved. This language should describe what the overall situation envisioned will be like after (a) the programme intervention and (b) its resulting changes or improvements. This is done through articulating objectives and identifying the outcomes necessary to achieve them (with associated performance indicators). This attachment deals with these essential elements of planning in their respective order. [There is some overlap between the approaches since they are stand alone items. Not clear what this means.]

Definition: Objectives are the situation that should result if in the identified Member State needs are to be met. As such, they describe the situation that should exist at the end of a specified period as a result of Agency activities in a measurable manner aimed at certain identified target groups in Member States within that period of time. They represent what we want to accomplish (not how it should be done) in terms of certain identified target groups.

Objectives, which should be limited in number, are the overall or the highest level result to which a programme element is expected to contribute, directly or indirectly, over time as specified in the planning document. Objectives are intended to help direct the programme element in the direction of its vision and the Agency’s Medium Term Strategy. In terms of hierarchy, objectives should be derived from the Medium-Term Strategy and objectives for lower-order programme elements (e.g., subprogrammes and projects) should have a direct and obvious linkage to higher order objectives as described below

Rationale: By envisioning the desired future and outlining a broad roadmap to get there, it is more easily achieved. Objectives are the strategic directions that provide the link between the present and the future, between the Agency’s overarching framework and the component parts. By aligning present action with the future and a narrower frame with the broader frame, everybody knows where they are going, creating a synergy that enables staff to perform more effectively than they otherwise could on an individual basis.

Characteristics It is important that objectives not be confused with:

Activities. Do not confuse what managers do with what they intend to accomplish. Activities usually involve use of terms like "to promote, to develop, to contribute to, to assist or to strengthen".

Hopes that are laudable but not convincing. An objective that talks about achieving world peace would not be achievable by a given programme. Objectives have to be realistic and achievable within a reasonable time.

Outcomes resulting from the Agency’s outputs (services or products) within a particular period. While objectives are overarching, transcending any biennium period, outcomes are intervening changes within particular biennium.

In terms of what they are, key features of good objectives include:

1. They describe an expected end-state at a distinct point in time - the changes that are expected as a result of a programme. They describe what should be seen at the end of a specific time period if the programme is successful.[1]

2. They represent what the Agency wants to accomplish, not how it should do it.

3. They are aimed at meeting specific needs and interests of identified target groups within Member States; They should be realistic and attainable, and not overreach or promise what cannot realistically be achieved.

4. They are concrete, time-limited and measurable (i.e., “verifiable”).

Process: Other suggestions to help make their formulation easier:

· To describe the end-state requires imagining or “visioning” what should be seen at the end of a specific time period if the programme is successful.[2]

· Objectives should be concrete and measurable.

· They are by definition time-limited.

· Don’t confuse activities which is what is to be done (activity) with objectives which is the intention of what is to be accomplished.

· They should be realistic and attainable, modest but contributory. Don’t overreach or promise what cannot realistically be achieved.

Use: By establishing an “end-state” or result, the programme planning/formulation process and resulting document help:

· set directions that map what the Agency (programme element) aims to accomplish;

· provide a guide for a programme formulation and development;

· establish a set of shared expectations within the Secretariat and between the Agency and Member States, and thus the basis for common commitment;

· foster co-operation and co-operation across organisational lines; and

· provide the basis to develop outcomes and guide actions.

Setting programme directions helps produce fundamental decisions and actions that shape and guide what a programme (or organisation) is, what it does, and why it does it.

FORMULATING OBJECTIVES, OUTCOMES & PERFORMANCE INDICATORS

Tool for Drafting Results-based Programme Text

The introduction of results-based programming in the Agency means that particular care must be shown in drafting programme text in medium-term strategies and programme budgets. Realistic and accurate descriptions of the intended results of programmes and projects will help Member States review proposals and set resource levels, build support for Agency programmes and facilitative effective implementation of programmes and projects

This tool consists of a series of phases and steps to follow that can lead to sound programming text. It goes from the most general, objectives, to the specific, outcomes, to the most particular, performance indicators.[3]

This tool is intended to help programme staff formulate their objectives, outcomes and performance indicates by providing a series of steps, “tests” to validate their ?? and “frames” for reformulating those found to be lacking.

Step 1: Set a Time Frame: Programme objectives have to be time-limited. Setting the time period to be covered is an essential step. While other organizations of the United Nations system use a six-year horizon and the United Nations itself uses a four-year horizon, the agency’s medium-term strategy currently covers a five-year period. Objectives for a programme should, at a minimum, have a time horizon that reflects that medium-term planning time-frame.

Step 2: Establish a Hierarchy of Programme Objectives: There are different levels of objectives. Each programming unit (which generally corresponds to a level in the administrative hierarchy) has its own objectives, with higher levels having more general objectives, and lower levels having more specific objectives. Keeping the objectives at different levels consistent and linked is important.

In the Agency the programming levels and the most common equivalent administrative units are:

Programming Level	Administrative Unit
Major programme	Department
Programme	Division
Subprogramme	Section
Project	Unit

Step 3. Formulate Programme Objectives: Objectives should describe the end-state that should be accomplished within a defined period of time. As such they should be concrete and therefore observable (“verifiable” or “measurable”). They are the expression of a desired achievement, involving the process of change and aimed at meeting certain needs and interests of identified target groups in Member States within a given period of time.

Some of the terms that describe an end-state include "achieve, complete and have in place". The exercise of description requires imagining (visioning) what should be seen at the end of a specific time period if the programme is successful.

Establish the Objective: Try to describe the situation that is intended to be seen in the future, in terms of conditions, structures and behaviours in terms of the subject matter being planned in the context of Member State needs and priorities. The objectives should be defined in terms of the end-states to be reached within a specific time period in each areas dealt with by the programme. They should be precise, time-limited and should be observable.

The easiest way to so this is to first do a critique of the objective as expressed in the programming document in terms of whether it really indicates what is supposed to be achieved over the longer-run. The following checklist below should be applied to the statement of objectives to determine how well formulated they are.

Looking at the objectives that have been formulated or reformulated, apply the following test:

Does the Objective	Yes	No
1. Clearly define the expected impact (change) on the problem or need?
2. State achievable results in verifiable terms?
3. Have long term applicability?
4. Have time specified?
5. Indicate how the target group will be affected?

Are the Objectives 'SMART'?	Yes	No
1. Are they directed towards Specific results?
2. Can progress be Measured (verified) for each of them.
3. Are they Achievable ?
4. Are the objectives Relevant?
5. Can they be done in a Timely manner?

If the answer to all of the questions is “yes”, the objectives are solid and can be evaluated. If the answers are “no,” go back to the frame for applying this step, namely the Formulation of Objectives” below until it is correctly done.

Frame for Applying the Step

Current Formulation of Objective

Critique/Analysis

Reformulation of the Objective:

Completion of Frame: an Illustrative Example based on MTS

Current Formulation of Objective

Critique/Analysis

A.2: Contributing to the building of international consensus on solutions for the safe, environmentally acceptable and efficient management of radioactive waste from both nuclear power and non-power sources by providing technical guidance and facilitating the exchange of information on matters such as ..

· Contributing to is an activity and an undefined one at that: what does contributing mean? [It could be just sending a letter!]

· when was the international consensus expected to be achieved? Within the next five years? Was it expected to be a complete consensus on everything, or is there some specific consensus on some specific solutions that was foreseen?

· what is meant by “safe, environmentally acceptable and efficient management of radioactive waste”? These are undefined activities

Reformulation of Objective:

Agreement by at least ten Member States having waste storage backlog to pilot-test specific options developed as a (partial) result of Agency assistance (by 2005).

After having reformulated the objectives, go back and apply the “test” once again. If the answer to all the questions is “yes,” the objectives are solid and well formulated. Then proceed to the next step dealing with the expected outcomes (below). If the answer is “no,” go back to the “Reformulation of Objectives” until it is correctly done.

Step 4: Map the Expected Outcomes: The relationship of ends and means – objectives, outcomes and outputs - that will be evaluated is seen below:

Goals - Agency’s overarching programmatic purpose in the MTS

Objectives

Objectives

- Specifically planned time-limited achievements to reach the goal

- Outcomes necessary if objective is to be achieved

Outputs

Outputs

- Outputs (products or services) that contribute towards
inducing the preconditioning outcomes

For each objective, determine the outcomes (i.e, changes) that the programme planners believed necessary for the objective to be achieved. Outcomes are changes in the behaviour of the parties involved or in the structure of the environment. Most objectives can be achieved only because a number of outcomes occur. These often have to take place in a logical sequence. The task is to set out what outcomes or changes would have to have been induced, in what order, in order to arrive at the specified end-state defined in the objective.

Outcomes are sometimes difficult to envision, in part because they are usually outside the complete control of the Agency. Governmental agreement, for example, is an outcome that is often needed in international management, but this is influenced by state politics as well as by the work of the Agency. Assuming that one can control one’s own programme activities, the question to ask for each objective is “what would I have to have induced in areas outside my control in order to reach the end-state?”

The reality testing of the objective should have provided a basis for judging whether inducing the outcomes is realistic or not. However, it is important to estimate how long it will take to induce each outcome. If the total time exceeds five years, it will be necessary to go back and look at the way the objective is formulated.

Once the outcomes have been mapped and the output necessary to obtain them generally specified (see below), the outcomes as contained in the programming documents should be tested as follows:

Are the outcomes	Yes		No
1. Specific and observable changes in the problem or need?
2. The intermediate benefit resulting directly from one or more outputs?
3. Induced by the Agency's outputs?
4. Intended to cumulatively lead to achievement of the objective(s)?
Are the Objectives 'SMART'?		Yes	No
1. Are they directed towards Specific results?
2. Can progress be Measured (verified) for each of them.
3. Are they Achievable ?
4. Are the objectives Relevant?
5. Can they be done in a Timely manner?

If they do not meet these tests, they should be reformulated to represent what the Agency really intended, using the critical analysis/reformulation frame below.

Completion of Frame: an Illustrative Example based on the 2002-2003 Programme Budget (Subprogramme J.5)

Current Formulation of Outcome(s)

Critique/Analysis

Increase in Member State capability to operate nuclear plants safely.

The Agency’s programme does not address all of the factors necessary to operate a plant successfully

Is not directly related to the Agency’s output.

Not clear what “capability to operate” means and therefore concept of “increase” is not clear.

Reformulation of Outcome:

Adoption of nuclear installation safety performance improvements identified by review missions and via an international set of safety performance indicators.

More often than not, what was given in the programme and budget as a performance indicator is, in fact, the intended outcome.

Step 5: Note the Outputs that should be produced: Note what the Agency should define as output necessary to induce the outcomes. This will be reflected in what was budgeted in terms of such output as technical assistance missions, expert group meetings, reports and the like.

If objectives are ends, then outputs – products or services – are means. In looking at the outputs, the following checklist should be applied:

Are the outputs either a product or service that	Yes	No
1. Is delivered by the Agency to the intended (MS) clients?
2. Are directly within the control of the Agency’s programme?
3. Directly result from the Agency's activities?
4. Lead to higher level results, i.e., outcomes and thence objectives?

If all are answered “yes,” the items are outputs. If not, some are not outputs and they need to be eliminated to take note of only the outputs. [As a point of comparison, activities are actions that transform inputs into outputs.]

Step 6: Define Performance Indicators: Once the objectives and outcomes have been mapped, the formulation of performance indicators is straightforward.

Performance Indicators are predicted features (characteristics, signs, checkpoints, yardsticks) used to verify or measure whether objectives have been or are being achieved. They indicate the progress of the programme in reaching its objective and provide a basis for early problem identification and solution as part of programme monitoring and to evaluation programme results.

These “expected results” are the tangible outcomes that are the direct consequence or effect of the generation of Agency outputs (products or services), leading to the fulfilment of a specified objective.

· Because Performance Indicators reflect outcomes, both must be “performance” or “result” – not activity -- oriented.

Performance indicators should be measurable or observable, relevant, meaningful and verifiable but should be limited only to the few most critical ones.
They should be selected for performance data which can be collected and appraised at least annually as to allow their measurement in a cost-effective manner.
Focus only identifying those that will tell you what you really need to know to ensure that expected results are achieved as and when required. Include those which you find useful and therefore can “live and work with”

The indicators should be directly linked to the programme outcome. They should be chosen and expressed in such a way as to permit a reasonable judgment to be made of the degree of success achieved. In effect, the indicator answers the question: “Did the outcome happen as planned?”

An example of defining a performance indicator from different programmes.

Frame for Performance Indicators:

Outcome	Corresponding Performance Indicator
· Increased application of standards and practices recommend by the Agency in its publication, databases, CRPs, technical cooperation projects and services.	Extent of utilization of and indication of interest in, Agency publications, Databases, CRPs, technical co-operation projects and services in the area of nuclear power.
· More laboratories participating in Agency's proficiency tests and demonstrating improved performance and number of national laboratories which have achieved 150 17025 or GLP accreditation for analysis of food contaminants and residues.	Number of laboratories
· Measurable safety improvements in nuclear installations resulting from international peer safety reviews of nuclear power programmes.	Extent to which recommended improvements were adopted
· Conclusion and subsequent implementation of safeguards agreements and Additional Protocols in an increasing number of States.	Number of States concluding and implementing the Additional Protocol.
· Improvement of the level of understanding of nuclear issues by key constituencies.	Readership surveys, providing feedback on publication content, quality and usefulness. Increased requests for print and audio-visual information materials, including bulk orders from Member States.
· Increased use of key programme/project information in programming and implementation by Member States	Extent of use.
· Timely recruitment of well-qualified staff required by managers to deliver their programmes.	Number of staff recruited compared with the number of requests for recruitment.

For each performance indicator the following test should be performed:

Do the performance indicators	Yes	No
1. Verify (measure) whether the outcome has occurred or not?
2. Relate exclusively to the intended outcomes?
3. Lie only partially within the influence of the Agency?
4. Base themselves on performance information about the intended results?

If the indicator fails the test, they should be reformulated so that they can be appropriately measured, using the same reformulation tool as provided below.

Frame for Applying the Step

Current Formulation of Performance Indicators

Critique/Analysis

Reformulation of Performance Indicators:

[1] Some of the terms that describe an end-state include "achieve, complete and have in place". This requires imagining what should be seen at the end of a specific time period if the programme is successful.

[2] Some of the terms that describe an end-state include "achieve, complete and have in place". This requires imagining what should be seen at the end of a specific time period if the programme is successful.

CONVERSION OF VISION INTO PROGRAMME OBJECTIVES AND OUTCOMES

Administrative Unit