Choose relevant indicators

Types of indicators

There are different types of indicators used for monitoring and evaluation:

• Context indicators do not directly concern the policy/programme being implemented but they help explain the bigger picture. They are used to contextualise the performance of a programme/policy.
• Inputs indicators measure what is invested in the implementation of the programme/policy. Inputs can be financial resources or human resources.
• Process indicators capture the activities put in place.
• Outputs indicators reflect what the programme directly produced, meaning the numbers of participants, products, services delivered, etc.
• Results indicators measure the direct achievements of the programme/policy. These should capture the direct change that the programme/policy led to. They should be related to what the programme aims to achieve.
• Impacts indicators measure the longer-term effects of a policy or a programme. Impacts can go beyond the individuals directly involved, and therefore it is important to look at effects at the level of organisations or society. 
• Structural indicators capture what is being done at the level of policy or system to tackle a certain issue and how it is done. They are typically either yes or no statements or they use other categories to capture systemic/policy features.

To get a full understanding of a programme/policy, one should analyse a broad set of indicators. A single indicator never tells the full story. Therefore, all of the above types of indicators should be reflected in the monitoring or evaluation framework.

Indicators can capture information that is quantitative as well as qualitative. Quantitative indicators focus on absolute numbers, percentages, ratios and other measures. They can also be formulated as a share of certain population rather than absolute number. Qualitative indicators capture types or categories, express breadth, denominate relations, etc. 
 

Examples of quantitative and qualitative indicators:

Relevant indicators

A key evaluator’s tool to develop a monitoring and evaluation indicator framework is the programme/policy intervention logic.

The result and impact indicators should be logically linked to what the programme aims to achieve. In other words, they should be able to measure the performance of the programme/policy against its objectives. They should be chosen carefully and there should not be too many of them. There can be a tendency among stakeholders to focus the programme delivery on those indicators that are being measured. This can have adverse effects on the quality of the service/programme. That is why:

• the indicators should be chosen wisely and capture what really matters
• they should be accompanied by a narrative that will translate the data collected into an explanation. This can be supported by the intervention logic.

The ultimate objective of programmes/policies concerned by this toolkit is to decrease the NEET by:

• creating education and/or employment opportunities for young people
• activating young people to enter the labour market or return to the education and training system

This however is not achieved directly by most programmes/policies. The NEET rate is a multi-faceted phenomenon which reflects a range of challenges and difficulties. Programmes/policies tackle these underpinning challenges and factors and lead to related changes at individual or institutional level. It is these intermediary results that are associated with a change in the NEET rate.

There is a certain time lapse between the moment when a person takes part in a programme and when s/he eventually finds stable employment. This can take several years. Often monitoring and evaluation information is needed to make decisions about programme/policy funding before this period elapses. It is hence very important to design indicators that also capture intermediary results.

Monitoring and evaluation indicators should be defined for each step in the logical chain as described in the programme theory or intervention logic. If indicators only capture outputs the evaluation does not say anything about the real change that can be attributed to the programme. If they only focus on the ultimate results and impacts, then it is not clear how concretely the programme made (or failed to make) a difference. If the change in ultimate impacts is small and no intermediate results are measured, it is not possible to see what aspects of the intervention logic and programme rationale are failing. This makes it difficult to recommend adjustments.

Cedefop research which underpins this toolkit shows that only a small number of evaluations pay sufficient attention to indicators for intermediary results. Many measure outputs and some measure impacts, without assessing what led to these impacts.

The links between intermediary results and impacts should be based on evidence. For instance, Cedefop research detected that skills degradation of young mothers who have spent extended time out of the labour force caring for their children is one of the factors leading to women being unable to find employment. Thus, a programme that aims at helping young women upskill to match the skills needed by employers in the area should ultimately contribute to tackling the high NEET rate.

How to define what a programme/policy is expected to change and why?

Every programme/ policy is underpinned by certain assumptions about what is the nature of the problem and how doing XYZ will change the status quo. These assumptions can be more or less explicit in programme documentation. Often it is necessary to discuss with those who designed the programme/ policy in order to understand these assumptions. It is by clarifying these that the evaluators develop the logical reasoning that captures: the problem, the nature of the activities put in place, the rationale that explains how doing X will help solve the problem. A good evaluation should not only measure whether the expected changes happened or not. It should also explain which of the initial assumptions were proved correct and which ones not.

Do evaluations only measure what is expected from the programme?

Evaluations are typically guided by the programme intervention logic which explains what is expected to happen. However, programmes can have unexpected effects too – both positive and negative. As these are unexpected, evaluators often become aware of them during the qualitative enquiry by talking to stakeholders or beneficiaries. In some cases, the unexpected effects may be so important that it is necessary to integrate them among the key indicators for the evaluation.