When to collect the information for precess, impact and outcome evaluation

Baseline data describe the situation before the project or intervention happens. Therefore, baseline information must be collected before the start of a project. It is used during the project to indicate progress towards the goal and objectives and after the project to measure the amount of change. The information collected in the needs assessment can be part of the baseline data.

It is important that the information collected and used for baseline data actually describes the situation that the project goal and objectives are addressing. If the information is not relevant, you will not be able to evaluate the goal and objectives properly.

The Northern Territory Living With Alcohol (LWA) program provides a good example of the use of baseline data in evaluation.

Baseline data and evaluation: an example

Objective
To reduce alcohol related road accidents by 56 per cent. (Note: This objective is one of several objectives. Only one is used for the purpose of this example).

Program Activities
Examples of strategies used to reach this particular objective include:

Although not shown in the table, in the period 1982-1990 the annual percentages of alcohol related accidents fluctuated, but was consistently within the range 19.7 per cent to 25.3 per cent. The 1991 figure of 20.4 per cent is used as baseline to measure progress towards the objective of a 56 per cent reduction by the year 2000. These data indicate a steady decline since the establishment of the program. Comparison of the 1997 figure with the 1991 baseline figure shows the reduction:

This information suggests steady progress towards the objective of a 56 per cent reduction by 2000.

Step 2: information for process evaluation

Data used to measure the project strategies are collected as the project proceeds. Data collection needs to be coordinated with the actioning of the strategies.

For example, the numbers of men who came to the well men's check and education session need to be recorded at the time. What people thought about a video being shown in the health centre needs to be observed while they are watching and their responses collected just after the video is finished.

As shown in the planning section of this chapter, the strategies of a project are the methods that the project is using to achieve its objectives. The strategies must be evaluated before the objectives can be evaluated. If the strategies are not going well (see the four questions about process evaluation explored before) then there is little chance of reaching the objectives. Therefore, there is little point evaluating whether the objectives have been reached.

Step 3: information for an impact evaluation

In order to see whether the objectives have been achieved, it is necessary to collect data which relate to the objectives:

Compare the two sets of information. Is there a difference? What is the difference? Is it what was being aimed for? Has the objective been achieved?

Example of process and impact evaluation processes

Objective	Double school children's knowledge about the link between smoking tobacco and health, by the end of the week
Strategy	Deliver education session to school children on link between smoking tobacco and ill health. Do it in partnership with Aboriginal teacher
Step 1 Collect baseline data	Before actioning the strategy, find out children's experiences with smoking tobacco. Conduct a simple test finding out what children know and believe. You now have baseline data.
Step 2 Implement the strategy and conduct a process evaluation	Show the effects of tobacco on the body. Build on children's stories and experiences. Conduct a process evaluation to make sure that the strategy was well implemented.
Step 3 Conduct an impact evaluation	After the strategy has be actioned, conduct a simple test. What do the children know and believe now? The difference in knowledge and beliefs now compared with before the education session is the measure for the impact evaluation

Step 4: information for an outcome evaluation

Collecting data to measure the goal is done at the end of the project. It is then compared with the baseline data that described the situation just before the project started.

The goal is reached by achieving the project objectives. If the impact evaluation shows that the objectives have not been met, then there is little point measuring to see if the goal has been met.

Sometimes changes will need to be monitored on a regular basis after the project has been implemented. It may take some time for the longer term outcomes to show up. It will also be necessary to see if any changes that occurred immediately after the project have been maintained.

The amount of time required for monitoring change and before final evaluation of the goal will vary, depending on what the goal is and how quickly change can be expected. This time factor is something the project team will have to think about and discuss when planning the evaluation. Knowledge levels may change before attitudes or beliefs change. Behaviours, policies, legislation and other social factors may need to change before health status will change. Some health conditions are more readily changed than others.

The information in the following case study is used to show an example of an outcome evaluation and how the different types of evaluation are related to each other.

Outcome Evaluation: A Case Study
Taps, showers, kitchen sinks and toilets: Anything to do with Chronic Diseases?

…Ramingining Community has over the last 12 months improved the functionality of the health hardware in most of the (54) local houses and this seems to have had a remarkable effect on the health of the people.

By surveying the houses (as part of the Environmental Health Infrastructure Minimum Standards Project - EHIMS) and testing every power point, tap, shower, toilet and then providing a list of all the problems to the tradesmen, the Community has been able to improve the functionality of the houses dramatically.

The houses were rated for health functionality out of a possible 21[points]. In 1997 the Mean Rating was 8.9/21, in 1998 this rose to 13.6/21.

At the same time there seems to have been some impressive movement on the health front. The numbers of people treated at the health centre [presentations for treatment] have dropped from around 300 a month (the mean over the 6 months prior to the project) to 150 a month (the mean over the same 6 months of the following year).

Dramatic drops seem to have occurred in the numbers of -

Skin infections/scabies cases - down 69%
Respiratory infections - down 39%
Diarrhoea and Vomiting - down 35%.
[Note: These health problems have been linked to poor environmental conditions.]

Extracts from a report by Bill Hardy (1998) in 'The Chronicle' 2(4):1

The goal of this project might be to reduce by 50% the number of people being treated each month at the health centre for environmental related health problems within 12 months of starting the program.

The objective might be to increase the functionality of the houses by 50% with 12 months of starting the program.

The major strategy might be to fix all damaged and non-functional taps, toilets, showers and kitchen sinks and faulty power points.

Step 1 Collect baseline data	Before the strategy is actioned: Assess the functionality of houses and rate them out of 21 possible points. The mean rating is 8.9 points out of 21 points this figure will be the baseline figure to measure achievement of the objective. Check health centre records for the number of treatments per month in the 6 months before the project. There is an average of 300 treatments per month. This figure will be the baseline figure to measure achievement of the goal.

Step 2 Implement the strategy and conduct a process evaluation	A plumber is employed and provided with a list of houses with damaged plumbing. The plumber carries out the repair work. At the same time an electrician is employed to repair faulty power points. Conduct a process evaluation to assess how well the strategy was carried out.

Step 3 Conduct an impact evaluation	Assess the functionality of houses and rate them out of 21 possible points. Compare this figure with the baseline figure. The mean rating is 13.6 points out of 21 points. There has been an increase of 52.8%. This figure is the measure for the impact evaluation. The objective has been achieved.

Step 4 Conduct an outcome evaluation	Check health centre records for the number of treatments per month in the 6 months after the project. There is an average of 150 treatments per month. There has been a reduction of 50%. This figure is the measure for the outcome evaluation. The goal has been achieved.

When to collect the information for process, impact and outcome evaluation

Step 1: baseline data

Baseline data and evaluation: an example

Step 2: information for process evaluation

Step 3: information for an impact evaluation

Example of process and impact evaluation processes

Step 4: information for an outcome evaluation