monitoring and evaluation protocol marine and rainforest operations

MONITORING AND EVALUATION PROTOCOL

Marine and Rainforest Operations

Let’s assume that this is an operator (Danta Tours) which carries out activities in the “El Jaguar”

tropical forest. The forest where they carry out their operations is a combination of well

conserved and secondary forests and it makes up part of a protected area. There are pathways

in the forest where tourists embark on walking trips, birdwatching, cycling and kayaking on a

calm river. The operator also offers transport services to and from the city. The local

communities are mainly involved in agricultural activities, as well as small scale hunting and

fishing.

Hypothetical tourism operations: tropical forestsHypothetical tourism operations: tropical forests

Let’s assume that the operator Delfin Tours is the owner of a small boat

driven by an outboard motor which operates tours in several of the

Islands in the archipelago “Islas del Pacifico” whose ecosystems are

considered fragile. The boat carries out trips to the islands where

tourists embark on walking tours, snorkeling and diving.

Hypothetical tourism operations: marine operationsHypothetical tourism operations: marine operations

What is sustainable development?What is sustainable development?

Economic development•Profits

•Market expansion

Social development•Social equity

•Minimal needs covered•Participation

•Stability•Cultural respect

Environmental development

• Ecosystem integrity• Biodiversity conservation

• Vital systems


Economic development Social development

Environmental development

Sustainable development


Sustainable development meets the needs of the present without compromising the ability of future

generations to meet their own needs.

Bruntland Report

What is sustainable tourism?What is sustainable tourism?

Monitoring is the collection and analysis of repeated

observations or measurements to evaluate changes in

condition and progress toward meeting a management

objective.

What is monitoring?What is monitoring?

The main objectives of measuring performance in

sustainable tourism are the following:

1) To determine if good tourism practices are mitigating otherwise

extensive impacts on the environment and welfare of local

communities.

2) To determine incremental progress towards achieving

sustainability through environmentally and socially responsible

practices without sacrificing business goals.

What is monitoring?What is monitoring?

Performance and impact based monitoringPerformance and impact based monitoring

Impact-based indicators

No. Indicators How To Measure Units Baseline Month 2 Month 3 Month 4 Month 5 Month 6

1Number of fuel gallons consumed by boats during tourism operations Keep receipts Gallons 15 15 12 10 7 4

2Percentage of satisfied visitors by the guide: visitors ratio Visitor satisfaction oriented surveys % 60% 63% 68% 72% 82% 91%

Baseline and Monitoring

Performance-based indicators


No. Compliance Non - compliance Does not apply ComplianceNon -

compliance Does not apply ComplianceNon -

compliance Does not apply

1x x x

2x x x

Indicators (based on best practices)

The operator has a code of conduct or another system in place which informs visitors of the norms relating to behaviour during visits to local communities.

The operator has established sites where visitors can carry out activities benefiting other local businesses

Baseline 1st evaluation 2nd evaluation

Performance-based: This type of monitoring is used in order to determine how efficient a operation can be by applying a measurements related to common criteria or benchmarks that must be met

Impact-based: This type of monitoring is used in order to measure negative impacts or threats caused by tourism operations

Performance and impact based monitoring frameworkPerformance and impact based monitoring framework

Reduce fuel consumption by

20% by the end of 2008

Reduce fuel consumption

Consider fuel efficiency when purchasing new

vehicles.

Perform regular maintenance to ensure they are

running efficiently.

Consider fuel consumption

when designing tour routes.



Impact/threat

Objective

High fuel consumption

Goal

Good practices (and activities)

Who monitors what?Who monitors what?

Monitoring carried out by tour operators, which must be led by administrators, business owners, guides and others who make up the operative personnel. E.g. monitoring the quantity of fuel used by the boats within the operation, monitoring customer satisfaction, amongst others.

Monitoring carried out by external parties, including park rangers, foundations, associations, chambers and ministries, amongst others. E.g. monitoring the number of illegal hunting and fishing incidents, monitoring the quantity of litter on pathways in a protected area, amongst others.

Monitoring carried out by evaluators, which must be led by the workshop. The evaluators monitor the operator’s initiatives and measure the implementation of good practices. E.g. monitoring the implementation of purchasing policies that benefits local producers.

A monitoring plan must be recurrentA monitoring plan must be recurrent

Start of plan

Evaluate and readapt

Use the results to learn

Develop a monitoring plan

Implement monitoring plan

Analyze information and

make conclusions

Develop objectives, goals and activities

Identify impacts and threats

Start of plan






make conclusions



Start of plan






make conclusions



Monitoring frameworkMonitoring framework

Identify threats and negative

impacts (triangulate information

when possible)

1

Workshop

6

Define activities (based on best

practices)

Evaluator and operator

Triangulation, revision of results by

responsible in charge

Identify causative

factors behind threats and

factors

2

Workshop

Redefine those threats and

impacts that fall within the TO

sole responsibility

3

Workshop

Define objectives to

address threats and impacts

4

Workshop

Define goals to address threats

and impacts

5

Workshop

7

Choose or define impact

based indicators and

performance indicators


Establish baselines for impact based indicators and performance

based indicators

8

Evaluator on a tour


10


Operator: impact based

Evaluator: Performance based

9

Monitor

1Identify threats and negative

impacts (triangulate

information when possible



when possible)

Identify causative


factors



sole responsibility




and impacts

1 2 3 4 5

7 6


based indicators

8


10

Workshop





practices)



responsibles in charge

Workshop Workshop Workshop Workshop


Evaluator on a tour




9

Monitor

Identify threats and negative impactsIdentify threats and negative impacts

Tourism in El Jaguar tropical forest


Conflicts with communities exist

due to tourist behavior

Low fauna observation levels

Visitor complaints

High levels of water contamination due to

oil leakages

Litter is present on the pathways

2Identify causative


factors



when possible)

Identify causative


factors



sole responsibility




and impacts

1 2 3 4 5

7 6


based indicators

8


10

Workshop





practices)






Evaluator on a tour




9

Monitor

Identify causative factors behind threatsIdentify causative factors behind threats






Visitor complaints

Fuel efficient vehicles are not

used

Large groups


oil leakages

Illegal hunting







Visitor complaints


used

Large groups


oil leakages

Illegal hunting



Triangulate information


Triangulate information

Threat/ Impact A B C D E Total Low levels of customer satisfaction

1 3 2 1 2 9

High levels of fuel consumption

2 2 1 3 1 9

Water contamination 3 1 3 2 3 12

3Redefine those

threats and impacts that fall

within the TO sole responsibility



when possible)

Identify causative


factors



sole responsibility




and impacts

1 2 3 4 5

7 6


based indicators

8


10

Workshop





practices)






Evaluator on a tour




9

Monitor






Visitor complaints


used

Large groups


oil leakages

Illegal hunting


Redefine TO’s sole responsibility factorsRedefine TO’s sole responsibility factors

Examples of threats and negative impactsExamples of threats and negative impacts

Examples of threats and negative impacts: Some examples of threats or negative impacts are:

1. High levels of fuel consumption 2. Lows levels of customer satisfaction in large visitor groups 3. Conflicts with communities due to the behavior of tourists 4. Low incidence of observation of fauna species 5. High levels of water contamination caused by gas and oil spillages

Monitoring tool (impacts and threats)Monitoring tool (impacts and threats)

Objective:

Goal:

Threats / Impacts/Problems:

Activities (based on best practices):

Impact based indicator:

Performance based indicator:


4Define objectives to address threats

and impacts



when possible)

Identify causative


factors



sole responsibility




and impacts

1 2 3 4 5

7 6


based indicators

8


10

Workshop





practices)






Evaluator on a tour




9

Monitor

Define objectivesDefine objectives

Some suggestions for establishing the characteristics of objectives are as follows:

VisionaryAn ideal objective gives all those involved in the operation a good overall vision of what is set out to be achieved and what is being focused on. The comprehension of objectives is fundamental for all members of the operation.

Bad. To increase the number of employees: It can be an objective but could be written in a more inspiring form. Good. Increase the number of jobs to improve the local conditions.

Relatively generalAn objective must be generally defined in order to incorporate all activities within the project that are to be fulfilled. Establishing dozens of objectives for each threat is not recommended.

Bad. To decrease organic and inorganic garbage in the office and on the field (too specific)Good. To reduce amount of waste

Define objectivesDefine objectives

BriefThe objective statement should be simple, concise and easy for all project participants to remember.

Bad. To reduce fuel consumption by using fuel efficient vehicles, appropriate driving techniques and regular maintenance (too long and confusing)Good. To reduce fuel consumption.

MeasurablePart of the plan’s success lies in the ability to measure and precisely evaluate how far its objectives have been achieved.

Bad. To reduce conflicts within the community (many conflicts may not be related to your activity)Good. To reduce conflicts with communities caused tourism

Examples of objectivesExamples of objectives

Examples of objectives The objectives for threats listed in the previous example could be:

1. To reduce fuel consumption 2. To reduce the number of visitors per guide 3. To reduce conflicts with communities caused by visitor behavior 4. To increase the number of primate observations 5. To prevent the water from being contaminated by gas or oil



Monitoring tool (objectives)Monitoring tool (objectives)

Objective:

Goal:






To reduce fuel consumption

5Define goals to address threats

and impacts



when possible)

Identify causative


factors



sole responsibility




and impacts

1 2 3 4 5

7 6


based indicators

8


10

Workshop





practices)






Evaluator on a tour




9

Monitor

Define goalsDefine goals

Some suggestions on how to set goals:

Directed towards a threat or impactPerhaps the most important criterion for an objective is that it must be relevant to the factor or factors in the conceptual model to which changes are to be affected. A good goal must be defined in such a way that its fulfillment guarantees the modification of the desired factor and will therefore eventually affect the condition of interest.

Threat: Little local involvement in the operation.Bad. To organize at least 3 monthly meetings with community members (the number of meeting is irrelevant when trying to reduce the impact) Good. Increase the percentage of local employees from 10% to 30% by the end of the year.

Defined time frameIf the goal is set without defining a date by which it is expected to be reached, it will be difficult to tell whether or not the plan has been successful. However, if a time limit has been specified, and by the end of this time period the goals have not yet been reached, the chronogram should be revised or the goal should be changed based on the efforts being made in to achieve the goal.Bad. To reduce the guide-visitor ratio to 1: 15Good. To reduce the guide: visitor ratio 1: 15 by the end of 2007

Define goalsDefine goals

SpecificA good goal must be specific, in such a way that all parties involved in the project understand it. If a goal is too general it is possible that you won’t be able to tell if it has been achieved or not. Bad. To reduce fuel consumption (it is not clear if it refers to the generator, vans or outboard motors) by the end of 2008.Good. To reduce fuel consumption of the vans in a 20% by the end of 2007.

PracticalThis is a difficult criterion to define without referring to local conditions at the operation site. Bad. To improve local capacity by creating a school in the village with governments funding by the end of 2008 (you may not get the funding and that is not your area of expertise) Good. To provide practical training to 5 locals between January and august, 2008.

MeasurableA goal must be able to be measured at any point within the project, even if it is general. How far are you from achieving your goal? For each goal set, there needs to be a scale in place (numbers, percentages, fractions or “all or nothing” measurement, which can measure its progress). Bad. To reduce oil and gas spills and leakages.Good. To reduce the number of incidences of water being contaminated by oil or gas generated by outboard motors to a maximum of one per week by June 2007

Examples of goalsExamples of goals





Example of goals The goals for the proposed objectives in the previous exercise could be:

1. To reduce fuel consumption by 20 per cent by the end of 2008 2. To reduce the guide: visitor ratio 1: 15 by the end of 2007 3. To reduce conflicts with communities by at least one per month by December 2007 4. To increase the observation of primates by at least one per tour by December 2008 5. To reduce the number of incidences of water being contaminated by oil or gas generated

by outboard motors to a maximum of one per week by June 2007

Monitoring tool (goals)Monitoring tool (goals)

Objective:

Goal:







To reduce fuel consumption by 20% by the end of 2008

6Define activities (based on best

practices)



when possible)

Identify causative


factors



sole responsibility




and impacts

1 2 3 4 5

7 6


based indicators

8


10

Workshop





practices)






Evaluator on a tour




9

Monitor

Define activitiesDefine activities

LinkedThe activity should be directly linked to the achievement of a specific goalFocusedDefine specific tasks which must be carried outFeasibleThe activity should be feasible taking into consideration the resources and limitations of the projectAppropriateThe activity must be acceptable and admissible with regards to specific cultural, social and biological norms in the area.

For this monitoring plan, activities related to the implementation of good practices (suggested in the “A tour operator’s guide to tropical forests” and “A tour operator’s guide to marine/coastal based tours”).

Examples of activitiesExamples of activities

Examples of activities Goal 1: Reduce fuel consumption by 20 per cent by the end of 2008 Applicable activities:

Consider fuel efficiency when purchasing new vehicles. Choose vehicles that minimize fuel consumption or use renewable energy. Perform regular maintenance to ensure they are running efficiently. Consider fuel consumption when designing tour routes.

Goal 2: Reduce guide: visitor ratio to 1:15 by the end of 2007 Applicable activities:

Keep tour numbers to a manageable and personable size. Smaller tour groups are always easier to manage and allows for the development of a more personal and tailored approach

Goal 3: Reduce conflicts with communities by at least one per month by December 2007. Applicable activities :

Negotiate with cultural groups on appropriate access, behavior and interpretation regarding heritage, culture and people.

Avoid highly sensitive or private sites on the tour and seek local community endorsement in the selection of sites visited.

Keep visitors to defined areas and routes and using shared infrastructure and services in off peak times will also help minimize disturbances to local residents

Monitoring tool (activities)Monitoring tool (activities)

Objective:

Goal:








Consider fuel efficiency when purchasing new vehicles

7Define activities (based on best

practices)



when possible)

Identify causative


factors



sole responsibility




and impacts

1 2 3 4 5

7 6


based indicators

8


10

Workshop





practices)






Evaluator on a tour




9

Monitor

Define impact and performance based indicatorsDefine impact and performance based indicators

The following indicators have been defined for this manual:

Impact-based indicators. Indicators that serve to measure the progress of the goals and objectives established beforehand.

Performance-based indicators. Indicators that serve to measure the implementation of activities related to best practices.

What is an indicator? An indicator is a unit of information which is measured in the passing of time, documenting the changes in a specific condition.

Define impact and performance based indicatorsDefine impact and performance based indicators

Reduce fuel consumption by

20% by the end of 2008

Reduce fuel consumption

Consider fuel efficiency when purchasing new

vehicles.

Perform regular maintenance to ensure they are

running efficiently.

Consider fuel consumption

when designing tour routes.



Impact/threat

Objective

Highfuel consumption

Goal

Good practices (and activities)

Impact based indicators

Impact-based indicators should be generated for each operation using goals and objectives as the base for doing so.

Define impact based indicatorsDefine impact based indicators


To establish an indicator, analyze the goal or activity for which the indicator is required and consider the various types of information that will need to be compiled in order to evaluate it.

Indicators must cover all information required. Goals, objectives and activities should be revised in order to determine the information that is needed to apply an indicator.

On establishing an indicator, the following criteria should be taken into consideration:

MeasurablePerhaps the most important criteria for an indicator is that it must be measurable either in terms relating to quality or quantity. Objectives as well as goals and activities should be defined in such a way that they are measurable.

Not measurable: Amount of tourists illegally taking natural objects from the forest “El Jaguar” (its not measurable given that it is illegal and tourists will try to hide such objects)

Measurable: Male/female employee ratio in Danta Tours offices and field employees


PreciseAnother important criterion for an indicator is that it must be defined in a precise manner and understood by all parties. The case is generally that several individuals will compile information in order to measure a given indicator. It is therefore important that all parties involved compile the information using the same method to ensure that the indicator is precise.

Not precise: # of education-oriented tourism offerings made by Delfin Tours on the Islas del Pacifico archipelago (is it tourism offerings per day? per tour? per group of tourists?)

Precise: Percentage of tourists who bought the tour from Delfin Tours due to recommendations by friends of tourists.


ConsistentAn indicator must not change with time, in the way that it always measures the same thing. If an indicator is providing a reliable measurement of the changes in an objective, goal or activity, is it important that the observed effects are seen into real changes in the condition and not changes in the indicator itself.

Inconsistent: (To determine the impact that boats have in the lake within El Jaguar reserve.): Number of bird species at the lake (it is not valid because some species are migratory)Consistent: liters of water consumed per month by Delfín Tours

SensitiveIt changes proportionally in response to real changes in the condition or concept that it measures

Not sensitive: Liters of fuel consumed by the canoes used by Delfin Tours (will reach a point where although canoe consumption is reduced, it can’t be reduced to less than that required for tours)Sensitive: Number of visitors that Danta Tours receives every year

Examples of impact-based indicatorsExamples of impact-based indicators

Examples of impact based indicators Some examples for impact based indicators are: Objective 1: To reduce fuel consumption Goal: Reduce fuel consumption by 20 per cent by the end of 2008 Activity: Redesign the tour routes to minimize the # of gallons consumed on each trip. Indicator: Number of fuel gallons consumed by boats during tourism operations Objective 2: To reduce the number of visitors per guide Goal: Reduce guide - visitor ratio to 1:15 by the end of 2007 Activity: Hire 2 extra guides to minimize the guide-visitors ratio. Indicator: Percentage of satisfied visitors by the guide: visitors ratio (gauged by visitors fulfillment surveys) Objective 3: To reduce conflicts with communities caused by visitor behavior Goal: Reduce conflicts with communities by at least one per month by December 2007. Activity: Organize a meeting with community members to identify sensitive aspects that are being affected by visitors’ behavior. Indicator: Number of registered incidents between communities and operator due to visitors’ behavior

Examples of impact-based indicatorsExamples of impact-based indicators





Example of goals The goals for the proposed objectives in the previous exercise could be:

1. To reduce fuel consumption by 20 per cent by the end of 2008 2. To reduce the guide: visitor ratio 1: 15 by the end of 2007 3. To reduce conflicts with communities by at least one per month by December 2007 4. To increase the observation of primates by at least one per tour by December 2008 5. To reduce the number of incidences of water being contaminated by oil or gas generated

by outboard motors to an average of a maximum of one per day by June 2007

Examples of impact based indicators Some examples for impact based indicators are:

1. Number of fuel gallons consumed by boats during tourism operations 2. Percentage of satisfied visitors by the guide: visitors ratio (gauged by visitors fulfillment

surveys) 3. Number of registered incidents between communities and operator due to visitors’ behavior 4. Number of registered primate observations. 5. Number of oil spills and leakages produced in water bodies by the operation’s outboard

motors.

Monitoring tool (impact-based indicators)Monitoring tool (impact-based indicators)

Objective:

Goal:








Consider fuel efficiency when purchasing new vehicles

Number of fuel gallons consumed by boats during tourism operations

Define performance-based indicatorsDefine performance-based indicators

Performance based indicators

Performance-based indicators should be capable of measuring the success of the implementation of best practices. These indicators help the evaluator to measure the aforementioned implementation: for each best practice there is an indicator based on performance, which should be rated in the following way:

• Compliance: the operator has fully implemented best practices according to the indicator.• Non compliance: the operator has not implemented best practices according to the indicator. • Does not apply: the indicator does not apply to the operator’s activities.

Define performance based indicatorsDefine performance based indicators

Performance-based indicators have been developed after “A tour operator’s guide to tropical forests” and “A tour operator’s guide to marine/coastal based tours”, but not all possible indicators are used in this workshop. New indicators or variations of those suggested can be utilized depending on the context of the site and operators but always related to the guides mentioned above.

Monitoring tool (performance-based indicators)Monitoring tool (performance-based indicators)


The performance-based indicators can be selected from a list within the Excel tool:

8Establish

baselines for impact based indicators and performance

based indicators



when possible)

Identify causative


factors



sole responsibility




and impacts

1 2 3 4 5

7 6


based indicators

8


10

Workshop





practices)






Evaluator on a tour




9

Monitor

Define impact-based indicators baselineDefine impact-based indicators baseline

Before establishing the baseline, the evaluator must enter the information into the indicator tool relating to how measurements will be taken and what units will be used (This information should be filled out for each indicator) When this information has been entered, a baseline will be established. This stage should be carried out in conjunction with the operator and the evaluator whilst the trip is underway.

A baseline is the initial measurement that the indicators will be compared with on a long-term basis. For example, if the indicator is the percentage of gift shop stock which corresponds to locally produced artisan goods, the baseline could be 30% of locally produced artisan goods in the gift shop at the start of the monitoring process.

The baseline should be defined at the start of each operation. The baseline indicates the state of the operation before embarking on the implementation of activities or good practices






3

Number of registered incidents between communities and operator due to visitors’ behavior Record sheet managed by operator # 5 3 2 0 0 0

4Number of registered primate observations Record sheet managed by guides # 9 3 14 10 14 21

5

Number of oil spills and leakages produced in water bodies by the operation’s outboard motors Record sheet managed by boat operators # 12 11 7 7 4 1


Define performance-based indicators baselineDefine performance-based indicators baseline

The performance-based indicators baseline is established by the evaluator.Once a performance-based indicator is defined, the baseline must be set by the evaluator during a tour. For performance-based indicators, there are only 3 possible measurements: compliance, compliance partially and does not apply. The rationale is the argument by which the evaluator assigns the baseline to the indicator.




No. Compliance Non - compliance Does not apply

1x

2x

Indicators (defined by evaluator) Rationale

Baseline




The operator does not have a code of conduct to inform visitors about behaviour norms

The operator does not established any site besides its own where visitors can purchase any service or goods to incentivate

local economy

9

Monitor



when possible)

Identify causative


factors



sole responsibility




and impacts

1 2 3 4 5

7 6


based indicators

8


10

Workshop





practices)






Evaluator on a tour




9

Monitor

Monitor impact-based indicatorsMonitor impact-based indicators

The operator is completely in charge of monitoring. Once the baseline is defined, the operator should input the monitoring information into the MS Excel tool.

The tool allows the operator to enter the recollected information on a monthly basis. The information obtained should be in the same units of measurement as the baseline in order to compare the readings over time. The monitoring information (compiled and entered into the tool by the operator) should be entered as a monthly value (the same as with the rest of the information for each indicator).






3

Number of registered incidents between communities and operator due to visitors’ behavior Record sheet managed by operator # 5 3 2 0 0 0

4Number of registered primate observations Record sheet managed by guides # 9 3 14 10 14 21

5

Number of oil spills and leakages produced in water bodies by the operation’s outboard motors Record sheet managed by boat operators # 12 11 7 7 4 1


Monitor performance-based indicatorsMonitor performance-based indicators

As the baselines, the monitoring for performance based indicators is done by the evaluator.Instead of having a monthly measurement, these indicators are only corroborated on field two more times besides the baseline. The first evaluation is done three months after from baseline and the second six months later. The ulterior evaluations (besides baseline) are done just by the confirmation of any changes on the initial operator’s condition (baseline). Any changes on the initial condition must be registered and rationalized.




No. ComplianceNon -

compliance Does not apply ComplianceNon -

compliance Does not apply

1x x

2x x

Rationale

The operator has established an arragement with 2 local handicrafts producers to offer visitors a site to purchase handicrafts besides the operator's shop.

2nd evaluation

Rationale

1st evaluation

The operator has adopted a code of conduct that includes visitor

behaviour norms

Same as baseline




10Evaluate and

readapt



when possible)

Identify causative


factors



sole responsibility




and impacts

1 2 3 4 5

7 6


based indicators

8


10

Workshop





practices)






Evaluator on a tour




9

Monitor

Evaluate and readaptEvaluate and readapt

EvaluateOn finalizing the monitoring process, results are evaluated on comparing the final results with the baseline, and conclusions are made. The key question is whether or not the measurements have increased or decreased on finalizing the process.

ReadaptThis step is fundamental in adopting adaptive management. Adaptive management deals fundamentally with evaluating and readapting. Readapting refers to the repetition of a process in a sequence of steps which successively lead more closely to a desired result.

Start of plan






make conclusions



Start of plan






make conclusions



Start of plan






make conclusions



Thank YouThank You

monitoring and evaluation protocol marine and rainforest operations

Documents

type of monitoring

sustainable tourism

tourism operationsperformance

good tourism practices

walking tours

operator delfin tours

operator danta tours

marine operations