Evaluation Plan for EcoQuest

Proposed by Zofia Gehl

University of Saskatchewan
December 14, 2009

Introduction and Background

Outdoor education programs are becoming increasingly popular these days and many learners
are choosing this method as a welcome alternative (Hattie et al., 1997). With this growing
popularity, educators have an increasing responsibility to ensure that this means of delivering
academic content is in fact effective and that it is closely monitored for opportunities to make
improvements. This particular evaluation will focus on examining the effectiveness of this type
of an approach in a program called EcoQuest which was first launched in September of 2004.
Upon approval of this plan, the evaluation will be conducted by Zofia Gehl, a second year
Graduate Studies student from the University of Saskatchewan currently enrolled in Curriculum
Studies.

EcoQuest is an outdoor educational adventure program for grade 8 students which is based on
three main themes: A Sense of Place, A Sense of Time and A Sense of Quality. These themes
interact collectively to provide a mechanism by which all of the required academic subjects are
integrated seamlessly in relevant and appropriate settings. The learning transpires via
experiential and inquiry-based projects and instructional field studies ranging in scope from day
trips in and around Saskatoon to major extended expeditions. Time in the classroom is also
incorporated to reflect on and to analyze the out-of-school learning experiences.

This program is taught by two teachers and is a part of the Saskatoon Public Schools Division.
Each year, 28 new Grade 8 students with equal ratios of males and females form the make up of
this class. Although EcoQuest has a “home base” classroom at Buena Vista School in Saskatoon,
most of the learning takes place outside in carefully selected settings all around the province of
Saskatchewan.

The main goals of EcoQuest are to help students develop intellectually, socially, physically and
emotionally while strengthening their connection and appreciation of the natural world. It is
student growth in these areas that will be the focus of this evaluation. Because of the small size
of the class, all willing students will form the sample in this study, as well as their parents and
Ecoquest teachers.

Purpose
The purpose of this summative evaluation is to examine the effectiveness of the EcoQuest
program in meeting its primary goals of helping students develop intellectually, socially,
physically and emotionally while strengthening their connection and appreciation of the natural
world. For the purpose of this evaluation, each of the above areas is defined using specific
program objectives.
Table 1: Definition of Focus Areas of Growth in Terms of Program Objectives
Focus Areas Program Objectives
of Growth
Intellectual Growth in positive communication and observational skills.

Improved research, writing, and presentation skills.

Skills that underlie academic excellence and achievement in all subject areas.
Social A strong sense of community with fellow classmates, teachers, and community
partners.

Leadership, risk management, and cooperative group skills that are necessary in
adventure education.

A greater awareness of Social Justice issues.

Physical A dedication to physical fitness and healthy lifestyles.

Emotional A heightened sense of accomplishment, confidence, and self worth.

Connection An ability to think critically about ecological, cultural, and community issues.
and
Appreciation Greater knowledge and appreciation of the fragility of Saskatchewan's natural
of Nature environment and ecological regions.

Since student growth is at the heart of this evaluation, it is essential that data be collected before
and after the completion of the program. A subsequent analysis of the difference between
participants’ pre and post assessments will provide a detailed depiction of the students’ overall
growth in each area.

This evaluation is essential not only to shed light on the true potential of EcoQuest but also to
serve as a catalyst for implementing any future changes and making informed decisions. This is
particularly significant as the program is still relatively new (5 years) and can still be considered
to be in its developmental stage. Furthermore, ongoing reflection and evaluation is part of any
sound educational program and should be carried out routinely.

The benefits of this evaluation, however, do not end with EcoQuest. Rather their scope may
reach many external institutions who share an interest in outdoor experiential education as they
consider the results of this evaluation in developing their own similar outdoor programs.
Audiences
The primary audience for this evaluation will be the two EcoQuest teachers. Being the main
driving forces behind the program, as well as working directly with the students, they are in an
ideal position to translate any recommendations of change into action.

Secondly, as the supervising party, the in-school administrators must be very familiar with the
process and outcomes of this evaluation in order to continue providing appropriate administrative
support. In addition, favourable results could potentially be used to drive student recruitment and
program expansion.

Thirdly, the Board of Directors of the Saskatoon Public Schools Division (SPSD) will also be
interested in gaining insight into the effectiveness of this program. As all new programs need to
gain approval at this level, the outcomes of this evaluation will help guide the Board in future
endeavours associated with similar programming. The results of this study will also potentially
impact future budgeting and sponsorship decisions.

Finally, SPSD will be able to decide whether or not to share this evaluation with the general
public. Interested parties would undoubtedly include past and potential future students and their
parents, external institutions and scholars interested in outdoor education, as well as current and
future educators around the world.

Questions and Decisions

The research question examined in this evaluation will be:
How effective is the EcoQuest program in achieving its primary goals? Specifically, does the
program stimulate growth in students’ intellectual, social, emotional, physical development, as
well as growth in the connectedness and appreciation of nature?
In finding the answer, data will be collected prior to the beginning and after the completion of
the program.

The decisions potentially influenced by the results of this program are summarized in Table 2
below.

Table 2: Potential Future Decisions

program change As strengths and weaknesses of the program are identified in
implementation each specific area of student development, future programming
can be improved.

administration, budgeting The source and allocation of funding can be changed. Ie. if the
and sponsorships results of this evaluation show favourable results of the
program’s effectiveness, sponsors may be more willing to offer
greater financial support and the school division may decide to
increase its funding, both in terms of monetary and supply-
based support.

student recruitment and A high effectiveness of the EcoQuest program could potentially
 program expansion spur higher student demand, eventually leading to program
expansion both internally and externally of the SPSD.

educator training As the success of the outdoor experiential teaching methods

become proven and more widely recognized, this approach to
teaching may become a greater focus in teacher training and
preparation.

scholarly research With outdoor education already being a popular area of study,
academics will be able to benefit from the results of this
program evaluation as the knowledge bank is expanded and
more research is driven.

Sample
This evaluation will require the participation of all willing EcoQuest students and their parents as
the class size is only 28; clearly, the more participants, the more reliable the results. In addition
to the students and their parents, the two EcoQuest teachers will also be asked to provide
feedback. An insight into their perspective will enhance the overall understanding of student
growth in each of the focus areas.

Method
To test the effectiveness of the EcoQuest program, a number of data collection instruments will
be used. The participants will be divided into three groups: students, parents, and teachers. All
participating students will be required to provide a written consent from their parents.

The first phase of testing will take place within a few days of the commencement of the program.
During this time, the students will complete the Life Effectiveness Questionnaire, Connectedness
to Nature Scale, and the Health and Fitness Survey. They will also be asked to self-assess their
skills in the five focus area using an assessment rubric.

Parents will be asked to complete the Life Effectiveness Questionnaire and the assessment rubric
based on their observations and knowledge of their child.

The teachers will not be asked to provide data at this point as they will not yet be sufficiently
familiar with their students.

The final phase of data collection will consist of similar instruments with a few key differences.
In addition to the questionnaires and assessments mentioned above, all three participant groups
will be asked to complete an additional reflection questionnaire summarizing the student growth.
Also, the teachers will share a completed assessment rubric for each of the participating students.

Once all the information is collected, the pre-program and post-program data will be carefully
compared in order to identify any significant differences for each student. Subsequently, this
information will be analyzed in terms of percentages of students showing significant growth as
compared to the total number of students. This procedure will be repeated for each of the five
focus areas.

A particular strength in the methodology of this evaluation is the repetition factor. Having more
than one participant group complete some of the same questionnaires will lead to increased
reliability and validity of the results. Furthermore, involving a variety of instruments to collect
data, allows for a more comprehensive analysis of the overall effectiveness of the program.

Instrumentation

Table 3: Initial (Pre-program) Data Collection

Instrument Participant Group Approximate Expected Duration
Life Effectiveness -Students and Parents 20 to 30 minutes
Questionnaire

Connectedness to Nature -Students 5 to 10 minutes

Scale

Physical Fitness and Wellness -Students 5 to 10 minutes

Questionnaire

Assessment Rubric -Students with the help 30 to 45 minutes

of their parents

Table 4: Final (Post-program) Data Collection

Instrument Participant Group Approximate Expected Duration
Life Effectiveness -Students, Parents 20 to 30 minutes
Questionnaire

Connectedness to Nature -Students 5 to 10 minutes

Scale

Physical Fitness and Wellness -Students 5 to 10 minutes

Questionnaire

Assessment Rubric -Students, Parents 30 to 40 minutes

Overall Growth Questionnaire -Students, Parents 20 to 30 minutes

Most of the data collection instruments suggested for this study were selected based on their
proven validity and reliability. Please refer to the appendices at the end of this paper for
examples of the data collection instruments.
Life Effectiveness Questionnaire (LEQ)
The Life Effectiveness Questionnaire was designed by Neill, J. T., Marsh, H. W., & Richards, G.
E. (2003) to measure a person's capacity to adapt, survive and thrive as defined by the following
seven scales:
time management, social competence, achievement motivation, intellectual flexibility, task
leadership, emotional control, active initiative and self confidence.
Refer to Appendix B for the descriptions of each of the scales.
For the purposes of this program evaluation, the LEQ will be used to assess student progress in
regards to intellectual, social, and emotional growth.

The reliability and validity of the Life Effectiveness Questionnaire has been confirmed and
supported by its frequent appearance in research. To date, the LEQ has been used in over 20
studies related to life skill education programs and outdoor education (Neill et al., 2003).
“It is an excellent tool for monitoring program effectiveness and is an
efficient way to develop a long-term database documenting program
quality. When analyzed and interpreted, the results can be informative for
program coordinators and instructors to help improve program quality.”
(http://wilderdom.com/tools/leq/WhatIsTheLEQ.html)

A reference list of studies which have used the LEQ can be found at
http://wilderdom.com/tools/leq/leqreferences.html.

Connectedness to Nature Scale (CNS)

The Connectedness to Nature Scale which was developed by F.Stephan Mayer, Cynthia
McPherson Frantz (2004) has also been proven to be effective. The validity and reliability were
tested with two community and three college samples and the data suggested that the CNS has
“good psychometric properties, correlates with related variables (the new environmental
paradigm scale, identity as an environmentalist), and is uncorrelated with potential confounds
(verbal ability, social desirability)” (Mayer & Frantz, 2004, p. 503). An analysis by Perrina &
Benassi (2009) also suggested that the scale is effective in measuring cognitive beliefs about
nature.

Furthermore, being comprised of a total of only 14 items, its length is especially appropriate for
young adolescents who may otherwise grow tired if subjected to longer testing.

Assessment Rubric
The assessment rubric was developed by EcoQuest teachers, Scott Thompson and Shelly Loefler
as a final assessment tool to describe student progress in all of the program’s objectives. Each
year, small changes were made in order to perfect its validity. One particular strength of this
rubric is that it utilizes vocabulary which is used and reinforced throughout the program, making
this assessment especially user-friendly and easily comprehensible for the students. The initial
completion of this rubric will be done by the children with the help of their parents since they
may not yet be familiar with some of the terminology.
Health and Fitness Survey & Overall Growth Reflection
The health and fitness survey, as well as the overall growth reflection are newly developed
measurement tools and will need to be piloted by a group of recent EcoQuest graduates in order
to ensure that they are reliable, intelligible and valid. Ideally, the sample size should consist of at
least ten participants, both male and female.

Based on this preliminary testing, amendments will be made to improve the overall quality of
these tools prior to conducting the official program evaluation.

Limitations
As with any research study, this evaluation has its limitations. Being designed specifically for the
EcoQuest program, the application of careful changes may be needed to expand its applicability
to other educational programs. When extrapolating the results of this evaluation to other
programs, one needs to take into account all aspects of this particular program including student
background and demographics, sample size, location, program content and content delivery. All
of these factors play a role in influencing the results of this evaluation and their significance must
be carefully weighed.

Furthermore, proving causality will be a particular challenge in this case, as the methodology
does not include a control group. It is important, therefore, to remember that correlation does not
equal causation. For instance, even though correlation is likely to be established between
program completion and student development, the growth may in fact be caused by extraneous
variables rather than the program itself. Repeating this evaluation on an annual basis will help
increase its validity.

Another inherent limitation to this study is the small sample size. With only 28 eligible students
in the class, some may question the generalizability factor of this evaluation. Again, performing
this evaluation on a yearly basis will generate a more reliable pool of information as the overall
sample size increases.

Logistics and Time Line

Approval and Leadership
This program evaluation is dependent on the approval of this proposal by the principal of Buena
Vista Public school and will not take place with out it. Once the approval is granted, the
evaluation will be headed by Zofia Gehl and all final decision making will need to include her
before implementation. Ms. Gehl will take every measure possible to ensure that positive and
comprehensive communication occurs at every level between the administrative staff, the
teachers and the evaluators.

It is essential that all participating students submit a signed parental permission form prior to any
participation. The forms and introductory letters containing specific information about this
evaluative study will be mailed out by EcoQuest teachers along with their welcome letters in the
summer months preceding the commencement of the program. Only those students who receive
written permission from their parents will be allowed to participate.
Data collection will begin one week before the commencement of the program in September.
This will allow time to gather data from any absent students who may be ill or away for other
reasons. It will also provide additional time for any unexpected delays. The initial round of
testing is expected to take no more than one and a half hours. The exact time of commencement
will be promulgated at a later date and will be shared with the participants in their welcome
letters.
All testing, with the exception of the assessment rubric will take place within the EcoQuest
classroom where Zofia Gehl and the EcoQuest teachers will have the ability to provide
instructions and answer any questions regarding the data collection instruments. Although
instructions will be given regarding the assessment rubric, students will be invited to complete
them at home with their parents and return them to the teachers on the first day of the program.
The logic behind this is to allow the students and parents to engage in discussion as they address
each rubric item without distracting or influencing other participants in close proximity.

To optimize confidentiality, participants will not be asked to include their names on any of the
assessment tools. Also, a mailbox for completed forms will be available for submissions.

The analysis of all data will be the responsibility of the evaluator, Zofia Gehl who will put
together a written report within three months of the completion of data gathering. The report will
be shared with the two Ecoquest teachers and the principal of Buena Vista Public School during
a private meeting. The decision whether or not to further publicize or share the results will be
made by the principal in collaboration with the EcoQuest teachers.

Budget
In developing the budget for this program evaluation the following considerations have been
taken into account.

- Funding source
- Budget contact
- Funding period
- Condition of payment
- Budget limit
- Personnel Costs
- Travel
- Supplies, Materials and Equipment
- Communications
- Copying and Printing

Refer to Table 4 for specific explanations for each of these considerations.

Table 5: Explanation summary for budget considerations

Consideration Explanation

Funding source & As this study has the most direct impact on the quality of the EcoQuest
Budget contact program, the funding should come from its host school, Buena Vista.
As such, the approval authority and budget contact would be the
principal.

Funding Period The funds must be available no later than May 1st of the evaluation year
in order to provide sufficient time for the generation of all necessary
materials, until the completion of the study as marked by the sharing of
results with the principal and EcoQuest teachers.

Condition of Payment will be available only once this program evaluation proposal is
Payment approved and accepted by the principal and the EcoQuest teachers.

Budget Limit The budget limit is not to exceed 5 % of the proposed total funding that is
anticipated.

Personnel Costs This evaluation will not be subject to any personnel costs as both the
evaluator and the Ecoquest teachers will conduct this study pro bono in
the name of research and for the benefit of all future EcoQuest teachers.

Travel Costs Travel costs are anticipated to be minimal as all parties involved in this
evaluation live within the city limits of Saskatoon where the study will be
conducted.

Supplies, Materials Funding will be required for the generation of all data collection
and Equipment instruments including forms, pens and pencils as well as publishing costs
of the final report. A computer memory device for storing all study data
will also be required. Additionally, a small cost will be incurred in order
to provide refreshments during the pre- and post-program data collection
gatherings.

Communication Communication costs, like travel, will not be significant due to the local
Costs nature of this study. Some long-distance calling charges may occur if
outside agencies or subject-area experts need to be consulted.

Copying and Copying and printing charges are anticipated to make up the bulk of the
Printing anticipated expenses for this study. The exact amounts will be dependent
on the number of participants available for this evaluation.

Based on the above considerations, it is estimated that the funding needed for this evaluation to
occur is approximately three hundred dollars. A more exact calculation will be submitted to the
principal of Buena Vista upon confirmation of the total number of participants taking part in this
study.
References
EcoQuest. (n.d.). Retrieved December 8, 2009, from http://ecoquest.ca/index.php

Hattie, J. A., Marsh, H. W., Neill, J. T., & Richards, G. E. (1997). Adventure education and
Outward Bound: Out-of-class experiences that make a lasting difference. Review of
Educational Research, 67, 43-87.

L.E.Q. - H©. (n.d.). Retrieved November 28, 2009 from

http://wilderdom.com/tools/leq/LEQuestionnaireversionH24items8scales.doc

Mayer, F.S & McPherson Frantz, C. (2004). The connectedness to nature scale: A measure of
individuals’ feeling in community with nature. Journal of Environmental Psychology, 24,
503–515.

Neill, J. T., Marsh, H. W., & Richards, G. E. (2003). The Life Effectiveness Questionnaire:
Development and psychometrics. Unpublished manuscript, University of Western
Sydney, Sydney, NSW, Australia.

Perrin, J.L. & Benassi, V.A. (2009). The connectedness to nature scale: A measure of emotional
connection to nature? Journal of Environmental Psychology, 29 (4), 434-440.