As with any evaluation, a number of key factors determine which approach would best suit a program such as the time, personnel and funding available. Assuming that these are substantial, my pick here would be to utilize a combination of an outcome-based or goal- oriented summative approach as well as a process-based formative evaluation. Using both these approaches would provide the most comprehensive and, hence, the most beneficial evaluation.
Firstly, a Michael Scriven’s outcome-based approach is appropriate in order to evaluate the effectiveness of the program in addressing the children’s needs as determined by their Individual Program Plans (IPP). Once clear parameters defining these needs are provided, the evaluation can then focus on determining which of them were effectively fulfilled by the ECS program, as well as to identify any shortcomings. Indicators of the program’s effectiveness could include the children’s improvement in skill, behaviours, independence, quality of life, and other areas identified in the IPP. In order to assess the level of change, a pre-test and post-test would be required.
The formative process-based approach would compliment this initial evaluation by examining the services rendered to the participants of the ECS program. As both the center-based programs and in-home programs form an integral link to the overall effectiveness of the program, evaluating them closely would certainly be worthwhile.
Focus areas of this approach could include examining how these services are provided in terms of the average number and duration of home visits or centre-based participation, the personnel involved during each session, how the participants’ levels of functioning are assessed, the process involved in determining the children’s needs for the IPP, the level of stress (if any) exhibited by the participants during the transportation to and from the centres, the level of involvement or interaction required from the children during different activities, the methods used and the reaction towards them as apparent in the actions of the participant, etc. Data collection in both approaches would involve a battery of questionnaires, observations as well as interviews with all involved participants including parents, care providers and specialists.
Once all information is gathered, the evaluators could then look for correlations between process and outcomes and hence be better equiped to draw conclusions regarding the overall effectiveness of the program. Accordingly, the combined approach evaluation has the potential to provide a very comprehensive and sound base for decision-making regarding the future direction of the program.
Saturday, September 19, 2009
Saturday, September 12, 2009
Assignment 1 - Evaluation Critique
An Evaluation of an Experiential Learning and Outdoor Education School Program on the Life Effectiveness Skills of Middle School Boys:
A Deconstructive Analysis
By Zofia Gehl
September 12th, 2009
A complete copy of the evaluation can be found at
http://www.latrobe.edu.au/oent/OE_conference_2004/papers/mcLeod.pdf
Introduction
An analysis of an evaluation of an Experiential Learning and Outdoor Education school program on the Life Effectiveness skills of middle school boys reveals a number of strengths and limitations. The purpose of this paper is to deconstruct this evaluation in order to reveal these factors in respect to the evaluation model and approach followed, the methods used for collecting and analyzing data, as well as the process of reporting the findings and drawing conclusions.
The document used in this analysis was written by Beth McLeod and Sandy Allen Craig and was entitled, “An evaluation of an Experiential Learning and Outdoor Education school program on the Life Effectiveness skills of middle school boys.” It was published in May, 2009. Both authors have extensive background in studying and teaching Outdoor Education and Physical Education. Craig is a lecturer in Outdoor Education for the school of Exercise Science at the Australian Catholic University, while McLeod has recently completed Honors in this program and is currently working on her Graduate Diploma of Education and Physical Education.
It is noteworthy to note however, that despite their knowledge and familiarity with outdoor education, their credentials presented at the end of the evaluation document, show neither specific qualifications nor specialized training in the field of evaluation.
Background and Purpose of Evaluation
The evaluation in question was conducted in order to assess the effectiveness of the Experiential Learning and Outdoor Education school program and to determine whether the program was achieving its outcomes. The program’s goals were to “enable (middle school aged) boys to develop life effectiveness skills in the areas of time management, social competence, achievement motivation, intellectual flexibility, task leadership, emotional control, active initiative and self confidence” (McLeod & Craig, 2009, p. 1). The program is targeted at boys in Grade 9 in a private boys school in Melbourne, Australia.
In the introduction section of the evaluation document, the authors suggest that there is a dire need for a program of this nature due to the difficult psychological, emotional and social challenges faced by boys in this age group. This program was developed in an attempt to help boys cope with these age-specific issues.
The participants of this evaluation were comprised of a sample of 104 grade 9 boys out of a total of 169 who attended the school. Unfortunately the process of selecting the participants was not described in the report.
A further weakness of this evaluation is that it fails to identify the participating boys’ backgrounds, social economic statuses, parental support, as well as program preconceptions and attitudes. Without this information, it is difficult to understand all of the key factors which may have acted to influence the findings of this evaluation. Consequently the correlations noted by the evaluators may be caused by extraneous factors that were not accounted for. It is valuable then to be reminded that correlation does not necessarily equal causation!
Evaluation Approach and Methodology
This evaluation follows Scriven’s model of outcome-based summative evaluation as it considers the effectiveness of the program as measured by the achievement of outcomes. In other words, this evaluation aims to find out whether this program actually does what it intended to do.
The results were formulated using two separate questionnaires: the Life Effectiveness Questionnaire (LEQ) and a Social Validation Questionnaire (SVQ). The LEQ provided the evaluators with quantitative data measuring change, while the SVQ provided a more comprehensive insight into participant responses as it contained open-ended questions. Participant SVQ answers were used to enrich the data collected via the LEQ. Implementing measurement tools which provide both quantitative and qualitative data is a definite strength of this evaluation.
One limitation of the methodology however, has to do with the level of evaluation. While the goal of this evaluation was to investigate the success of the program in regards to the development and improvement of “life effectiveness skills,” the data used in the assessment dealt exclusively with reactions and feelings. A criticism of this approach is that feelings are considered to be poor indicators of the program’s level of impact over an extended period of time. Perhaps a test where the participants have the opportunity to demonstrate actual learned skills rather than just reporting their feelings and perceptions would have been a better measure of the effectiveness of this program. Furthermore, the measure of outcome utilized here only includes the initial stage as it looks exclusively at attitudes and feelings rather than striving for the intermediate and final measures by analyzing behaviour and long-term outcomes.
Additional Strengths and Limitations
This program evaluation contains a number of factors which make it a valuable document. Namely, the introduction section and purpose behind conducting the evaluation are very comprehensive and consequently provide the reader with a good overview of the issues involved in outdoor education as well as its need. Furthermore, this study is based on fairly current research as evident in the citations and references throughout this document. As sound theoretical foundation is essential in avoiding evaluation pitfalls, the recency of research definitely plays a big role. Combining quantitative data with qualitative data also serve as an asset in this evaluation. This blended methodology serves to make the results more credible and comprehensive. Finally, the simple language with which this evaluation was written can also be deemed an asset as it makes this document accessible to a wide audience including not only academics and scholars but also the students involved as well as their families and other interested parties with little background in scholarly writing.
As with most evaluations, it is difficult to completely eliminate all limitations and pitfalls. This document’s limitations are most obvious in its omission to address a number of key factors. Information regarding the following questions is missing entirely from the report:
- where the evaluation funding is coming from
- who asked for the evaluation to be completed
- what is the evaluator-stakeholder relationship (resulting potential biases)
- what stage is the program development in/how old is the program
- how were participants selected for the sample
- what are the students’ backgrounds, level of parental support, motivations for completing this program and social economic status
- what extraneous factors may have affected the results
In addition to the above stated factors, the report also mentions that not being able to use a control group due to the nature of this program is a further limitation of this evaluation.
Finally, when discussing an evaluation’s limitations it is also important to consider the party or parties who conducted the evaluating and the impacts this could have made on the results of the study. In this case, the evaluators, McLeod and Craig are passionate proponents of outdoor education as reflected in their professional and educational backgrounds. Avoiding a biased perspective in their desire for this program to be deemed a success would indeed be a difficult task. A vested interest of this nature is definitely a factor that would have to be considered when considering the results of this study as well as their presentation. An evaluation conducted by educational traditionalists or opponents of outdoor education would undoubtedly look a little differently.
September 12th, 2009
A complete copy of the evaluation can be found at
http://www.latrobe.edu.au/oent/OE_conference_2004/papers/mcLeod.pdf
Introduction
An analysis of an evaluation of an Experiential Learning and Outdoor Education school program on the Life Effectiveness skills of middle school boys reveals a number of strengths and limitations. The purpose of this paper is to deconstruct this evaluation in order to reveal these factors in respect to the evaluation model and approach followed, the methods used for collecting and analyzing data, as well as the process of reporting the findings and drawing conclusions.
The document used in this analysis was written by Beth McLeod and Sandy Allen Craig and was entitled, “An evaluation of an Experiential Learning and Outdoor Education school program on the Life Effectiveness skills of middle school boys.” It was published in May, 2009. Both authors have extensive background in studying and teaching Outdoor Education and Physical Education. Craig is a lecturer in Outdoor Education for the school of Exercise Science at the Australian Catholic University, while McLeod has recently completed Honors in this program and is currently working on her Graduate Diploma of Education and Physical Education.
It is noteworthy to note however, that despite their knowledge and familiarity with outdoor education, their credentials presented at the end of the evaluation document, show neither specific qualifications nor specialized training in the field of evaluation.
Background and Purpose of Evaluation
The evaluation in question was conducted in order to assess the effectiveness of the Experiential Learning and Outdoor Education school program and to determine whether the program was achieving its outcomes. The program’s goals were to “enable (middle school aged) boys to develop life effectiveness skills in the areas of time management, social competence, achievement motivation, intellectual flexibility, task leadership, emotional control, active initiative and self confidence” (McLeod & Craig, 2009, p. 1). The program is targeted at boys in Grade 9 in a private boys school in Melbourne, Australia.
In the introduction section of the evaluation document, the authors suggest that there is a dire need for a program of this nature due to the difficult psychological, emotional and social challenges faced by boys in this age group. This program was developed in an attempt to help boys cope with these age-specific issues.
The participants of this evaluation were comprised of a sample of 104 grade 9 boys out of a total of 169 who attended the school. Unfortunately the process of selecting the participants was not described in the report.
A further weakness of this evaluation is that it fails to identify the participating boys’ backgrounds, social economic statuses, parental support, as well as program preconceptions and attitudes. Without this information, it is difficult to understand all of the key factors which may have acted to influence the findings of this evaluation. Consequently the correlations noted by the evaluators may be caused by extraneous factors that were not accounted for. It is valuable then to be reminded that correlation does not necessarily equal causation!
Evaluation Approach and Methodology
This evaluation follows Scriven’s model of outcome-based summative evaluation as it considers the effectiveness of the program as measured by the achievement of outcomes. In other words, this evaluation aims to find out whether this program actually does what it intended to do.
The results were formulated using two separate questionnaires: the Life Effectiveness Questionnaire (LEQ) and a Social Validation Questionnaire (SVQ). The LEQ provided the evaluators with quantitative data measuring change, while the SVQ provided a more comprehensive insight into participant responses as it contained open-ended questions. Participant SVQ answers were used to enrich the data collected via the LEQ. Implementing measurement tools which provide both quantitative and qualitative data is a definite strength of this evaluation.
One limitation of the methodology however, has to do with the level of evaluation. While the goal of this evaluation was to investigate the success of the program in regards to the development and improvement of “life effectiveness skills,” the data used in the assessment dealt exclusively with reactions and feelings. A criticism of this approach is that feelings are considered to be poor indicators of the program’s level of impact over an extended period of time. Perhaps a test where the participants have the opportunity to demonstrate actual learned skills rather than just reporting their feelings and perceptions would have been a better measure of the effectiveness of this program. Furthermore, the measure of outcome utilized here only includes the initial stage as it looks exclusively at attitudes and feelings rather than striving for the intermediate and final measures by analyzing behaviour and long-term outcomes.
Additional Strengths and Limitations
This program evaluation contains a number of factors which make it a valuable document. Namely, the introduction section and purpose behind conducting the evaluation are very comprehensive and consequently provide the reader with a good overview of the issues involved in outdoor education as well as its need. Furthermore, this study is based on fairly current research as evident in the citations and references throughout this document. As sound theoretical foundation is essential in avoiding evaluation pitfalls, the recency of research definitely plays a big role. Combining quantitative data with qualitative data also serve as an asset in this evaluation. This blended methodology serves to make the results more credible and comprehensive. Finally, the simple language with which this evaluation was written can also be deemed an asset as it makes this document accessible to a wide audience including not only academics and scholars but also the students involved as well as their families and other interested parties with little background in scholarly writing.
As with most evaluations, it is difficult to completely eliminate all limitations and pitfalls. This document’s limitations are most obvious in its omission to address a number of key factors. Information regarding the following questions is missing entirely from the report:
- where the evaluation funding is coming from
- who asked for the evaluation to be completed
- what is the evaluator-stakeholder relationship (resulting potential biases)
- what stage is the program development in/how old is the program
- how were participants selected for the sample
- what are the students’ backgrounds, level of parental support, motivations for completing this program and social economic status
- what extraneous factors may have affected the results
In addition to the above stated factors, the report also mentions that not being able to use a control group due to the nature of this program is a further limitation of this evaluation.
Finally, when discussing an evaluation’s limitations it is also important to consider the party or parties who conducted the evaluating and the impacts this could have made on the results of the study. In this case, the evaluators, McLeod and Craig are passionate proponents of outdoor education as reflected in their professional and educational backgrounds. Avoiding a biased perspective in their desire for this program to be deemed a success would indeed be a difficult task. A vested interest of this nature is definitely a factor that would have to be considered when considering the results of this study as well as their presentation. An evaluation conducted by educational traditionalists or opponents of outdoor education would undoubtedly look a little differently.
Subscribe to:
Posts (Atom)