NASA's Elementary and Secondary Education Program Review and Critique (2008) / Chapter Skim
Currently Skimming:
5 Program Evaluation
5 Program Evaluation
Pages 90-111
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 90... ...
subjects; increasing students' knowledge of STEM content; and encouraging young people, especially those from groups that are underrepresented in STEM fields, to become familiar with and pursue STEM careers. Evaluation of NASA's K-12 education program and its related projects is challenging and requires significant resources and expertise in evaluation.
|
From page 91... ...
Following an initial discussion of evaluation issues with some reference to NASA, the chapter is organized by the major components involved in evaluating programs, from design to evaluation of impact. The chapter draws in part on a paper the committee commissioned by Frances Lawrenz to review a set of ten external evaluations of NASA's K-12 projects, including the Aerospace Education Services Project (AESP)
|
From page 92... ...
Explorer Framework: Schools Evaluating the Quality and Impact of the NASA Explorer Schools Program (McGee, Hernandez, and Kirby, 2003) NASA Brief 2 -- A • What is the profile of schools designated as NASA Explorer Schools?
|
From page 93... ...
interactions. NASA Evaluation Plan • Overall Question: What is the relationship of the nature and extent Data not yet available; proposed Explorer 2006-2007 (Paragon of a school's involvement to their success in developing teachers' NEEIS data and other surveys Schools TEC, 2006)
|
From page 94... ...
Aerospace Evaluation of the There are 19 evaluation questions addressing the following 5 areas: Delphi survey, surveys of Education NASA Aerospace • program design and management. specialists and telephone Services Education Services • support of systemic improvement.
|
From page 95... ...
• What are some exemplary cases in which AESP specialists' work has impact? NOTES: STEM-G = Science, technology, engineering, mathematics, and geography; NEEIS = NASA Education Evaluation Information System.
|
From page 96... ...
Currently, the NASA Office of Education lacks an overall evaluation plan for the K-12 education program and its projects. Given resource constraints, evaluations of individual projects can be scheduled on a cyclical basis, with high priority given to projects intended to have the greatest impact on student engagement and learning and to projects that face important questions about activities, participants, staffing, funding, or organization.
|
From page 97... ...
Currently, the overall Elementary and Secondary Program is periodically reviewed, but it has not undergone a true external evaluation. Moreover, the timing of external evaluations of individual projects appears to have been determined by individual project officers with little strategic coordination across the program.
|
From page 98... ...
This framework and model, however, have very little specificity. More detail about mechanisms and expected effects based on research is needed for individual projects.
|
From page 99... ...
. These specifications are important for guiding both internal and external evaluations of the overall program.
|
From page 100... ...
training opportunities to educators that training elementary and secondary educators 2.2.4 Percentage of NASA teacher program result in deeper content understanding and/or who partner with NASA in their STEM participants who become active within a competence and confidence in teaching teacher educator programs. national network to train other teachers.
|
From page 101... ...
relevance of NASA educational resources. support resources that use NASA themes 2.3.2 Quantity, type, and cost of educational 2.3.5 Customer satisfaction data regarding and content to (a)
|
From page 102... ...
The outcome measures are the number of teachers who use NASA content or resources as a result of another teacher's direct involvement with a NASA program; the percentage of participants who become active in a national network to train other teachers; the percentage of participants who use NASA resources in their classroom instruction; and evidence that teachers who use NASA resources perceive themselves as more effective teachers of STEM subjects. For this objective, there is a mismatch between three of the outcome measures and the objective.
|
From page 103... ...
In engaging students, the intent is to inform students about STEM career opportunities and communicate information about NASA's mission activities. The output measures include quantity, type, and costs of materials produced and approved through the NASA review process and the percentage of materials that are accessible electronically.
|
From page 104... ...
It notes that the goals for most NASA projects are very broad and that it would be difficult for any project, much less one with limited funding available, to achieve these goals in any depth. It suggests that these issues might be resolved during the evaluation planning stages with careful discussions that would include development of targeted goals for projects that would be more amenable to evaluation FORMATIVE EVALUATION The purpose of formative evaluation is to provide feedback on the development of a program or project and its implementation.
|
From page 105... ...
. Lawrenz's review of NASA's external evaluations of projects suggests that the headquarters Office of Education is doing an adequate job of formative evaluation.
|
From page 106... ...
Currently, some evaluators and the organizations that fund them advocate randomized clinical trials as the preferred evaluation design (sometimes called the "gold standard" of evaluation)
|
From page 107... ...
. There are major difficulties to conducting a randomized clinical trial in order to determine a program's impact, especially for the types of programs and projects that NASA supports ������������������������������������ (see Rossi et al., 2003, and Weiss, 1998, for discussions of the challenges of conducting randomized clinical trials)
|
From page 108... ...
One possible approach is to mount a large-scale, multiyear evaluation study for the Elementary and Secondary Program as a whole, rather than attempting to do longitudinal studies for individual projects in the program. Alternatively, longitudinal studies might be carried out only for those projects in which tracking individual students is facilitated by the design of the project, such as the proposed INSPIRE project.
|
From page 109... ...
. ACCOUNTING AND PROJECT MONITORING The new plan for the Elementary and Secondary Program specifies accounting and review requirements for individual projects.
|
From page 110... ...
Accessing data as it is gathered in standard NEEIS forms is straightforward. However, summarizing data in nonstandard ways requires building a form through a complicated interface or having the central NEEIS staff build such a form.
|
From page 111... ...
One solution might be for individual projects to maintain their own databases, though there are inefficiencies in this model given that projects are required to enter data in NEEIS. Currently, individual projects appear to vary in terms of whether they maintain databases or other systematic project files outside of NEEIS.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.