Preparing for Terrorism Tools for Evaluating the Metropolitan Medical Response System Program (2002) / Chapter Skim
Currently Skimming:
5 Measurement and Data Collection in Evaluation
5 Measurement and Data Collection in Evaluation
Pages 75-90
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 75... ...
Systematic assessment is a means to distinguish evaluation from subjective impressions or anecdotal evidence. Systematic assessment may be qualitative or quantitative in nature, but in all cases it is self-conscious about the need for validity and reliability in the assessment.
|
From page 76... ...
Because true terrorist attacks are, fortunately, still rare, an assessment of the ultimate outcome of the MMRS is not likely to be available for many cities. Instead, intermediate outcomes, referred to hereafter as outputs, are more feasible and are represented by progress of various elements of the system in response to exercises, false alarms, and nonterrorism events.
|
From page 78... ...
Here the objective is to diagnose how well the program as a whole or individual grantees are performing with the objective of remedying shortcomings and identifying and replicating best practices. For a program that is likely to continue irrespective of current levels of performance because its substantive purpose is regarded as a critical public need, the management improvement function of program evaluations may well be as important as or more important than accountability.
|
From page 79... ...
The accountability function is often the prime motivation for evaluation requirements in federal grant programs. The information gathered during evaluations to assess accountability is useful in determining whether future funding commitments should be made in an agency's internal budget preparation process or in the congressional appropriations process, showing stakeholder groups that a problem is being effectively handled, or determining whether to reward a specific grantee with additional resources (or, possibly, not to renew or complete funding)
|
From page 80... ...
However, there are some definite tensions between these purposes: · Managers in grantee organizations are more likely to feel wary of and are less likely to be cooperative with evaluations undertaken primarily for accountability purposes than with those aimed at management improvement.
|
From page 81... ...
For example, federal decision makers often request information intending to make summative judgments, but they often use the information in a formative fashion to "tinker" with programs (Cronbach et al., 1980; Leviton, 1987; Leviton and Boruch, 1983~. In the federal MMRS program context, OEP might indicate to a city that it had made satisfactory progress in building MMRS capacity, a summative judgment of worth, and then go on to indicate areas that still required improvement, a formative judgment.
|
From page 82... ...
Yet, Congress might then request information about how to overcome barriers to getting these requisite inputs in place, a formative question. · OEP might also indicate that an MMRS program city had not engaged in a satisfactory number of situational exercises or drills within the past year, an evaluation of process and a summative judgment.
|
From page 83... ...
The sheer number of possible variations in terrorist incidents (the weapon used, the mode of delivery, the range of crisis capacities that are essential or that could be used, the targets of the terrorist attack, and relevant aspects of the urban situation) means that even though the preparations may be perfect for one incident, they may have limited applicability to another incident.
|
From page 84... ...
Even if abundant examples of terrorist events were available, the "crisis capacity" itself could be inferred only from the responses of many different players at the federal, state, and local levels operating within complex systems and reacting to complex events. However, OEP personnel already infer crisis capacity in ways beyond the scope of the checklist that they use.
|
From page 85... ...
First, MMRS capacity or preparedness is more usefully viewed as a complex policy goal than an absolute set of conditions that can be directly ascertained. In this respect, MMRS capacity or preparedness is like other big policy goals discussed in the United States, for example, policies related to access to medical care, privacy, and child health and well-being.
|
From page 86... ...
permit quantification of the degree of preparedness or the capacities of the MMRS. Low-frequency events such as terrorist attacks offer measures that are not sensitive to real improvements because the level of measurement (disaster versus no disaster)
|
From page 87... ...
Because the skills of the individuals who make up these component parts are periodically assessed, to the degree that their functions are relevant to the performance of the program in a terrorist incident, these assessments can partially represent the capacity of the MMRS. The performances of the component parts of the MMRS program in response to various actual emergencies and incidents that bear some resemblance to what might occur in a terrorist attack could be assessed.
|
From page 88... ...
The Federal Emergency Management Agency's National Urban Search and Rescue Team program may provide a model for the evaluation of an MMRS at different stages of development. The Urban Search and Rescue Team program uses a three-stage process designed with the idea of a progression of developmental phases.
|
From page 89... ...
Communications Channels Evaluations undertaken for management improvement purposes have less defined boundaries than evaluations undertaken for accountability purposes. One of the key side effects of such evaluations is therefore the establishment and densification of lines of communication from a grantee to other grantees and outside experts.
|
From page 90... ...
Qualitative studies provide greater depth of understanding about a small number of cases or subjects, often identify new variables for study or new relationshins among variables, but that understanding may not generalize beyoncr the few cases studied. Quantitative measures typically provide greater breadth of understanding and, depending on the research design, may allow for strong inferences about causation, but the depth of knowledge will be limited (Cronbach, 1982; Francisco et al., 2001~.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.