21st Century Manufacturing The Role of the Manufacturing Extension Partnership Program (2013) / Chapter Skim
Currently Skimming:
4 Development of MEP Center Metrics
4 Development of MEP Center Metrics
Pages 55-70
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 55... ...
After that, adjustments were made as needed, for example, through an increase in the minimally acceptable score for individual centers in FY 2007. Starting in 2009, MEP undertook an effort to address some of the criticisms that were raised with existing metrics and their use for evaluation, notably that they rely on data from the client survey, which some critics held to be unreliable.
|
From page 56... ...
NOTE: MAIM refers to Minimally Acceptable Impact Measures, against which NIST MEP has evaluated MEP center performance since the early 2000s. FIGURE 4-2 Program review and evaluation -- three levels.
|
From page 57... ...
Magnet Ohio, for example, ran 57 events attended by 1,601 people from 791 companies in FY 2011, which contributed to acquiring 99 client projects.1 Bottom-line Impact MEP has traditionally measured the program's bottom-line impact by aggregating data on client cost savings and new sales, drawn from the client survey. This is the source of MEP's claims about overall program effectiveness, 1 James Griffith, "The MAGNET Story: From Lean Manufacturing to Partnerships for Innovation," National Academies Workshop on "Diversity and Achievements: The Role of the Manufacturing Extension Partnership in the Midwest," March 26, 2012.
|
From page 58... ...
for example, that its efforts led directly to $9 billion in new and retained sales for its customers in FY 2008.2 Jobs MEP also utilizes aggregate metrics focused on jobs. The most recent MEP performance report leads off its summary data section by indicating that according to the MEP client survey, MEP clients in aggregate claim to have generated or saved more than 53,000 jobs in FY 2010.3 CLIENT-LEVEL METRICS AND PERFORMANCE ASSESSMENT Client-level metrics are (except for success stories)
|
From page 59... ...
The survey identified a total of $3.3 billion in increased sales and $5.5 billion in retained sales, as shown in Figure 4-4. The report for FY 2009 indicated that about 29 percent of projects reported increased sales and 38 percent reported retained sales.
|
From page 60... ...
Finally, the survey sought to identify "promoters" and "detractors" of the MEP program, a metric used to measure customer satisfaction. Respondents are asked whether they are likely to recommend MEP center services to other local companies.
|
From page 61... ...
MAIM is a bar against which to measure center performance which is either minimally acceptable or not. Each center is reviewed quarterly using four-quarter rolling averages, then again annually and biennially, using a weighted scoring system: 1.
|
From page 62... ...
Cost savings are seen as falling directly on to the bottom line. Bottom-line impact ratio.
|
From page 63... ...
&+$37(5 8VLQJ WKH PHWULFV DV GHILQHG E\ 0(3 LW LV SRVVLEOH WR WUDFN PHGLDQ FHQWHU SHUIRUPDQFH RYHU WLPH ) LJXUH VKRZV PHGLDQ SHUIRUPDQFH IRU DQG 0LQLPDOO\ $FFHSWDEOH ,PSDFW 0HDVXUH 0$,0 (DFK RI WKH ILYH FRPSRQHQWV RI 0$,0 KDV LWV RZQ PLQLPDOO\ DFFHSWDEOH OHYHO RI SHUIRUPDQFH DQG ZHLJKWHG VFRUH VHH 7DEOH 7KH ILYH VFRUHV DUH ZHLJKWHG WR UHIOHFW WKH UHODWLYH LPSRUWDQFH RI HDFK ,QGLYLGXDO VFRUHV DUH DJJUHJDWHG DFURVV WKH PHDVXUHV WR JHQHUDWH D 0$,0 VFRUH IRU HDFK FHQWHU $ UROOLQJ IRXUTXDUWHU DYHUDJH RI 0$,0 VFRUHV LV XVHG IRU PHDVXULQJ &HQWHU SHUIRUPDQFH ,I D &HQWHU PHHWV RU H[FHHGV WKH PLQLPDOO\ DFFHSWDEOH OHYHO RI SHUIRUPDQFH IRU D SDUWLFXODU PHDVXUH WKH\ UHFHLYH DOO WKH SRLQWV DVVLJQHG WR WKDW PHDVXUH ,I WKH\ GR QRW PHHW RU H[FHHG WKH PLQLPXP SHUIRUPDQFH OHYHO IRU D SDUWLFXODU PHDVXUH WKH\ GR QRW UHFHLYH DQ\ RI WKH SRLQWV DVVLJQHG WR WKDW PHDVXUH $JJUHJDWH VFRUHV FDQ UDQJH IURP D PLQLPXP RI IDLOLQJ WR PHHW WKH PLQLPXP WKUHVKROG IRU DQ\ RI WKH ILYH 355 314 Clients Served 170 204 17.27 Investment 23.25 9.72 Leverage Ratio 6.25 2010 2008 2004 2001 25.61 Bottom-Line Client 29.46 Impact Ratio 14.05 5.89 79 64 Impacted Clients 50 48 0 100 200 300 400 )
|
From page 64... ...
for help.12 Perhaps most interestingly, it appears that the single most important factor in helping MEP of Nebraska increase scores to meet MAIM benchmarks was improved reporting via the MEP survey, rather than an actual improvement in service.13 MAIM's Weaknesses MEP itself has recognized that the existing MAIM approach has a number of significant weaknesses: It is heavily reliant on data from the client survey. Centers and their staff are rewarded based on high levels of positive responses, which may lead some to coach clients on their responses.
|
From page 65... ...
16 Material in this section draws on MEP's Scoring Methodology document, privately provided by NIST MEP.
|
From page 66... ...
For each metric, data collected by the center is normalized by using the amount of federal funding received, and is then compared to a threshold level established for each metric by NIST MEP. Centers scoring at more than 100 percent of the threshold are graded excellent and receive the maximum score.
|
From page 67... ...
Aggregated data from clients is divided by federal funding to generate a ratio. The retained sales ratio is then compared to a threshold established by NIST MEP.
|
From page 68... ...
The new metrics are best understood as capturing six broad areas of Center activity -- Innovation Practice, Next Generation Strategy, Market Understanding, Business Model, Partnerships, and Financial Viability -- each of which has several subcomponents. Assessment is not simply focused on a limited number of outcomes; it also covers design and implementation of the center's strategy as well as its implementation of Next Generation Strategy services.
|
From page 69... ...
NIST MEP is using CORE metrics to drive the transition from the traditional lean-oriented consulting practice to an innovation and growth practice at every center; for example, the first metric listed requires a "transition from a focus on process improvement to the
|
From page 70... ...
As one center director observed, "If we are doing things right on the ground, we are confident that this will be reflected in the metrics." However, it should be noted that if at some future point NIST MEP is able to allocate additional funding to individual centers, or to reallocate existing funding between centers, the qualitative aspects of the balanced scorecard approach will require further bolstering: The transparency and independence of the metrics will undoubtedly attract close scrutiny.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.