21st Century Manufacturing The Role of the Manufacturing Extension Partnership Program (2013) / Chapter Skim
Currently Skimming:
5 MEP Center Performance Measures and Evaluations of Program Outcomes
5 MEP Center Performance Measures and Evaluations of Program Outcomes
Pages 71-102
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 71... ...
The evaluations themselves have usually been carried out by various third-party private consultants, university researchers, policy foundations, or government examiners. This chapter begins by considering the broader challenges involved in measuring MEP center performance and evaluating program outcomes and by discussing how the committee addressed these challenges.
|
From page 72... ...
A number of center directors have observed that the very high level of survey responses that MEP headquarters considers to be acceptable has the effect of ensuring that local MEP centers expend considerable efforts and resources on ensuring that clients comply and complete the survey. Current average response is 85 percent, and the amount of effort expended by centers to ensure compliance is reported to range from 5 percent to 35 percent of total staff time.1 Incentives and Performance Indicators In addition, several center directors have noted that the reliance on survey data as a primary metric of center performance introduces related incentives for centers (whose performance evaluations largely rest on survey data)
|
From page 73... ...
The fact that a company is willing to engage with an MEP center and to seek improvements in its processes may in itself be a distinguishing characteristic, which separates such firms from essentially identical firms that are not prepared to take this step. A few controlled studies have sought to address this concern by matching comparison groups and adjusting for performance prior to program entry.
|
From page 74... ...
Carrying out multiple site visits and interviews with MEP center directors and staff b. Commissioning a comprehensive review of previous MEP evaluations in the professional literature c.
|
From page 75... ...
However, it does not appear that MEP has focused on this question. Such factors metrics might include demographics of the client population, including size of company, skills levels within client companies and in the local population, degree of existing integration into global supply chains, role of local universities and colleges, nature of existing industry clusters, current average value added per employee, and transportation links and ease of center access to clients.
|
From page 76... ...
Population Size of Service Area MEP centers vary enormously in the size of the SME population that they serve. Based on MEP estimates, the range is from more than 31,000 companies in Southern California to approximately 500 in Alaska.
|
From page 77... ...
While there are outliers on both dimensions, with a maximum of just under $4,000 per TABLE 5-2 Largest Centers by Funding Volume, FY 2010 California Manufacturing Technology Consulting (CMTC) CA $28.8M Ohio MEP OH $22.4M New York MEP NY $21.2M Texas Manufacturing Assistance Center (TMAC)
|
From page 78... ...
Area SME Alaska MEP AK 1,960,534 501 3,913 North Dakota Manufacturing ND 2,370,782 752 3,153 Extension Partnership MEP Utah UT 9,143,913 3,312 2,761 Northwest Pennsylvania Industrial PA 3,590,377 1,633 2,199 Resource Center (NWPIRC) IMC-PA PA 2,095,214 996 2,104 Manufacturing Works WY 1,193,002 568 2,100 MRC PA 3,206,326 1,537 2,086 New Mexico MEP NM 3,110,313 1,535 2,026 Mid-America Manufacturing KS 5,476,200 3,143 1,742 Technical Center (MAMTC)
|
From page 79... ...
The three centers with more than $1milion in claimed impacts per client were: Mississippi Technology Alliance $3,628,316 GENEDGE ALLIANCE $1,777,412 Arkansas Manufacturing Solutions $1,752,289 The next highest claimed just over $900,000.
|
From page 80... ...
There are a number of centers in the top quintile for impacts per client who score much lower when considering their impact on the overall SME population; and conversely, some that score low on client impacts score high in the context of the overall SME population, as shown in Table 5-4. This review shows that while Alaska scores only moderately in terms of bottom-line impact per client, normalizing by the number of SMEs indicates that Alaska had a relatively high impact on the overall SME population.
|
From page 81... ...
CO Colorado Association for Manufacturing and 199,414 2,131 Technology SOURCE: MEP data, FY 2010, NRC staff calculations. Those centers with the largest jobs impact per SME are also to a considerable degree the centers with the largest jobs impact per client.
|
From page 82... ...
Where jobs per impacted client are 30 or more, the center is probably not primarily focused on smaller firms. EFFICIENCY MEP gathers a considerable amount of data related to efficiency, which we define as federal resources per unit of defined output.
|
From page 83... ...
. These control group studies generally find that MEP clients performed better than comparable firms who did not receive MEP services, as measured by changes in value added, productivity, or resilience during recession, although the Cheney et al.
|
From page 84... ...
The data indicate that there are massive differences between the most and least frugal centers. The Northwest Pennsylvania Industrial Resource Center uses less than $14,000 in federal funding per impacted client, while the Maine MEP utilizes more than $87,000.
|
From page 85... ...
SCOPE AND OUTREACH Centers have been under some pressure to improve penetration, and perhaps in part as a result have been reporting a substantial increase in "touches" -- contacts with potential clients. However, the number of actual client engagements has been essentially flat since at least 1999.
|
From page 86... ...
a given size population may be addressable by the local MEP center -- they may, for example, be uninterested in MEP services, or already have access to alternative services. FIGURE 5-6 MEP market penetration trends.
|
From page 87... ...
MEP itself suggests that 80 percent of all SMEs are not sufficiently engaged in strategic thinking to be ready for MEP services.4 Chuck Spangler of South Carolina MEP observed that most of his clients are in the range of 100 employees.5 Other center directors have made similar observations. MEP centers are understandably cautious about expending efforts to engage potential clients who are not ready for their services: There is longer-term value in mentoring and readying potential clients, but such efforts may well impinge on the immediate delivery of measurable and revenue generating projects.
|
From page 88... ...
The need to meet the federal match through fees and the reliance on in-kind contributions may also be pushing the program away from its presumptive target group -- i.e., away from smaller companies and rural firms that should be a part of MEP's public vocation. MEP centers that work with the target audience of SMEs also tend to have smaller jobs impacts.
|
From page 89... ...
Of the top 10 positions, 5 are occupied by Pennsylvania centers, including the top 2. The data suggests that where regions have large SME populations, centers may struggle to touch a substantial share of these companies, and hence to generate a high rate of client acquisitions relative to the population.
|
From page 90... ...
90 21ST CENTURY MANUFACTURING TABLE 5-7 Measure of Outreach Success, FY 2010 Percent Touch quintile Impacted Clients/year #SME in service area Impacted clients per Percent Impacted Touch Percent of Sum of quintiles Impacted clients Percent SMEs Touches/yr quintile scores SMEs touch Center Name Highest scores PA Catalyst 3,282 635 139 19.3 4.2 21.9 5 5 10 PA NE IRC 1,131 219 117 19.4 10.3 53.4 5 5 10 NM MEP 1,535 306 73 19.9 4.8 23.9 5 5 10 PA IMC 996 292 70 29.3 7 24 5 5 10 AK MEP 501 397 43 79.2 8.6 10.8 5 5 10 WY Mf- 568 124 41 21.8 7.2 33.1 5 5 10 Works SC MEP 4,260 854 158 20 3.7 18.5 5 4 9 IA CIRC 3,718 710 152 19.1 4.1 21.4 4 5 9 PA NW IRC 1,633 223 151 13.7 9.2 67.7 4 5 9 MRC PA 1,537 256 102 16.7 6.6 39.8 4 5 9 DE MEP 634 82 47 12.9 7.4 57.3 4 5 9 HA MEP 970 467 28 48.1 2.9 6 5 4 9 Lowest scores CA Manex 12,395 506 89 4.1 0.7 17.6 1 1 2 WA Imp 7,582 435 86 5.7 1.1 19.8 1 1 2 IN Purdue 8,790 355 68 4 0.8 19.2 1 1 2 MEP OR MEP 5,658 285 61 5 1.1 21.4 1 1 2 AZ MEP 5,005 211 59 4.2 1.2 28 1 1 2 CO MEP 5,240 253 56 4.8 1.1 22.1 1 1 2 MD UMD 3,601 124 51 3.4 1.4 41.1 1 1 2 MA KY MAC 4,064 174 32 4.3 0.8 18.4 1 1 2
|
From page 91... ...
. The logic models describe MEP services, showing inputs, work processes, intermediate outcomes, and impacts.
|
From page 92... ...
(the association of MEP centers) , along with other nonprofit organizations, has turned the emphasis of MEP evaluations towards strategic redirection of the program (NAPA 2003, 2004; AMSC 2009; Stone & Associates and CREC 2010; MEP Advisory Board, Yakimov, and Woolsey 2010)
|
From page 93... ...
members and found that only 1 percent indicated that government was an important source of assistance in technology investment decisions. However, many MEP centers are known through their university or center name rather than as a source of government assistance, making it difficult to measure use of the program beyond counts of manufacturers served.
|
From page 94... ...
found that MEP sponsorship led to greater service coordination than individual center efforts alone or state government demands would have provided, which in turn generally improved the services to MEP clients, albeit at a significant expenditure of resources for validating and coordinating with these providers. Partnership with state governments has been an important element for some centers.
|
From page 95... ...
However, GAO (2011) found that 80 percent of MEP centers were very or somewhat likely to give preference to revenue-generation projects with larger clients.
|
From page 96... ...
Technology Adoption Shapira and Rephann (1996) observed that clients of a manufacturing technology assistance program in West Virginia were more likely to adopt individual technologies and were also more amenable to technological investments than nonparticipants.
|
From page 97... ...
Jarmin's analysis of eight MEP centers from 1987 to 1992 found productivity increases in clients over nonclients ranging from 3.4 percent to 16 percent. Nexus Associate's analysis of Pennsylvania centers reported higher labor productivity of 3.6 percent to 5 percent in clients as compared with nonclients.
|
From page 98... ...
Winning Ways Client Engagement Case Studies," prepared for the National Institute of Standards and Technology Manufacturing Extension Partnership, January 2008. See also Lynne Manrique, Kamau Bobb, David Roessner, Jan Youtie, and Philip Shapira," Eureka!
|
From page 99... ...
, Nexus Associates (1996) , and Michigan Manufacturing Technology Center (1996)
|
From page 100... ...
Examine consistently high-performing centers' service mix and survey practices. SRI and Georgia Tech Focus marketing on target customer characteristics; think (2008)
|
From page 101... ...
Researchers identified some significant issues with the MEP survey: Methodologies required by MEP for firms to calculate value added are quite complex and may lead to undervaluation of outcomes. Incentives from MEP for survey completion and for positive results reported through the survey likely provide strong incentives for center staff to encourage the most positive view possible of firm results.
|
From page 102... ...
Similarly, no assessment exists of other aspects of the new strategy, green manufacturing, and supply chain integration and export-led growth. There does not appear to have been a significant review of program management since NAPA (2004)
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.