Learning from Our Buildings: A State-of-the-Practice Summary of Post-Occupancy Evaluation (2001)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

4 Post-Occupancy Evaluation Processes in Six Federal Agencies The federal government is the largest owner of Council who volunteered to provide information about facilities in the United States. More than 30 individual their post-occupancy evaluation processes. In some federal agencies own, use, and acquire facilities to sup- cases, more than one person in the agency responded port agency missions and programs. Some federal and the responses were combined and reconciled. The agencies conduct post-occupancy evaluation (POE) agency representatives were asked to respond to the and lessons-learned programs as ways to improve cus- following questions: tomer satisfaction, to increase building quality and per- formance, and to facilitate organizational learning. This 1. Approximately what year did your agency estab- chapter provides information about POE processes in lish a post-occupancy evaluation program? On six federal agencies: the U.S. Air Force, Office of the average, approximately how many POEs have Civil Engineer; the General Services Administration, been conducted by your agency in each of the last Public Buildings Service (PBS); the Department of the five fiscal years? Interior, National Park Service (NPS); the U.S. Navy, 2. What were the driving factors for establishing a Naval Facilities Engineering Command (NAVFAC); the POE program? U.S. Department of State, Office of Overseas Buildings 3. What is the focus of the POE information-gather- Operations (OBO); and the U.S. Postal Service (USPS). ing process (e.g., user satisfaction, achievement The six agencies are sponsors of the Federal Facili- of design objectives, building performance, ties Council and volunteered to participate in the study. other)? The information was gathered through a questionnaire 4. What are the expectations, goals, and objectives and telephone interviews conducted by Krista Waitz of for the program? Have they been achieved? Have Kwaitz Consulting. National Research Council staff there been unanticipated results? wrote the summary of findings and descriptions of POE 5. How is the information gathered through POEs programs. tied into the feedback loop (lessons learned) for The study design was not a scientific one, nor was it planning, programming, and capital asset man- based on random sampling. Thus, the information pro- agement? vided should not be generalized. The remainder of this 6. Is the information gathered through POEs used in chapter contains information about the survey ques- real estate decision-making and capital asset man- tions, a summary of findings, and descriptions of the agement? If yes, please note what information is POE programs in the six agencies. used, how, and when it is used. If no, why not? 7. Is it your agency’s policy to conduct POEs for all buildings or for selected facilities? What are the SURVEY QUESTIONS criteria for determining whether a POE will be In January 2001, a questionnaire was designed and conducted? issued to six sponsor agencies of the Federal Facilities 35 

36 LEARNING FROM OUR BUILDINGS 8. What do you consider to be barriers to conduct- for client satisfaction and for management-related ing successful POEs? issues. 9. Who is typically involved in conducting a POE in terms of in-house personnel and consultants? POE Process and Technologies (Please list positions or types of skills involved not individuals.) What types of technologies are No two agencies use the same process or tools for used? conducting POEs and capturing lessons, although some 10. What is the estimated cost in time and dollars for share common elements. The National Park Service is conducting a typical POE? in the process of developing new procedures and tools 11. What data collection methods, technologies, and for conducting POEs and sharing lessons learned. Cur- survey forms have been used over the life of the rently, at the NAVFAC, an independent agent remote program? Please provide copies of survey forms from the designer of record conducts the POE using a that have been used. statistically based questionnaire; a focus group discus- 12. To what extent does your agency make use of data sion is then conducted to summarize the results of the management systems, Internet tools, or other survey. The NAVFAC criteria office administers information technology applications to share survey results. The questionnaire is being modified so information and disseminate results of POEs? that it can be administered from field agencies and can be completed on-line or downloaded from the Web. The survey documents and results can be downloaded SUMMARY OF FINDINGS from a database on the Web. The Air Force, in contrast, uses a questionnaire Establishment of POE Program, Timing of Surveys, administered by a staff team; feedback is given prima- and Number Conducted rily to the construction agent, although the Air Force Each of the six agencies studied had had a POE plans to also share the results with users, the base civil program in place at least since the 1980s. The POE engineer, and the major command. programs of the PBS, NPS, and NAVFAC were being The Office of Overseas Buildings Operations uses restructured to meet new objectives, and the results of pre-trip user questionnaires, on-site interviews, and the reorganized programs were not yet available. facilitated town meetings to gather the information, The number of POEs conducted annually, on aver- which is then summarized in a formal report. OBO’s age, ranged from less than 1 to 30. Post-occupancy POE is conducted by a multidisciplinary in-house team evaluations are typically performed within 4 to 24 that is customized to address known deficiencies. months following occupancy of a new or renovated The PBS performance measures-oriented approach facility and are performed only once for an individual uses a set of questionnaires developed in cooperation building. with the Center for the Built Environment at the Uni- versity of California at Berkeley. The questionnaires are being designed to be administered over the Web. Focus of POE Programs The lessons learned will provide input to the ongoing Each of the six agencies used POEs to determine performance measures program of the Office of Busi- client or user satisfaction at some level, but it also used ness Performance. them to fulfill other objectives. These objectives The U.S. Postal Service uses two levels of surveys. included determining building defects within the con- The first, a basic questionnaire that can be completed struction warranty period, supporting design and con- in about an hour by the administrative service officer struction criteria, supporting performance measures for and the postmaster of a new facility, is required for all asset management, evaluating construction inspectors, new construction. For larger, more complex projects, lowering facility life-cycle costs by identifying design POEs are conducted over two to four days using a errors that could lead to increased maintenance and multidisciplinary team. Both types of surveys use elec- operation costs, clarifying design objectives, improving tronic questionnaires in Microsoft Excel. The informa- building performance, and supporting corporate sales tion gathered is sent directly to the staff maintaining and image objectives. The restructured programs in the agency’s design standards. NAVFAC and PBS are focused on developing metrics 

POST-OCCUPANCY EVALUATION PROCESSES IN SIX FEDERAL AGENCIES 37 How POE Information Is Used tered through a headquarters’ office, there may be a lack of field-level attentiveness to the process. Obtain- Information obtained from POE programs has been ing the clients’ attention to ensure adequate participa- used by the OBO, NAVFAC, PBS, and USPS in sup- tion in the survey or getting people not originally port of their design criteria and guidelines. The NPS involved with the project to participate in a survey was and the Air Force programs anticipate using POE also identified as a barrier. Organizational structures results for the support of design criteria, among other can also create barriers when responsibilities are objectives. To date, none of the six agencies reported assigned such that POE administration and database that POE information was used directly in future real development require interoffice collaboration. estate decision-making and capital asset management, although PBS’s POE program was being restructured with those objectives in mind. Costs The costs reported to conduct POEs ranged from Barriers $1,800 for a simple standard questionnaire that could be completed in one hour to $90,000 for an in-depth A number of barriers to more effective use of POEs analysis including several days of interviews, multi- and lessons-learned programs were identified. These disciplinary teams, site visits, and writing up reports. could be categorized generally as resources, feedback, The costs did not include implementation of any and participation and commitment. changes resulting from a POE study. Costs per square foot of space evaluated were not available. Other vari- Resources ables accounting for the range of costs included whether the facilities were located in the United States Several agencies noted it was difficult to obtain or or abroad, whether in-house staff or consultants were earmark the funding needed to conduct POEs regard- used, and how the resulting information was packaged less of whether the POEs were to be conducted using and distributed. consultants or in-house staff. In some cases, in-house staff may not be available to conduct the POE or may not have the technical skills needed for quality results. DESCRIPTIONS OF POE PROGRAMS National Park Service, Department of the Interior Feedback In the mid-1980s the National Park Service com- Because POEs often focus on identifying deficien- pleted formulation and development of an extensive cies, they risk becoming instruments to focus or deflect POE program; however, due to changes in staff and blame for unsatisfactory results. One agency cited the downsizing of the NPS central design office, the concern by senior executives that lessons learned may program was not fully executed. In 1998, the POE pro- be considered a weakness by Congress or the Inspector gram was reinstituted as part of the business practices General. Other agencies noted that conducting a focus for the NPS central design office. At that time, it was group to achieve consensus about the cause of failures decided to reestablish the POE program for three without judgmental discussion can be difficult, and as reasons: (1) to create a feedback loop that would allow a consequence, they may be reluctant to do so. designers to interact with facility users and learn if facilities were meeting the needs of the users; it also Participation and Commitment served as an opportunity for users to assess their origi- nal requirements and determine if they had adequately One agency noted that because of construction identified their needs; (2) to evaluate the effectiveness schedule constraints, staffs may be focused on future of consultant construction inspectors who had recently projects or those under visible construction. Thus, once replaced all NPS construction inspectors; and (3) to a project has been completed and occupied, items such improve long-term facility life-cycle ratio costs by as financial closeout, construction as-builts, and POEs identifying any design errors that could lead to may not be a high-priority item and may not receive increased maintenance or operational costs. adequate oversight or attention. For programs adminis- 

38 LEARNING FROM OUR BUILDINGS The focus of the NPS POE program is to improve Naval Facilities Engineering Command, user satisfaction, building performance, and designer Department of the Navy efficiency. The central design offices for the NPS seek The Naval Facilities Engineering Command has to retain highly talented designers over a long-term conducted POEs since the 1960s. In its original form, career. By building relationships with facility users, NAVFAC’s POE program had an instructional base designers better understand user needs and anticipate and was conducted by a project team after construction requests, allowing for more efficient use of design to identify lessons learned. In 1997, a new statistically funds. The POE program seeks to build on this type of based concept was initiated. The goal is to establish a relationship through open communications and on-site statistical basis from which NAVFAC can measure review of completed construction projects. The NPS agency improvement and work toward continuous expects the value-added component of the POE will be improvement by implementing process changes and in lessons learned and improved design efficiency. The modifying design criteria. The POE program is intended NPS noted that maintenance and operations costs for to apply to each completed facility within its warranty facilities are escalating annually, and if the POE pro- period. On average, 20 POEs have been conducted in vides information for future designs that lead to each of the last five fiscal years. improved maintenance or operations, the payback will There were two driving factors for restructuring the be dramatic. In addition, if the POE provides feedback POE program at NAVFAC. The first was a published on products or techniques that improve user satisfac- Department of Defense-level survey of occupant tion, reduce maintenance, or improve operations, this satisfaction with their facilities. The second was a information can be shared with designers to reduce the NAVFAC headquarters’ management initiative to cre- cost of design development. ate metrics measuring how, or if, the agency exceeded Data collection tools and the forum for sharing infor- client expectations. The agency focuses on improving mation are still under development. However, the POE client satisfaction, determining where its product lines program is envisioned to include an evaluation form or processes give rise to client dissatisfaction and and follow-up meetings approximately six months to improving the individual facility on which the survey one year after completion of construction. The POE is conducted. NAVFAC is issuing policy that will meetings will include the users, project manager, and require a survey of all facilities within 6-10 months of design team captain. After completing the evaluation, building occupancy. The focus of the POE informa- the materials will be shared within the project manage- tion-gathering process is to measure user satisfaction ment and design divisions. The results of the POE will from the perspectives of the building owner, the cus- also be placed in the central technical information files tomers (student, family occupant, day care family and may be accessed by various levels of project man- member, etc.), the building occupant at the working agers, designers, and other technical staff. level, and the staff maintaining the facility. Client The NPS anticipates its POE teams will consist of satisfaction at the user level is measured whether or not (1) users, including park superintendent, chief of main- the client was involved in the planning, design, or con- tenance, park rangers, park interpreters, and adminis- struction phases. The measuring process includes a trative staff; (2) project managers, a multidisciplinary survey of participants after which a focus group dis- group consisting of park planners, architects, landscape cussion is conducted to summarize the positive and architects, and various disciplines of engineers—the negative aspects of the facility. technical expertise will vary by project; and (3) design Currently, all POE surveys are conducted by an team members. Generally a team captain for a building “independent agent” remote from the designer of project would be a senior architect; for a road construc- record, typically a consultant. The NAVFAC criteria tion project, a senior civil engineer or senior landscape office administers the surveys. The criteria manager architect would attend the POE; and for a utility project, for the facility type identifies criteria issues and a senior civil engineer would attend. Depending on the action(s) to be initiated and implements interim guid- complexity of the project, other members attending a ance to the organization when required. The survey will POE meeting could include a mechanical engineer or be used in developing and modifying planning and an electrical engineer. design criteria. The data will be reviewed annually to determine process improvement needs. NAVFAC is 

POST-OCCUPANCY EVALUATION PROCESSES IN SIX FEDERAL AGENCIES 39 working to determine how the survey will interface the POE information-gathering process is on user with its knowledge management system, combining to satisfaction, achievement of design objectives, and yield a single lessons-learned concept for the organi- building performance, including interior flexibility and zation. functionality. When the POE program process began, NAVFAC The POE methodology followed was adapted from used a checklist format focusing only on design and the U.S. Postal Service. Over the life of the POE construction. The survey was changed to accommodate program, occupant surveys, on-site interviews, and all of the agency’s processes (i.e., planning, design, facilitated town meetings have been used for data col- construction, and maintenance turnover). Additional lection. The first step in the POE process is to send an minor changes are anticipated to better assess safety occupant survey to an overseas post. The preliminary and procurement issues. A copy of the survey is con- results from the occupant survey are used to determine tained in Appendix D. what disciplines should be represented on the multi- The database is being modified to Web-enable the disciplinary team that will be conducting the POE; thus, survey content and to create a field-managed site as the team is customized to address known deficiencies. opposed to a headquarters central database. In addi- Typically, architects; electrical, mechanical, and struc- tion, the database is being made integral to corporate tural engineers; facility maintenance; and security ORACLE-based management systems. It will draw specialists are involved in conducting POEs. Once at information from the management system and alert the site, the team conducts a walk-through followed up assigned individuals when it is time for the survey. by interviews with occupants. At the conclusion of the site visit, members of the team reconvene to discuss their observations and to generate recommendations for Office of the Civil Engineer, U.S. Air Force a report. Due to constrained resources, a POE may con- The focus of the Air Force POE information- sist solely of the occupant survey. gathering process is user satisfaction, achievement of One result of the POE program has been the revision design objectives, and improved building performance. of design guidance on such topics as roofs, elevators, The purpose of the POE is to note all defective work, and Ambassadors’ residences. POE results were also report construction deficiencies to the construction used for developing a serviceability demand profile for agent for correction by the contractor, and document generic embassy office buildings slated for design and problems or mistakes made during design for use as in design guidelines for future embassies being acquired lessons learned on similar projects. under a specialized procurement process. The POE is conducted by a staff team using a ques- tionnaire. The Air Force plans to share the results of Public Buildings Service, the POE with the construction agent, the user, the base General Services Administration civil engineer, and the major command for use in any future designs and for incorporation into Air Force The Public Buildings Service (PBS) of the General design standards. It is Air Force policy for a POE to be Services Administration (GSA) first instituted a POE accomplished sometime during the ninth to eleventh program oriented toward design and construction crite- month following beneficial occupancy (acceptance of ria development in 1977. In-house technical experts, the facility by the user agency). including an environmental psychologist, architects, and engineers, conducted the surveys. The program was curtailed in 1982 due to a reduction in staff. PBS’ Office of Overseas Buildings Operations, POE program was reinstituted in 1986, using contractor Department of State support. Between 1986 and 2000, the PBS completed At the Office of Overseas Buildings Operations approximately 30 POEs for a variety of projects, (formerly Foreign Buildings Operations) of the U.S. including courthouses, office buildings, U.S. border Department of State, lessons learned and design or con- stations, major renovations, and historic restorations. struction alerts have been issued since 1985. The driv- In 2000, the PBS restructured its POE program to focus ing factors for establishing a POE program were the on performance measures for asset management. Both concern for user satisfaction, comfort, and safety and a the design criteria-oriented POE program and the general desire to capture best practices. The focus of performance measure-oriented POE program were 

40 LEARNING FROM OUR BUILDINGS intended to provide an information stream that would (CDs). A compendium of lessons was distributed every inform program managers, criteria managers, and three years. In 1998, PBS prepared a Compendium of project managers about design- and delivery-related Lessons Learned CD-ROM that was widely distributed, problems and associated best practices. and a DVD (Lessons Learned, Volume 2) is being pre- pared for mass distribution. The POEs are also in the construction criteria database of the National Institute Criteria–Based POEs for Building Science. Criteria-based POEs were developed to provide A second program that resulted from the POE les- technology- and procedure-based feedback to those in sons is HVAC (heating, ventilation, air conditioning) PBS’s central office responsible for national program Excellence in Federal Buildings. PBS held numerous and design criteria direction. Equally important were workshops that included staff, architects, engineers, the perceived benefits of offering those same lessons to and representatives from professional societies and the delivery teams responsible for new projects. The technical organizations as part of an awareness pro- primary focus was on building systems evaluation, gram to highlight HVAC issues. client satisfaction surveys, and interviews with major client agencies. The building systems evaluations Performance Measure-Oriented POEs included functionality issues as well as an overview of how well the building complied with design criteria. The driving factor behind the shift to a performance The surveys were customized for different types of measure-oriented POE program was PBS senior man- buildings and project delivery systems, including office agement’s desire to evaluate how well PBS’s assets are buildings, courthouses, border stations, and lease-build achieving their objectives on a project and program or design-build projects. (An example is included in basis. Performance measure-oriented POEs are being Appendix D.) Major lessons from the POE process pursued to help indicate whether delivery practices and related to long-term building maintainability, building criteria are effective and to identify systemic problems, functionality, client needs, and property manager and whereby specialty studies could be pursued. asset management needs. Specific issues concerned The focus of this POE information-gathering process energy efficiency; indoor air quality; heating, ventila- is financial asset assessment. A set of extensive ques- tion, air conditioning, and electrical systems; thermal tionnaires is used in an attempt to identify customer comfort; design of loading docks; access to equipment; satisfaction with various building components or window washing; and accessibility for the physically features. Different questionnaires are directed to dif- disabled. The criteria were constantly revised to incor- ferent key personnel, including operating staff and porate lessons from POEs. design-delivery team members. The measures are From 1986-2000, criteria-based POEs typically in- intended to help determine if GSA is meeting a number volved a team of five to six design-related disciplines, of key management indicators including comparison using outside architect-engineer professional services. of construction “pro forma” with final pro forma, main- PBS central office coordination and involvement were tenance and cleaning costs benchmarked against provided through site visits, access coordination, and national standards, utility costs, sustainability, energy report critiques. A report and a 30-minute video were usage against FY 2010 goals, accessibility for the created and distributed to each of GSA’s field offices physically disabled, and client satisfaction. (11 regions) and to senior-level management at head- Support for questionnaire development and database quarters. management is currently provided by the Center for The tieback to lessons learned was direct: the people the Built Environment, within the University of Cali- who conducted the POEs also developed the criteria. fornia at Berkeley. Questionnaire delivery and assess- However, because the POE reports were voluminous ments are being coordinated by senior architects and and oriented toward detailed evaluations of technology engineers within the PBS Office of Business Perfor- applications, getting project managers and designers to mance. PBS is currently designing tenant, operations read and adhere to findings was a challenge. Thus, vari- and maintenance, and design and construction survey ous forms of information exchange media were applied tools that can be administered over the Web. to mitigate this problem, including condensed video- The goal is to perform a POE for every Congres- tapes and interactive “lessons-learned” compact disks sionally approved new building one year after full 

POST-OCCUPANCY EVALUATION PROCESSES IN SIX FEDERAL AGENCIES 41 occupancy. Once the performance measure-oriented much more powerful support for overall corporate POE program is fully implemented, 10-20 POEs will identity than previously realized. likely be completed each year. The tieback from the Two levels of POEs are used currently. A basic POE ongoing performance measure POE program will go (completing the questionnaire) is required for all new from PBS’s Office of Business Performance to other construction and for owned facilities greater than 9,000 appropriate offices within the agency. square feet within four and six months of occupancy. The administrative service office manager and the post- master complete the basic POE questionnaire, which U.S. Postal Service typically takes 30 minutes to an hour. An example is The U.S. Postal Service established its POE program included in Appendix D. in 1986. The driving force behind the establishment of More extensive POEs are conducted on larger the POE program was the desire to improve the plan- projects (more than 30,000 square feet) or other special ning, design, and construction of future facilities. The projects. These POEs involve architectural or engi- focus of the POE information-gathering process is user neering firms (including environmental psychologists satisfaction (customers and employees), clarification of as consultants) and are conducted over a period of two design objectives, achievement of design objectives, to four days. Customers and employees are inter- building performance relative to technical systems such viewed, and extensive lighting and HVAC data are as cooling and lighting, and supporting corporate sales gathered. and image objectives and economics. On average, the The results of POEs go directly to the staff main- agency has conducted approximately 30 POEs in each taining the Postal Service Building Design Standards. of last five fiscal years. The information gathered through the POE process is The first POE application was an effort to standard- not used in real estate decision-making or capital asset ize the hundreds of Postal Service retail spaces pro- management; however it is used in planning, design, duced each year. Appropriate design was found to be a and construction decisions. Real estate decisions are much more powerful factor in customer satisfaction affected only as site-planning criteria are modified. than had been anticipated. Also, building image was a